Other than the complete lack of organization issue, of course.
I don't envy anyone's. I do envy people who know what they like and get it -- people like Aardvark, Deepak, postjack, even Colin. And grawk of the multiple copies of one album fame. I admire their focus.
But on the other hand, having so much music that you're constantly discovering new stuff within one's own collection is rewarding in an entirely different way.
I've been accused of liking music that is less than best -- and it's true. But I think I had adopted Hirsch's rule of 'if you don't make some missteps/fall flat on your face, you're not trying hard enough to stretch your boundaries', subconsciously, before it was even formulated as a thought, much less stated out loud.
I'm not saying everyone has to stretch their boundaries. But I do. I enjoy being surprised by something completely out of (my) left field. I also enjoy tried and true, of course, too.