My opinion about all that is yes, portable hifi is silly, as long as you use a mp3 player as source.
Even the Imod + Icap hardly beats the best discman, and is smoked by the Sony Pro Walkman WM D6 playing a good tape (yes of course, because of analog superiority).
Yes i do agree that portable amps are not that good, but try the best ones (Iqube, Pico) with a great home source i.e and efficient good headphones and you'll hear that the overall result can be great, even if it's miles behind a home amp.
So the problem is the MP3 source which is too limited (playing only uncompressed files of course, compressed ones being not listenable
at all). Mp3 is the most convenient, but what you win in convenience, you loose it in quality.