I know exactly what you mean. It is a trap for sure, but simplifying it this way can be a slippery slope also. For example, saying Paganini or Art Tatem are bad musicians because they go for technical difficulty and speed would seem odd. But they are exactly the same. The list would clearly go on.
Buckethead offers both, super crazy riffing and whole albums with soulful rhythms and melodies without a single speed riff.
Anyway, I think I agree with your general point, but I do think all of the above are great musicians. I think Yngwei Malmsteen sucks and would be a shining example of your point, however.