Yup, pretty much what aerius said. My Spendors were the same way -- they sounded great at lower volumes, but they were easily pushed past a threshold where they initially started compressing, and pushed further they distorted. I do not find driver-based distortion pleasant in any way other than for the purposes of guitar tone. Give me a good optical compressor any day.
Yeah, maybe 'fail' is too strong a word, they don't break. They just 'fail' to deliver music musically.