Blind testing with speakers (general public)

[deepak]

I thought this was an interesting read over at an AV forum: Best Sounding Speakers According to Scientific Tests - Audio Video Revolution Forum

Random and blind testing with members of the general public choosing their favorite speakers without knowing anything about them (ie cost or looks).

I was asked by Wes to post some results and ratings from a scientific study I'm a part of.

What we've done:

2,000 "Test Subjects," had their hearing checked and passed. We decided they had "above average to good," hearing for their age and sex.

Then we tested different parts of the audio chain: cables, amp, pre-pro, speakers, and CD players. Only 8 people could be tested at a time due to the limited "sweet spot," for sound listening for the tests. For speakers our reference rig consisted of a complete Arcam FMJ setup: amp, cd player, pre-pro. The wiring was all mid-priced Cardas. They listened to a track of Beethoven, Pink Floyd, Waylon Jennings, and Public Enemy (just so we could affect the political race, and get the phrase "corrupt like a senator," ingrained in their brains). 5 identical listening rooms were used. Each had 2 sets of speakers properly set up. They'd listen to one track and move to another room. Speakers would randomly be switched from "A," and "B" for example in room 1. So say a PSB and a Def Tech were in that room, one was A and one was B, this could be switched without the listener knowing we had changed designation for a different track.

The test was completely blind. The test subjects were never told what speaker they were listening to, nor could they see them.

We tested hundreds of different speakers at first (pretty much every brand you could think of). After many tests some speakers were removed because it was apparent they were "disliked," from results. Basically in one day 40 people would test 40 different speakers. We did this by price point. The first tests were for $0-$1000 pair of speakers. They weren't asked to "Rate," just pick the best 5 from a group (of 10). It was never the same 10 vs the same 10 more than per 40 people. We mixed it all up. Still we were able to identify speakers that were "preferred," to "disliked." Interestingly this process of elimination only resulted in nearly 30% of the speakers priced under $1,000 to be eliminated. The entire process was repeated multiple times until we had valid data showing the 10 most preferred speakers. Then those 10 speakers were "rated," by the test subjects.

Then we took the top 5 rated speakers under $1000 and added them to the speakers under $2,000 and repeated the tests with the speakers priced $1000-$2000. We again reached a point where we had the top 10 speakers. These were rated and the top 5 were then compared to $2001 to $5000 speakers. Ten speakers were preferred, then rated. Then $5001 to $10,000. Then $10,000+.

These tests are not finished. So my posting results should be taken as a work in progress. So here are some incredibly interesting findings:

The Top Rated Speaker for under $1,000:

Magnepan MMG, $550 direct from Magnepan. This speaker was the overwhelming choice in the category.

The Top Rated Speaker for under $2,000:

Magnepan 1.6 for $1,800 from Magnepan Dealers. Interesting note: The Magnepan MG12 and MMG (the top under $1,000) also made the top 5 highest rated under $2,000 pair of speakers. The Dali Ikon 6 and Martin Logan Source also made the top 5 rated. (Maggie 1.6, Martin Logan Source, Maggie MG12, Dali Ikon 6, and Maggie MMG in that order for those wondering).

The Top Rated Speaker for under $5,000:

Martin Logan Vantage, Magnepan 3.6, Anthony Gallo Reference 3.1, Def Tech Mythos Super Towers, Martin Logan Purity

The Top Rated Speaker for under $10,000:

Martin Logan Vantage, Martin Logan Summit (technically more expensive than the Vantage)

Top Rated Speaker for over $10,000:

Martin Logan Vantage

We're not done with our testing. We're about 60% finished, so the numbers could change. We got most of our gear from donations, the rest we bought or have on loan (the 7 speakers we have over $10,000 for instance). We had to skip certain models and generally chose the most expensive of a line or 2nd highest in a line.

Speaker Brands we tested: Bose, Klipsch, Polk Audio, Martin Logan, Magnepan, Def Tech, PSB, Mirage, Wilson, Anthony Gallo, Sony, KEF, Canton, and Krell just to mention some of them. Like I said we tried to test as many cheap speakers as possible and removed complete lines that didn't test well. So take Bose for an example. They tested horribly, and by the time we eliminated their 101s and 201s, we only tested one of the Life series. Polk Audio, Klipsch, and many others suffered the same fate, with us just picking one of their line from a price range.

I'll let everyone here know when we're finished, because we'll be selling the gear.

Edited December 9, 2008 by deepak

[.BB]

This could be a very good potential guide when they finish the testing. Interesting read.

[postjack]

maggies! now I really want to hear some.

[spritzer]

It's official, planars rule the world. Not that we didn't know it here with more electrostatics and isodynamics then anywhere else.

[Hopstretch]

Yeah, only one "liked" speaker in any category with a dome tweeter. Oof.

[luvdunhill]

geez, if they'd only do one of these for say wine... Franzia FTW!

[laxx]

Guess I'll go listen to some MMG's to see what the fuss is about.

[Sherwood]

They are delicious, the maggies.

I prefer my PSB's, though. Interesting.

[Duggeh]

A set of MMGs is now vying for the biggest hole in my burning pocket.

[Sherwood]

Hopefully word of this study spreads far and wide, and I can unload my MG12s on Craigslist for a good price before Christmas.

[Duggeh]

Does anyone know what the deal is for ordering an MMG in Europe? Magnepans website isn't exactly brimming with information. Maybe its because over here they'll be

[Grahame]

Maybe its because over here they'll be

[Voltron]

Yeah, only one "liked" speaker in any category with a dome tweeter. Oof.

And I have Gallo Ref. 3s for sale right now! FOTM anyone?

[Duggeh]

Not to mention VAT and/or import duty!

At which point you start hearing voices in you head saying the words

Quad, ESL and Dew eet!

Need

[spritzer]

Does anyone know what the deal is for ordering an MMG in Europe? Magnepans website isn't exactly brimming with information. Maybe its because over here they'll be

[Duggeh]

The trick is in knowing how much the postage would be beforehand, which isn't easy, and it will be pretty monstrous I'd wager. With the pound tanked 30% against the USD $550 is now

[Sherwood]

Doug, I'll make you a deal for my MG12s, if you're interested. I'll bullshit customs however you please.

[Icarium]

And I have Gallo Ref. 3s for sale right now! FOTM anyone?

Yeah that reinforces my wish to get the Ref 3s.. so tempting...

[Hopstretch]

And I have Gallo Ref. 3s for sale right now! FOTM anyone?

They're some sort of omni, aren't they? I was thinking of the Def Techs as the only conventional dome + cone design.

I also thought the planar mafia was down on the latest generation of Martin Logans as generally crappy?

[Dusty Chalk]

Gallos are win.

And ha-ha to all you Martin-Logan haters.

[Dusty Chalk]

They're some sort of omni, aren't they?

More like hemi.

[Sherwood]

It would be nice had they been able to get a pair of 20.1s in the test to compete with the vantage and the like.

[Duggeh]

Doug, I'll make you a deal for my MG12s, if you're interested. I'll bullshit customs however you please.

I'd have to sell the quads for a chance of that, and at that money I could probably get ESLs in fair repair. I'm just spouting cos of my positive paypal balance. Thanks for the offer though.

[Sherwood]

Thanks for the offer though.

Not a problem at all. They're great speakers, and I'm sure they'll find a home. I'm always game to cheat customs, for anything you Europeans may need.

[Torpedo]

So the conclusion of this is that untrained people under blind conditions prefer the sound of Magnepan speakers to listen to 5 tracks of music for a while. And probably the more tired they were for the high number of tests going on in a single day, the more they liked them