Enjoy the Music.com
The Absolute Sound
August 2008

The Blind (Mis-) Leading the Blind

  The latest in this long sad history is a double-blind test that, the authors conclude, demonstrates that 44.1kHz/16-bit digital audio is indistinguishable from high-resolution digital. Note the word "indistinguishable." The authors aren't saying that high-res digital might sound a little different from Red Book CD but is no better. Or that high-res digital is only slightly better and not worth the additional cost. Rather, they reach the startling conclusion that CD-quality audio sounds exactly the same as 96kHz/24-bit PCM and DSD, the encoding scheme used in SACD. That is, under double-blind test conditions, 60 expert listeners over 554 trials couldn't hear any differences between CD, SACD, and 96/24. The study was published in the September, 2007 Journal of the Audio Engineering Society.

Such tests are an indictment of blind listening in general because of the patently absurd conclusions to which they lead. A notable example is the blind listening test conducted by Stereo Review that concluded that a pair of Mark Levinson monoblocks, an output-transformerless tubed amplifier, and a $220 Pioneer receiver were sonically identical. ("Do All Amplifiers Sound the Same?" published in the January, 1987 issue.)

Most such tests, including this new CD vs. high-res comparison, are performed not by disinterested experimenters but by partisan hacks bent on discrediting audiophiles. But blind listening tests lead to the wrong conclusions even when the experimenters' motives are pure. A good example is the listening tests conducted by Swedish Radio to decide whether one of the low-bit-rate codecs under consideration by the European Broadcast Union was good enough to replace FM broadcasting in Europe.

Swedish Radio developed an elaborate listening methodology called "double-blind, triple-stimulus, hidden-reference." A "subject" (listener) would hear three "objects" (musical presentations); presentation A was always the unprocessed signal, with the listener required to identify if presentation B or C had been processed through the codec.

The test involved 60 "expert" listeners spanning 20,000 evaluations over a period of two years. Swedish Radio announced in 1991 that it had narrowed the field to two codecs, and that "both codecs have now reached a level of performance where they fulfill the EBU requirements for a distribution codec." In other words, the codecs were good enough to replace analog FM broadcasts in Europe.

After announcing its decision, Swedish Radio sent a tape of music processed by the selected codec to the late Bart Locanthi, an expert in digital audio and chairman of an ad hoc committee formed to independently evaluate low-bit-rate codecs. Using the same non-blind observational listening techniques that audiophiles routinely use to evaluate sound quality, Locanthi instantly identified an artifact of the codec. After Locanthi informed Swedish Radio of the artifact (an idle tone at 1.5kHz), listeners at Swedish Radio also instantly heard the distortion.

How is it possible that a single listener, using non-blind observational listening techniques, was able to discover in less than ten minutes a distortion that escaped the scrutiny of 60 expert listeners, 20,000 trials conducted over a two-year period, and elaborate "double-blind, triple-stimulus, hidden-reference" methodology? The answer is that blind listening tests fundamentally distort the listening process.

As exemplified by yet another reader letter published in this issue, many people naively assume that blind listening tests are somehow more rigorous and honest than the "single-presentation" observational listening protocols practiced in product reviewing that the undeniable value of blind studies of new drugs, for example, automatically confers utility on blind listening tests.

I've thought quite a bit about this subject and written what I hope is a reasoned analysis of why blind listening tests are flawed. This analysis is part of a larger statement on critical listening in general, which I presented in a paper to the Audio Engineering Society entitled "The Role of Critical Listening in Evaluating Audio Equipment Quality." A revised and updated version of the paper is now available on AVguide.com.

I invite readers to comment on the paper, and discuss blind listening tests, on the Forum of AVguide.com. We've created a new section called "Evaluation, Measurement, Testing, and Perception: Discussion of how to evaluate products, how to report on that evaluation, and link that evaluation to real experience/value." I look forward to reading your ideas.

 


 

     
Quick Links


Audiophile Review Magazine
High-End Audio Equipment Reviews

Equipment Review Archives
Turntables, Cartridges, Etc
Digital Source
Do It Yourself (DIY)
Preamplifiers
Amplifiers
Cables, Wires, Etc
Loudspeakers/ Monitors
Headphones, IEMs, Tweaks, Etc

Superior Audio Archives
Ultra High-End Audio Reviews

Videos
Enjoy the Music.TV

Columns
Editorials By Tom Lyle
Viewpoint By Roger Skoff
Viewpoint By Steven R. Rochlin
Various Think Pieces
Manufacturer Articles


Show Reports
Rocky Mountain Audio Fest RMAF 2018
KL International AV 2018 Show Report
High End Munich 2018 Show Report
AXPONA 2018 Show Report
CanJam SoCal 2018 Show Report
CanJam NYC 2018 Show Report
Capital Audiofest CAF 2017 CanMania
TAVES 2017 Toronto Show Report
CanJam 2017 Denver RMAF
LAAS 2017 Show Report
Click here for previous shows.

Other
Audiophile Contests
Cool Free Stuff For You
Tweaks For Your System
Vinyl Logos For LP Lovers
Lust Pages Visual Beauty

Resources And Information
Music Definitions
Hi-Fi Definitions
High-End Audio Manufacture Links

 


Daily Industry News

High-End Audio News & Information

Partner Magazines
Australian Hi-Fi Magazine
NOVO (CANADA HiFi)
hi-fi+ Magazine
HIFICRITIC
HiFi Media
Hi-Fi World
Sound Practices
VALVE Magazine

For The Press & Industry
About Us
Press Releases
Official Site Graphics

Contests & Join Our Mailing List

Our free newsletter for monthly updates & enter our contests!

Our Social Media & Video Channel
      

 

 

       

Home  |  Sitemap  |  Industry News  |  Equipment / Music Reviews  |  Press Releases  |  About Us  |  Contact Us

 

All contents copyright  1995 - 2018  Enjoy the Music.com
May not be copied or reproduced without permission.  All rights reserved.