November 2018

How To Do A Comparative Listening Test
Article By Cookie Marenco Of Blue Coast Records And Music

  Setting up a listening test to compare various types of files is not easy. You need a base of equipment and a trusted set of files to start. Set aside several hours and have a few people involved. We rarely do a test with fewer than three people. Here we'll explore some of the basics steps to compare files.

Keep in mind, the objective is to determine differences in formats, cables, devices, conversion tools, etc. You don't need to like the music. The question is "do you hear a difference". That difference might be insignificant, but if it repeatedly appears, then there is a difference. Differences may be covered and only appear in certain sections of the music, so it's important to do the test with different music and repeat the test. Again, if there are repeatable differences, regardless of how small or how few times it occurs, then that constitutes a difference.

Does it matter that these differences exist? Only to the listener. Our job is to determine either there is a difference or there is not.

Here's What You Need:
The Gear
One person to be an operator of the device. We'll call this person the "Operator"

Two people to do the listening test. We'll call them the "Listeners"

Optional – a separate room where the devices are located and where playback occurs. If you don't have this, the second choice is to setup the playback device in a position where the Listeners can not see the Operator.

Pencils and paper to track answers (computers don't work for us because the typing noise and chances for movement from Listeners)

A high end speaker system capable of full frequency and dynamic results. Professional audio headphones (Beyerdynamic, Sony, and Sennheiser) work well. Avoid consumer headphones due to manipulation of phase response.

We suggest solid-state over vacuum tube amplification for testing (has to do with dynamic and frequency response, not whether the sound is pleasing -- we're looking for ways to hear very small differences).

Chairs positioned optimally for speaker response and where Listeners can hold their position without movement.

The Music
Music clips recorded at the highest fidelity possible – we suggest DSD256 recordings from a live performance.

Acoustic music works best for testing because electronically produced music often has frequency range cutoffs from sampling or limitations of the electronics.

Look for music that is not over compressed and has full dynamic range. If you can find music that is "pre mastering" with full dynamic response, wonderful. Pop, rock and other music are often over-compressed for commercial release.

Piano can work well; orchestral is wonderful, acoustic jazz, folk and singer-songwriter, world music, etc. You don't have to like the music, just need something with full dynamics and frequency.

Stereo is a must for these tests – large hall sizes with reverb helps. Reverb trails are one of the easiest places to spot differences.

You'll only be listening to 15 to 30 seconds for the test. Start with 3 pieces of music or one with variety. You'll want to repeat the 15 to 30 second test several times.

Stillness And Hand Signals
Remaining still is imperative when testing. Even in headphones, the slight twist of your head can alter the results. It is imperative that the Listeners maintain the same position through the test. For this reason, we suggest creating a set of hand signals to tell the Operator to repeat the music clip, change to next clip, and then go back to first clip. Reminder, make sure to eliminate any movement that might alert the Listeners to which file is being played. Sometimes comparing files has an audible click associated. These can be hard to avoid. Do your best. 

When changing devices or cables, there is a longer period of time involved between playbacks. These tests are best to setup where the Listeners cannot see the devices being switched. And it's important that the Operator add time to mask when doing a series of playback that might be AABABBA for example.

Compare Two Things At A Time – A/B Testing
We sometimes compare three items at once for a longer test but we suggest two at a time. We use the A/B technique. A being one clip and B being the comparing clip. The Operator must keep track of the sequence of playback. It might look like ABBAABBBAA. Never repeat the same sequence.

As you get more sophisticated in your listening you may want to alter this test. For instance, when first listening to a clip you might find a spot where a reverb trail extends through a long silence or a high piano note appears in a specific area of the stereo image. Listen to that spot again – maybe as little as five seconds. Listen to the B clip in the same spot. Does the reverb trail die before the silence ends? Does the high note of the piano move to an outer part of the image? These are great giveaways to something changing.

Headphones Versus Speakers And What To Listen For
Headphone are often easier to spot these image changes right away and is the quickest way to hear differences in formats. Speakers might be harder but could offer a different experience of the sonic differences.

Here Are A Few Things We Listen For To Dear Differences...
Dynamic response... does the cymbal hit or guitar note or piano note die off at the same speed for each clip. We find that this is one of the biggest giveaways. Lower sampling rates tend to be "thicker" almost more compressed. Some people enjoy this sound. Again, this is not about what is better only to determine there is a difference and what that is.

Frequency response... do each clip reach the lowest lows and highest highs?

Room changes... does the reverb of the room cut off at different times (shorter or longer) when a silent passage occurs.

Wideness of the stereo image... especially helpful to hear differences in formats. Listen for the wideness of an orchestral passage or other acoustic ensemble. Piano can work well for this too. As sampling rate increase, generally we've found wider imaging and more clarity with the widening.

Phasing issues... this is tricky to hear, but when comparing first and second generations of digital transfers at the DSD256 range, comparing FLAC conversion levels, etc, we have found that using an orchestral or multiple mic recording of live jazz can expose shifting phase issue (seemingly instruments move differently in various clips).

Noise floor changes... Funny enough, listening to the noise at the beginning, end or silent section of a music clip can offer an easy way to spot differences.

In Conclusion....
I'm sure I'm forgetting a few steps and welcome questions to update this article. Contact support@dsd-guide.com. This is my first draft of our processes.

I'm sure there are many methods for comparison tests. If you have suggestions for other tests, please submit them.

Remember this is not about putting down anyone's listening style or beliefs. These are tests for us to determine "is there a difference". Listening to noises may not be your cup of tea, but if there is a difference, then there is a difference. That's all we're looking for.

When we sell our files at Blue Coast Music we want to stand behind what we sell with our ears. It's not a musical judgment. It is a technical judgment. If you prefer the sound of lower sampling rates because they sound "thicker" – wonderful! If you prefer the wider image with full frequency and dynamic response that DSD offers – wonderful!

Thank you for reading,

Cookie Marenco
Blue Coast Records And Music