Everything that matters...

 "Not everything that matters gets measured and not everything that gets measured matters"

 

Nope. It wasn't him expressing it.  Consider us kindred spirits. ;) Alberts original version reads

"Not everything that can be counted counts and not everything 
that counts can be counted"   

What's the point?

Since the beginning of audiophile times we all face a major ongoing dispute in the audiophile audio world. There's one group, let's simply call them "Wannabe-Scientists", claiming that if an audio device measures great, showing qualities (common specs) beyond the audible threshold, such a device is simply not to be questioned regarding its sound qualities and performance. Basically saying: If you can't measure it, it can't exist.

And then there is the other group. Open-minded audio folks who simply listen and judge. Let's call them GoldenEars. That's the term Wannabe-Scientists usually use to belittle members of this group. People such as audio-gear manufacturers and designers, audio-measurements gear manufacturers,  professional and hobbyist reviewers, engineers,  die-hard audiophiles, audio enthusiasts and even our beloved wifes/partners. All these folks have one thing in common, they can tell A from B apart by simply using their ears. The vast majority of this group would confirm that not one device sounds the same, no matter how well it measures.

By writing this article I intend to (1) point out the hypocrisy and ignorance of these Wannabe-Scientists and (2) encourage fellow people of the GoldenEar group not to let you get intimidated by these Wannabe-Scientists. Simply trust their own ears.

What triggered this article ???

The situation around Wannabe-Scientists and GoldenEars is not new. Both groups coexist side-by-side

since decades. I do know many manufacturers who simply consider it the most basic knowledge that measurements AND listening tests are a MUST to design audio equipment. I do also know quite a number of reviewers who would also confirm it - specs AND listening tests are a MUST.  Actually the vast majority of interested audio folks will be confirming it. Most of these folks run the hobby for several decades based on reading (and understanding) the specs and listening to numerous systems and devices over the years. I actually consider the GoldenEar audio folks the main reason why our audio systems still evolve and never sounded better than today. 

Now. All this is not new. But, why are these "Wannabe-Scientists" so ignorant?  It almost feels they follow some kind of ideology.

In 2021 I came across a Youtube video from 2015 called  "RMAF15: What The Specs Don’t Tell You… And Why". It's a presentation on USAs largest audio show by Jonathan Novick from Audio Precision. Audio Precision is manufacturing the best audio measurement gear money can buy. Basically all serious audio manufacturers and professional reviewers will have an AP product at hand for the work to be done. (You can watch the video later on)

What is the key message ????

To make a long story short and simple. Jonathan in, as a part of his presentation, clearly outlines that standard audio measurements as provided by Audio Precisions own, even its best-in-class measurement equipment the APx555@$28000, would not sufficiently describe the audible sound of an audio device or system. 

Let that sink in for a while!

To hear that message from such a party. That's gotta be a major bummer for our Wannabe-Scientists out there. 

Let's rephrase a bit what's been said. I try to get to the essence of the story.

Widely used measurements, such as 

• total-harmonic-distortion and noise (THD+N), 
• signal-to-noise and distortion ratio (SINAD), 
• dynamic range,
• linearity
• jitter, 
• asf.,  

using specific test-signals (incl. multi-tone signals), would not sufficiently mirror the sound signature of a highly complex music signal on the same system. 

According to Audio Precision, there are simply no measurements that would properly reflect the underlying real music complexity. ( And I'd like to add to that - the real world audio system complexity.)

However. All these audio measurements as provided by their devices would sufficiently prove that all the basics are right. These kind of measurements are useful and usually sufficient to make sure that production quality gets maintained. 

Great. 

Great to get such a message from a manufacturer who sets the baseline for large parts of the audio industry and the audio market, the audio sellers and users, the reviewers, you name it.

What now???

The Audio Precision message proves the GoldenEar fraction right. Ok. We always knew we were right. And that message puts these Wannabe Scientists to shame. It basically confirms to me that beside running a standard set of measurements, it'll require listening tests to get the full picture about a devices overall performance. 

It's obviously a known fact in the industry that all measurements that'd be required to define to 100% the overall performance of an audio device or system are not identified to date.

How can we cope with this situation. The message implies that we need 

1. standard measurements AND
2. listening tests

to describe the sound signature of a system.  That's the situation we're actually in - since decades.

I am not aware of a scientific study or paper that would analyze the topic. No idea why science never really "properly" looked into this particular subject. 

Talking about science (or perhaps better Wannabe Scientists). There's a popular place called Audio Science Review founded by Amir Majidimehr, an electrical engineer who used to work for Microsoft. 

I really like this place. Not that many places like that on the net. Its name implies scientific reviews to begin with. How can a serious audio lover stay away from such a promise ??? 

As expected, just by using highest end measurement gear the site founder claims to provide scientific

reviews. Hmmh. And no. Listening tests are not scientific according to that site. Listening tests are subjective.  As a consequence listening tests are simply left out to make things "scientifically" easier.

Amir also judges devices, he even recommends devices based on just measurements. Hmmh. 

And not to get anybody confused out there, people who start talking about listening tests will get a pretty tough time over there. I know, it sounds awful. It actually IMO is.

Why than liking this place? Right. I said that. I like ASR for its measurements of all kind of audio gear. Amir uses the top of the line measurement gear from Audio Precision. A statement! Amir IMO produces reliable measurements.  Presents them nicely and also presents list rankings. For me this is simply a good starting point when looking into new gear. 

What you can also see is that manufacturers respond rather quickly to his findings. I'd go that far that a certain number manufacturers seem to be using the ASR review charts and rankings to define their design goals for beating the competition. Not a bad thing. We as users love it.

But that has another weird side effect. If you look at the ASR rankings you'll notice that the number of devices on top of the list is continuously growing.  You can consider is a saturation effect. It simply can't get any better. Because the measurement gear limitations simply prohibit from showing even better results. According to ASR a lot of that gear under test runs nowadays in areas far beyond audible thresholds. One could wonder why running the tests at all???

Amir's place is also a nice spot to get in touch with Chinese audio gear manufacturers, such as SMSL, Topping, Gustard, you name it. It's one of the very few places on the net where you can get in touch with them inside the forum. 

 

There's a huge problem with this place - Audio Science Review. It's name already refers to the problem. The site name implies a scientific approach.  What's presented are basic audio measurements.

And people who talk about listening experiences are attacked, ignored or even expelled. 

We know by now, ignoring listening experiences or tests - which are a must - can't lead to reliable reviews or conclusions about a device or system. Such an approach would never fulfill the most basic scientific demands of being called scientific. 

People need to be aware that the reviews and even recommendations over at Audio Science Review

can be quite misleading, because listening tests are not part of the story.

Don't get me wrong. This is no Audio Science Review bashing. Getting your own baseline right by looking at the ASR measurements is absolutely OK.  If you simply ignore Amirs conclusions, recommendations, demystifying videos asf and you trust your own ears - it'll do. 

One more thing that's bugging me. If you've watched the Audio Precision video from earlier, you might have noticed the ASR founder Amir also being present at that event. He basically was being told by his own measurement gear provider that the measurements he creates are not sufficient to explain what we all actually hear and experience. 

Then, why is he still pushing his "scientific" storyline!?!?  He didn't listen, forgot about it or he actually didn't want to listen - didn't want to hear it. Not good.

It's not just ASR though. You'll find people all over the net trying to shut you up about listening impressions.  

Don't let you intimidate you! Trust your own ears...  ...and share it. You'll be surprised how many people

will hear the same good or bad thing you experienced by yourself.

Back to that Audio Precision RMAF presentation which was called "What specs don't tell you...".

There seem to be some ideas floating around about what's missing and where to look for it. But even now in 2023 - 8 years later - I haven't seen any progress on clearly explaining what's missing. And at that point I do hope that real "science" steps in at one point.

 

Bottom line

"Not everything that matters gets measured and not everything that gets measured matters". 

Buying your next audio device simply based on measurements or pseudo-scientific reviews might wrong-foot you.

Make sure you also run listening tests on your own and make sure you'll be able to return the device under test. 

Your listening skills will improve over time. The more you hear the better you're able to judge a device. 

You learn what to "hear" for. It's difficult in the beginning to judge what's right or what's wrong.

Audio is about finding the best compromise that meets your personal taste. 

Sharing your findings with others will grow your confidence. Been there done that. ;) 

Perhaps one day we see a sincere scientific approach that explains properly the gap of what we measure today and what we actually hear. Basically describing the audio experience to almost 100%. I am sure training an AI for the task would be an interesting development.

The big Q. Does the industry really want a close to 100% definition of sound quality??  Do they want a set of measurements that explains close to 100% of what we hear !?!?

All differentiation would be gone in no time. And a huge industry would face a major problem.

For us audio folks times would be great. Audio gear would become a commodity @ low prices and excellent and predictable performance. 

We finally would be able to listen to music! Nah. That'd be boring. We'd get our hobby eliminated.

Enjoy.