Am I the only one that feels the supposed "Blind Mastering Taste Test" is a farcical and totally pointless gimmick in its current form?

I understand what they are trying to do, but with anything like this, unless you level match all the files, including the premaster (a version where all highest peaks match, and then a second version with all RMS matched) you can not fairly assess the results.  It has been proven in studies that even trained engineers will say the louder file is better over 70% of the time and the FM curve theory has been studied in depth to explain why this anomaly occurs.

By offering all the files in both matched forms (peak and RMS) we could fairly have a blind test, otherwise, the loudest will win.

I'm probably willing to bet one of the LANDR versions is the loudest.

SimpleM
I understand what they are trying to do, but with anything like this, unless you level match all the files, including the premaster (a version where all highest peaks match, and then a second version with all RMS matched) you can not fairly assess the results.  It has been proven in studies that even trained engineers will say the louder file is better over 70% of the time and the FM curve theory has been studied in depth to explain why this anomaly occurs.

Level-matching would make sense when comparing mixes, but mastering has to work in conjunction with a defined and unalterable amount of headroom. Although I haven't measured the waveforms in the "taste test," it appears they peak at the same level so I assume the peak levels are matched. To match RMS levels, those files with higher RMS levels would have to be brought down in overall level (which is an unrealistic comparison because you're giving up headroom, which you would do only in a level-matching context for an album) or ones with lower RMS levels would have to be brought up to the point where the peaks would exceed the available headroom, which is equally unrealistic because it would result in distortion.
 
Because headroom defines maximum levels, it's up to the listener to adjust the volume to their satisfaction. For example if you put on a CD mastered in the 80s, because it likely had more dynamics you'll almost certainly need to increase the hi-fi system's level control to have the same apparent volume as CDs mastered recently.
 
Like it or not, one of the tasks of modern-day mastering is to find the sweet spot tradeoff between dynamics and volume, and that is one of the elements to consider when judging the different masters. A track with more dynamics will indeed sound softer, but that's why hi-fi systems have volume controls. What would be an unfair test of mastering would be to level-match everything. I would, however, level-match different mixes to determine which mix I liked best, knowing that the average levels would change during the mastering process.
 
 

Oh snap, Anderton does it again. Dispels a myth.

Level matching is something we will look at for next time. This is meant to be fun and based on the results, completely subjective.

I think most of us who frequent a forum like this and have experience working with audio know how to adjust our volume knobs to reasonably compensate for changes in loudness.  That certainly isn't a scientific level of precision but polls on the internet usually aren't expected to be.  The one I liked best was one of the louder ones but I picked it because of specific qualities unrelated to loudness that it brought out to my ears on my speakers that the quieter ones did not contain.  I'd guess that if anything skews the poll, it'll be people picking the quieter tracks rather than trusting their own ears in some attempt to fight the loudness wars.  

Easy enough to record them and do what you like.

I chose 2 specifically because the louder tracks lost the feel of the original track. If you write a crescendo into a song, how would you feel if it was completely crushed out of the mix in the press? Also, turning up the mix brings back what you think might have been missed as opposed to the louder mixes. This is what the loudness button was for on your stereo. The eq curve changes with amplitude. That is why level matching is essential.

Anderton
 
 
Like it or not, one of the tasks of modern-day mastering is to find the sweet spot tradeoff between dynamics and volume, and that is one of the elements to consider when judging the different masters. A track with more dynamics will indeed sound softer, but that's why hi-fi systems have volume controls. What would be an unfair test of mastering would be to level-match everything. I would, however, level-match different mixes to determine which mix I liked best, knowing that the average levels would change during the mastering process.
 
 

I feel what you are saying Craig, part of mastering is the final loudness, but still, It has been proven in truly blind tests over and over again the "louder" is almost universally perceived as better to the mind when comparing sounds side by side, even otherwise identical ones.  As little as 1 or 2 db is all it takes so unless you are sure of how much exact difference there is, turning up and down a master fader as one reply suggested is not really the answer either.  Now of course, there comes a limit to this where the squashing might make the more trained ear start to turn the other way but none of the examples are necessarily to those extremes.

One of the earliest procedures I was taught (by more than one accomplished mastering engineer) was to keep the unprocessed pre-master handy and keep it simply level matched to what you are doing as you work on it so that you can discern whether the changes you are making are actually moving in the right direction or you are just getting the "betters" because you are making something louder.

I am in no way a big time mastering engineer, nor do I have all the bells and whistles or cavernous mastering suite the big boys have but I have been paid to do mastering a lot over the past 20 years and my customers have returned for more and brought others to me based on the results.  I got those results and reputation from following the level-matching method and therefore I still feel I am right that if this is to be taken as any kind of serious comparison tool as to the capabilities of Sonar's tools or LANDR, then the playing field should be one that allows true evaluation and truly makes the test blind.  Sure, I could record each file from Soundcloud and level match myself, but that would have been best done before the files were posted.

I know Andrew says it was done just for fun and on the fly, but I am sure if the results show the Sonar tool or LANDR versions preferred, it would by talked up on these forums at the least. 

I have staked a lot of reputation on Cake/Sonar for 20+ years and had a lot of potential clients go elsewhere because I told them used Sonar instead of PT.  If the results were to show people prefer the ones done with Sonar or even the LANDR version, I think it is important that it be because they truly preferred the differences in tone, compression and summing, not simply because they heard the "better" in the loudness. 

Based on your original reply, I feel there is probably a good chance we may have to respectfully agree to disagree on this one but that is ok.

SimpleM
One of the earliest procedures I was taught (by more than one accomplished mastering engineer) was to keep the unprocessed pre-master handy and keep it simply level matched to what you are doing as you work on it so that you can discern whether the changes you are making are actually moving in the right direction or you are just getting the "betters" because you are making something louder.

 
I agree with this 1000%, and it was a Great Day when Wavelab introduced the option to balance the levels of the mastered and unmastered version for instant comparison. But think about it for a second: in this case you're not comparing the master to a master. You're comparing the master to the mix. They're different animals, and one part of mastering is to find that sweet spot between satisfying dynamics and having the master jump out of the speaker. 
 
I'll put it another way. I've had some clients who wanted as loud a master as possible. I believe that a song needs to retain some degree of dynamics to remain emotionally compelling, so I do what they want, and then I do what I want and let them choose. The squashed master will sound louder; mine will sound a bit softer. But almost all clients choose the one with more dynamics because they're comparing master to master and then decide they are willing to trade off less level for more emotional impact. If I balanced the levels, they would not be able to make an informed choice. They need to know they are trading off lower levels for more emotion.
 
I agree completely that if you're comparing masters to determine how faithfully they translate the mix into a final stereo track, then you'd want the levels matched. But if you're comparing one master to another master, how loud you can get while still preserving dynamics and having a satisfying emotional experience is one element of a basis of comparison. It's clear from the responses to this "taste test" that hardly anyone prefers the loudest master, which simply means to me that if someone is using SONAR, they probably have educated enough ears to listen past just the impact of the level, and judge the master on its other qualities as well.
 
The one assurance I would want with this particular kind of "taste test" would be that all the examples were normalized to the same peak level, because that would mean each master took advantage of all available headroom. This would allow for a fairer master-to-master comparison.

Anderton
 
 
The one assurance I would want with this particular kind of "taste test" would be that all the examples were normalized to the same peak level, because that would mean each master took advantage of all available headroom. This would allow for a fairer master-to-master comparison.

I think I would have been satisfied with the taste test much better if at least peak matching had been done.  RMS matching is nice to check transients/dynamics from mix to master and it is a luxury to be able to do (I used Soundforge vs Wavelab back in the day and I just normalized peaks and or RMS to match, not sure SF would do it automatically), but peak matching is almost a must to at least not have the potential loudness of an overall example artificially low.

To be honest, I did not listen to the entirety of any of the tracks, just the first bit till the percussion came in stronger, at that point is when I felt it would be fruitless to compare, even as experienced as I am because the levels were so different that I did not trust my own preferences to not be biased to louder.