Cakewalk's "Blind" mastering test.

Anderton
 
 
Like it or not, one of the tasks of modern-day mastering is to find the sweet spot tradeoff between dynamics and volume, and that is one of the elements to consider when judging the different masters. A track with more dynamics will indeed sound softer, but that's why hi-fi systems have volume controls. What would be an unfair test of mastering would be to level-match everything. I would, however, level-match different mixes to determine which mix I liked best, knowing that the average levels would change during the mastering process.
 
 

I feel what you are saying Craig, part of mastering is the final loudness, but still, It has been proven in truly blind tests over and over again the "louder" is almost universally perceived as better to the mind when comparing sounds side by side, even otherwise identical ones.  As little as 1 or 2 db is all it takes so unless you are sure of how much exact difference there is, turning up and down a master fader as one reply suggested is not really the answer either.  Now of course, there comes a limit to this where the squashing might make the more trained ear start to turn the other way but none of the examples are necessarily to those extremes.

One of the earliest procedures I was taught (by more than one accomplished mastering engineer) was to keep the unprocessed pre-master handy and keep it simply level matched to what you are doing as you work on it so that you can discern whether the changes you are making are actually moving in the right direction or you are just getting the "betters" because you are making something louder.

I am in no way a big time mastering engineer, nor do I have all the bells and whistles or cavernous mastering suite the big boys have but I have been paid to do mastering a lot over the past 20 years and my customers have returned for more and brought others to me based on the results.  I got those results and reputation from following the level-matching method and therefore I still feel I am right that if this is to be taken as any kind of serious comparison tool as to the capabilities of Sonar's tools or LANDR, then the playing field should be one that allows true evaluation and truly makes the test blind.  Sure, I could record each file from Soundcloud and level match myself, but that would have been best done before the files were posted.

I know Andrew says it was done just for fun and on the fly, but I am sure if the results show the Sonar tool or LANDR versions preferred, it would by talked up on these forums at the least. 

I have staked a lot of reputation on Cake/Sonar for 20+ years and had a lot of potential clients go elsewhere because I told them used Sonar instead of PT.  If the results were to show people prefer the ones done with Sonar or even the LANDR version, I think it is important that it be because they truly preferred the differences in tone, compression and summing, not simply because they heard the "better" in the loudness. 

Based on your original reply, I feel there is probably a good chance we may have to respectfully agree to disagree on this one but that is ok.