how to set up a fair listening comparative test for converters/interfaces (?)

Perhaps despite popular opinion, doing a "sort of" objective test is useless. Our brains are wired to unconsciously influence our subjective experiences and there's nothing we can do about it and it doesn't matter how much experience we have or how smart we think we are. Therefore we need to be very careful to eliminate everything except what we are actually hearing with our ears (not what we think we are hearing). 
 
Just eliminating some things is a waste of time as it produces unreliable results, so you either do it right or don't bother unless you're making a subjective evaluation to begin with - i.e. "Which mic do I prefer?" (a preference doesn't matter too much if it's influenced by expectation bias), not "Can I hear a difference?" (an objective fact that requires very careful testing to eliminate all subjective bias because otherwise it isn't objective).
 
Proper objective comparison requires:
 
Identical source material.
Careful level matching to within ~.1dB SPL.
Double blind or completely blind switching.
No switching artifacts when switching sources.
Truly random switching.
Reasonable source material and listening environment/levels.
Statistically significant differences to minimize the possibility of guessing.
 
If you can't achieve all of these things, you're fooling yourself if you think you're getting more objective results. Science doesn't require all these careful controls just for jollies - they're required because otherwise the results are simply unreliable. It's actually relatively easy when comparing digital audio files/formats - just get and learn to use some ABX SW, but unfortunately it's often very difficult when trying to compare hardware.
 
You might find it easier to measure the differences using test tones or something like RMAA to see exactly what the differences are:
 
http://audio.rightmark.org/products/rmaa.shtml
 
 
EDIT: The short answer is either do it right or don't bother. Anything else doesn't make any sense if you actually want to learn anything about reality. 

Rob[atSound-Rehab]
... I just would want to see if I can hear anything different between a good interface of 10yr old design and a recent design ... but I don't want to get overly scientific about it ...

I agree with drewfx's assessment... That being said, I think if you have decent hearing, you will likely be able to hear a difference between a much older (say, 10+ years) interface and a new one without much difficulty.
 
I have heard a lot of older Delta 1010s and Layla's in my day, and when the owners upgraded to something new (designed in the last few years) the difference has always been easily distinguishable to everyone around.  We've chalked it up largely to the newer converters.

JonD
I agree with drewfx's assessment... That being said, I think if you have decent hearing, you will likely be able to hear a difference between a much older (say, 10+ years) interface and a new one without much difficulty.
 
I have heard a lot of older Delta 1010s and Layla's in my day, and when the owners upgraded to something new (designed in the last few years) the difference has always been easily distinguishable to everyone around.  We've chalked it up largely to the newer converters.

I finally got around to try my totally subjective listening test using  ...
 
.. analog input signal (digital piano triggered by MIDI) leveled to some certain input level (not very hot, maybe at -20 dFS), record in Sonar ensuring consistent pre-amp gain settings
... do A/B comparison of bounced files
 
Since both interfaces (MOTU 1248 AVB, Roland Octa-Capture) are only 4 years apart in their releases the differences are not that obvious, but they are there when using A/B comparison with high end head phones / decent monitors ... the biggest difference you get in the noise floor, which is slightly audible on the Octa-Capture before you hit the first note (but is not there on the MOTU). This does indeed impact the sustain periods of e.g. long held piano chords; while the MOTU stays clean, the Octa-Capture gets a little swooshy before the signal fades out ... also the Octa-Capture (despite ensure identical peak limits) sounded a bit louder and denser, as if there was a little bit of compression going on ... interestingly, there's nothing to see on the analyzers that I threw at it ... so, either the ear is still superior to the eye, or I'm totally losing it ;-)
 
 
 
 

JonD

Rob[atSound-Rehab]
 
I agree with drewfx's assessment... That being said, I think if you have decent hearing, you will likely be able to hear a difference between a much older (say, 10+ years) interface and a new one without much difficulty. 
 

 
I just did a comparison a week ago.
Between a Panasonic DA7 (circa 98-2001) and an RME AIO (I got in 2011).
My speakers have 2 sets up inputs so I ran each interface outs into one set of speaker inputs. So, the A/B was immediate. I sure couldn't hear a difference. Quite surprising actually. With all the hoopla surrounding RME, I thought I would hear "Something".. Of course, I guess my hearing could be at fault. I don't really know.
 
I don't actually know the manufacturer's date of the AIO (maybe someone here knows). But, I would think anything after say 2008, convertors really started to get their **** together. But have they gotten much better since then?
 
I'm wondering where (in time) the quality curve peaks out? Or, is it still rising? I mean are 2016 convertors that much better than say 2010 convertors?
 
Sorry. I don't know why my post got 'quoted' (or how to fix it.)

post edited by timidi - 2016/10/06 17:17:05