Factors for determining bit depth needed in the real world:
 
1. Quantization error + dither (i.e. SNR of the medium).
2. Peak to average ratio of the audio plus any headroom (i.e. how far below 0 dBFS the average level is).
3. Average playback level in the listening environment (dB SPL).
4. Noise in playback equipment and listening environment.
[EDIT] 4a. Source noise in the recording.
5. Absolute threshold of hearing (ATH).
 
Once the level of the QE + dither is below #4 or #5 (with a reasonable margin for error), you're done. Increasing bit depth can't add resolution because the resolution is in fact limited by #4 and/or #5 and not the bit depth.

Older analogue recordings sound fantastic when playing at the source rate. Even though we can only hear a certain audible range. A couple of albums I have are 24/96 hand originally were mastered for vinyl. It just sounds better than a clinical CD with post production to colour the peak transients.

Is it the 24/96 or the "_mastered_ for vinyl" that makes it sound better than cd (or mastered for cd)?

It's pyscho-acoustics really. For example, if the recording was done on analogue gear all the way. Captured really high to maintain as much nuances as possible.

Again only in my opinion the results are more musical because no digital limit had been set by the mastering engineer in terms of headroom. Further more when left at the higher sample rate only thing between you and compromise is your equipment.

Modern digital music is often compressed for radio. I use soundcloud as it places no format limit as such.

Here's a honest plugin developer :

For many years, processing speed, sample rates and copy protection were limiting.  Things have changed.  Digital Technology can now come very close to having all the wonderful non-linearity and “warts” of the best analog gear.  We say “close” because most of *******’ analog gear will pass 150KHz easily, and that is something that is impossible with current digital technology.  But hey, who hears above 20KHz?  ::Spock Eyebrow Raise::  But it is difficult getting everything right up to that 20KHz, especially at sample rates of 48KHz and below.

Want to know who it is ?, just copy the text and google it ;-)

interpolated
It's pyscho-acoustics really. 

 
I'm not sure quite what you mean by "psycho-acoustics" here. But just to be clear for anyone out there who doesn't know, the term "psycho-acoustics" refers not just to "psychological" aspects of perception, but to entire process of how we perceive sound, especially the physiology of the ear. In fact most of lossy compression is based on the workings of the inner ear before any signals even get to the brain.
 

For example, if the recording was done on analogue gear all the way. Captured really high to maintain as much nuances as possible.

 
Capturing really high doesn't maintain any more nuances once the digital exceeds the other limiting factors.
 

Again only in my opinion the results are more musical because no digital limit had been set by the mastering engineer in terms of headroom. Further more when left at the higher sample rate only thing between you and compromise is your equipment.

In the real world, any limit is always going to be the limits imposed by the analog source, or perhaps our hearing. The only thing one gets with a higher sample rate is stuff we can't hear anyway (and with vinyl, anything > 20 kHz is mostly just high frequency noise).

Ok it just music peeps.

I really don't see much point expecting an obvious improvement in sound. Lots of things in life are needlessly OTT to keep someone else in a job.
 
Having said that my copies of Pink Floyd Division Bell and Endless River (or something) sound a bit more better than my other versions. Also I have the vinyl of Endless River although still in the box.
 
My point is we have options and why do something one way just because biology says so.

From my own point of view, 24/192 is a bit over the top, as too is 24/96, but I can certainly here the difference between 24/44.1 and 16/44.1. As one poster has mentioned, if you use a lot of saturation plugs, the more you have to reduce bit depth...the more distortion becomes obvious. Not the wave file mind you, but when you convert it to Mp3 you can definitely hear the problem. This is a larger problem though, within the misunderstanding of the digital medium and the analogue emulation aesthetic. If you understand the digitized medium properly, and it is used properly i.e. 32bitFP or 64bitFP, as well as, when the audio gets digitized the signal cannot be fed out of the box again for further processing, because this is where a lot of the problems occur, and is the only reason you would need boutique and expensive converters, but so long as you understand this, and understand the sweet-spot theory in regards to real world and virtual processors...digital far surpasses analogue. This is one of the reasons why I bounce out all my final masters as 24 bit now, and then convert them. When I listen on my phone, its all 24 bit wav files too. There is no reason not to use 24bit files now, because storage on our devices can handle larger files.
 
The other thing that needs to be mentioned is the obvious difference in time based effects and 96k, it's the reason why there's a switch for that in Sonar...if you listen carefully its in the tail, the difference between 44.1 and 96k. It just sounds nicer. I also like 96khz for recording anything acoustic, there is obviously more air, but I always convert the signal to either 48 or 44.1 for general listening.    
 
Ben

All very edumacational stuff.