What's the problem in just lovering the gain those 3 or 4 dBs and leaving those unused? A loss of 4 dBs of SNR leaving your average "only at 98 dB" above floor will compromise your "artistic judgement"?
It's because we're not talking about peak levels and not about SNR.
Sure, if I find my master bus is too hot, I just turn every track down. Nobody will notice if it's -3db versus -0.1db like your typical commercial recording. It's also not just about preventing clipping, but also about giving your bus limiter something to work with.
Turning your peak levels down by 4db is not the same as lowering your average RMS by 4db. One is a technical decision that listeners won't even notice; the other is an artistic decision that dramatically affects the way the song is perceived.
Some people might actually
prefer the sound of DR4 over DR14. If your target market is diehard Metallica fans, for example, you'd better get that compression ratio up there or you won't snag them. It's not about being competitive loudness-wise, but simply meeting expectations.
My personal tastes are all over the map. One of my favorite recordings is Alisson Krauss and Union Station's "New Favorite", with an average RMS of -14db and a DR rating of 11. It's mellow progressive bluegrass, great falling-asleep music. And it's DR11.
For wake-up music, a piece I like is Jordan Rudess' version of Tarkus. It's much more agressive than ELP's original: -9db average RMS, DR7. It does not sound overcompressed to me, just very aggressive and loud. Definitely not for falling asleep to.
For pure uncompromised quality, there are few better pop examples than Paul Simon's "Graceland". -18db average RMS, DR13. (Note the discrepancy between the DR rating and the average RMS, a symptom, I believe, of the DR meter looking at only the loudest parts of the song.)
Stevie Wonder is often held up as a shining example of dynamic integrity. "Yester-Me, Yester-You" has an average RMS of -13db, a DR rating of 11.
My point here is there is no magic DR number where everything starts to sound great. Even a DR rating of 10 or 11 isn't necessarily dynamically-compromised.
EDIT:
I dug through my record collection looking for a super-squashed song that I nonetheless enjoy listening to. There aren't many, but I came up with "Jesus of Suburbia" by Green Day. I consider it a pop masterpiece, even if I wouldn't listen to it twice in a row.
-6db average RMS, peak amplitude 0db, and it gets a DR rating of 5. Would it be a better song at DR7? DR10? DR14? Can't say. But I do think that people who buy Green Day records don't expect subtlety, and like it or not, if I were GD's producer I'd recommend heavy compression because that's their niche and their fans' expectation.