Latency tolerance

I realize that ultimately I have to rely on my own ears and wallet to make this decision for myself, but I'm really curious what latencies different folks experience and find acceptable in different situations.  For reference, the most I ever record at one time is a single stereo pair.  I have several devices with USB audio capability as well as an older PCI audio interface.  The worst of the USB devices can NOT operate reliably at a measured round-trip latency of less than 19.5ms. 
 
What are you guys getting?
 
 
 

 My buffers are usually set to 512 so I get around 20ms round trip.  I don't do a lot of recording so that works for me. If I have to record I can set it lower and revert back to 512.
 From what I've read RME devices are great low latency devices. You can't touch one for under $500.  If I could do it over again I would have bought a Hammerfall PCI instead of the Yamaha SW1000XG and they were about the same price.  Any PC hardware today will not run the Yamaha.
 

At 24bit/48KHz, 12 input channels open, of which one stereo pair is recording and another 8 are being monitoired via track echo with delays on several and a low cpu load reverb buss I get a (measured) round trip latency of 4.8 milliseconds.
 
So long as I don't use any plugins that add latency of their own of course. I should probably add that I rarely use VSTi synths other than for drums so the cpu load is usually pretty low.
 
RME really are worth the money of you need stable, ultra-low latency operation. To be honest my previous UA-101 gave just under 10 ms round trip (the minimum possible at 24/48) without a hitch, but any higher latency than that and I start to have problems.
 
Oh, and add another 3 or 4 ms for the time it takes for the sound to get from the nearfields to my ears.

On FW with the TC Konnekt the newest drivers are down to 4 ms.  W/ USB on the Tascam UH-7000 it is like 12 ms. round trip.   I couldn't work at 20 ms - since I do a lot of soft synths.  Far too spongy.
 
@

I do everything in the box so a project usually has 1 instance of Kontakt, 1 or 2 Superior Drummers ( one for the kit, another for latin percussion!), 2 or 3 TH2s, 1 GK Amp Pro (or Mark Studio 2/ just bought!), 4 or 5 synth vsts, and all the prochannel toys as needed on the channels and busses.
Since Craig's eye-opening tweak for lower latency I have been able to hover around 128 and 256 samples with no problems. Even while tracking! I do set the latency back up to 512 for playback when the mixing and mastering is ready. If I trim things down some, I have actually gotten down to 64 samples!

Thanks all for sharing your thoughts.  I'm trying out a USB mixer right now, a Behringer unit, that operates really well at 19.5ms RT as measured (it reports 22.4).  The way it's setup I can direct or live monitor as I'm recording - I gotta tell ya, I don't feel any difference.  With Record Latency Adjustment set to the measured number of samples, everything lines up perfectly.  Maybe it feels OK because the vast majority of the latency is on the output (~82%+ of it).  If my thinking is correct, all of the audio I'm playing to is experiencing this, so in effect it's the input latency that matters most for feel; 3-4ms in this particular case. 
 
I like this unit because I need to replace my old, dead mixer anyway.  I can always add a lower-latency interface later if latency becomes a problem.
 
It's a shame about my Delta66 and Win8.1.  The big problem now is that even though the Delta may be working perfectly, it will always be the scapegoat for any problems with Sonar that come along.  The Easy Way Out, if you will.  I wonder if Avid *really* thinks that everybody they force to abandon Delta series interfaces will end up buying one of their others.

Avid/M-Audio's driver support for new releases of Windows became so poor that I stopped buying anything by them two or three years ago. A pity really, the Delta series were very good value and being PCI gave great latency times compared with most USB interfaces at the same sort of market point.

I'm not sure people understand 100% about latency. 
First, there is a separation between recording with MIDI and recording audio tracks. 
 
I can have the LP64 multiband compressor on the master buss and record audio without issue. 
I have to bypass it when recording midi. There is huge latency. Why? 
 
Input echo. Monitoring your signal path correctly. 
 
We have to use input echo to record midi using a soft synth or we won't hear anything. It is well known that "look ahead" plug ins cannot be "on' because of the latency they add. The midi information arrives at the soft synth within 1-4ms. But the processing of the DAW now slows everything to a crawl before it makes it back out to your monitors. So we need to minimize processing and have low output latency. 
 
We do not have to use input echo to record Audio. We monitor at the interface. They miss name this feature "Zero latency"  to make you think your $100 interface is awesome. 
Of course it's almost zero. But it's got little to do with round trip latency. It's got a lot more to do with how Sonar and the audio drivers calculate the interfaces latency. 
 
Your listening to your playback that Sonar has politely adjusted to sync with the incoming audio signal. It is actually playing the DAW output "ahead" of time so this will happen. 
So your playing to a time aligned playback signal along with your performance in real time at the interfaces analog headphone and output stage. Therefore the term Zero latency monitoring. 
  
Even using the worlds worst performing RTL interface you will hear no latency or very, very little as long as that input echo is off.  
Turn on input echo and you will now hear your RTL as a sort of digital delay. 
Mess with buffers etc and this can be minimized to almost acceptable levels with the right interface. Myself, I'd rather use the direct monitoring. I can't take much more than 5ms and it bothers me. My systems can't get below 12ms. Too much. 

johnnyv...how can I use direct monitoring with MOTU 8Pre...and correct me if I am wrong, but direct monitoring would not allow me to hear TH2 processing, so some other means of tone for my guitar would be needed to allow playing the parts with the correct attitude, if you know what I mean.