I do notice the latency, if not I would have gone down the Guitar Rig route a long time ago for recording purposes. The other problem with latency is that to work at 1.9ms puts a lot of stress on the CPU, and Guitar Rig proportedly puts a good deal of stress, plus reverbs and other effects on other tracks, and we're asking for trouble.
The computer is not the only source of latency. I bet my V-AMP has a minute latency, then there's the whole signal path and multiple conversions that have to take place before my speakers can get the sound to me at the speed of sound (another latency factor).
There is a point here though, if someone is just strumming out rythm parts at 100bpm I doubt latency will be a factor, but shredding out 16th note triplets at tempos around 160bpm and there is no chance of keeping in time. I may be part of a dying breed, but I love shredding through arpeggios and creating intricate tapping runs at extreme velocities.
I never thought about this from a drummers perspective. It must drive them nuts too, especially with V-Drums.
Anyway, this is a great topic and let's keep it going and keep adding to it.