If you're using a convolution reverb, that is often the culprit. Seeing as you run X1 Producer, then Perfect Space will always do it.
I wholeheartedly agree with John's recommendation of Lexicon's Panthion Reverb. I'll often use it for tracking, and maybe replace it at mixing stage. Having said that, I use it less now that I've moved to X2, and use Breverb instead. It seems very low latency, but sounds GREAT.
Also any "look ahead" plug ins, such as PC Concrete Limiter, or the LP64 plugs, will increase latency. Save them for mixing.
If I have any of those plugins on other tracks I disable them, or freeze them, and only use Pantheon or Breverb on the vocal track. I do always put the PCL2A on vocal when tracking, because it sounds so good, helps the singer deliver a better performance, and seems to introduce little or no latency.
Just beware of convolution reverbs and look-ahead plugins (or freeze them), and you'll be fine. Your machine is monster enough to handle everything else.