Latency has but two sources:

• Your audio interface
• Latent plugins

Your audio interface will ultimately determine the lowest latency you'll be able to achieve (your machine has to be able to keep up with the load).
 
With automatic plugin-delay-compensation, when you insert a latent plugins *anywhere* in a project, ALL other audio is delayed by that amount (to maintain sample-accurate sync).
"Mastering" or "Convolution" type plugins often use a larger processing buffer... which adds latency.
 
When tracking, avoid using latent plugins... or you can temporarily disable automatic PDC.  (Make sure to re-enable automatic PDC once finished)