John
When mixing as Dave was alluding to latency is unimportant. When tracking (recording) where you are monitoring the recording, latency can become an issue. Best advice is don't use plugins when recording. When done with doing the recording that is when one can use plugins. All plugins will introduce some latency. However many will introduce a lot for the reasons listed above in Dave's post.
Thanks John for the reply. The strange thing is, i don't use plugins while recording or monitoring, this issue is coming up while trying to mix. For example, I have a project with dozens of tracks, all sounding fine and on time EXCEPT for my "overhead cymbals" track. This track plays back almost a whole second later than the others (which really kills a big break haha). This is the only track that had the PHA-979 plugin on it. When removing the plugin, it plays back fine and on time with the other tracks. So this is more than just simple latency that some plugins always introduce, this is literally throwing off my mix and timing completely. Some mixes i had to really rework the space entirely because of it.