Increased Latency

I route a slave computer loaded with Kontakt 4 samples via spdif to the incoming spdif signal on my sequencer computer.  I turn on the Echo Input (I think that's what its called) so I hear the spdif signal in real-time, and everything works dandy.

However, when I start to load to an excessive amount of vstis (I'm talking 3 stacked kontakt 4s, PLAY, Superior Drummer, 4 WIVIS, etc.) within SONAR X1 Studio, the latency is increased during playback JUST on the SPDIF signal (everything within SONAR is fine).  However, if I create an audio track and RECORD the spdif signal, when the recorded clip is finished, it plays as if there was no latency.

Throughout all of this, my buffer size never changes.  When I have my normal template up (2 full kontakt 4s, 4 instances of WIVI, ~100+ midi/audio tracks) there is no noticeable latency.  Again, it's only when I start adding VSTis on top of this (obviously the more added the higher the latency). 

Is there any way to fix this? 


My current setup is as follows:

i7-950 3.06ghz
12GB DDR3 RAM
M-Audio Audiophile 2496
Windows 7 Home 64-bit
Running SONAR X1A 32-bit (I believe it does this with 64-bit, as well, but Altiverb doesn't work properly with the 64-bit version at the moment)

Any help would be GREATLY appreciated!

-Steve

Yes, I believe the latency is only noticed using the echo input.  I would try an alternative method, but SONAR requires echo input in order for any outside source to be heard. 

I guess technically I could hook up my slave pc to my monitors and just keep the echo input off (I have the Behringer MonMixer unit so I am able to have up to 4 audio sources going to my monitors)?  It doesn't really "fix" the problem as much as it evades it.

Does that sound like something that would work? (I'm not at home right now, so I can't test it)

I believe so - assuming hardware monitoring is what I think it is :)

Do you mean can I send audio directly through my pc without having to use an input echo (or even have SONAR up at all)?

Well, the VSTis are being loaded on the SEQUENCER machine, not the slave machine (I guess aside from 1 stand-alone Kontakt 4 on the slave).  And it would make sense if everything was receiving latency, but only the incoming spdif signal is.  So if I have a percussion rhythm (slave machine) on 16th notes, and violin staccatos (sequencer machine) doubling the percussion on 16th notes, the violin notes will be in time with SONAR, but the percussion will be a little off unless I record the spdif signal to an audio track.

Hopefully that made sense?

Also, I forgot to mention I am using MIDIoverLAN to send midi to my slave machine.