Still having latency issues

I've searched everywhere and still can't find the answer I'm looking for. I have my Fast Track Pro 4x4 all set up and it's working well with Sonar 8 PE, but I'm still having latency issues. I have my buffers set all the way down to 128. I have a MIDI drum track I wrote using EZDrummer and I'm trying to record my rhythm guitar tracks. I record my first track and it sounds like I'm dead on with the timing. I pan that track hard left and start recording my doubled track and I just can't nail the timing. I know my timing is impeccable because if I put the same MIDI track on my digital recorder and record guitars, they are spot on. Of course, I freeze and make the MIDI, audio before putting it on my recorder, but I don't figure that would make any difference. Is there something I'm missing? If you need any more info, please ask and I will give more details if I have them. This is getting pretty frustrating. Thanks in advance for any help.