Hmmm... something not making sense, That big timing offset should be making audio from the keyboard
earlier relative to soft synths and audio, which was the goal. If that's not improved, what is? In any case, that big Manual Offset is going to cause all kinds of weirdness in different scenarios so you;re not going to want to leave it that way. It was meant to deal with the usual MIDI-Audio offsets on the order of 3-6ms.
You'll definitely want to look into the latency/dropout issue. As you can see from my signature. I run a MOTU 2408 with PCIe424 , and it can play pretty heavy projects (like the Cakewalk demos with all synths and FX unfrozen) with the buffer at 256, and all my usual stuff at 128. Have you run DPC Latency Checker or LatencyMon to check for spikes in Deferred Procedure Call times?
The real-time performance latency you're getting is due to a combination of MIDI transmission/response delay and audio latency. But you should be able to get the total down to about 10ms (playing the Fantom by echoed MIDI), which is quite tolerable, and soft synths will be better because they're only subject to inbound MIDI delay and outbound audio latency.
In addition to getting your buffer down, I think you need to do some simultaneous MIDI-Audio record tests from the Fantom - first with Local Control On, and then by echoed MIDI. And also maybe a MIDI and metronome loopback recordings to figure out where the delays and sync errors are. The more you understand about all the sources of delays in you system and the smaller you can make all of them, the better off you'll be.