MIDI Drum input..ABOUT TO SCREAM!!!!!

Just a couple quick thoughts:
 
Is the latency/buffer on the Audiobox set to as low as you can make it?  Not sure how low the audiobox goes, but hopefully to atleast 256.
 
Do you have any plug ins active which might be delaying the signal?

What Mike said. I use a Roland SPD-20 driving XLN addictive drums VST, and any of the grunty buss effects such as LP64, etc, will have this side-effect of introducing latency.
 
Keep your active FXs disabled when tracking drums.
 
Do you hear the latency on a clean, empty project?

I will add to the above excellent advice, which is what I always do too, that no matter what, I will still have enough latency to put me off of trying to trigger a Soft synth drummer while  tracking drums with my Yamaha  kit.
It's only something like 5ms but to me it is enough to loose my groove. So I monitor the brain of the kit and all is fine. It depends on your interface and set up, but I use inputs 3/4 on my Scarlett for the drum brains audio output. Mix Control allows me to track using that.
I think it is the Yamaha USB driver that cause this latency, I don't have latency when recording my Roland A 49 controller IF I shut down all offending Efx bins.