minimise MIDI JItter

Hello all. I'm not on this forum often, as I have been tooling up the hardware and audio side of my studio. Now I'm trying to address the MIDI side of things.
 
I'm seeking the best hardware solution to the lowest possible MIDI jitter and lag. I type in the term MIDI jitter and nothing comes up as a result within this forum.
 
The application is MIDI drums to BFD3. I'm considering a network MIDI interface like this: Alyseum 

 
My A/D D/A sound card is already faster than USB MIDI. round trip reported latency is under 1.8 ms with the input side slower than the output side. I would estimate the D/A sound card latency is under 0.8 ms. 
 
The drum triggers range from 3.2 to 2.2 ms in latency just because the drum brains need that much time to ascertain a repeatable velocity. 
 
When total latency is over 4.5 ms, drums start to feel mushy and are difficult to play vs. real drums.
 
Sonar is set up to 64 buffers with no issues. At 96K sample rate, in theory Sonar is introducing 0.6 ms latency. Also BFD likes about 1ms to do it's thing.
 
So right now I'm at 3.2+0.8+0.6+1= 5.6 ms before you add in USB to MIDI. since all of the other issues are already at minimum I'm working on getting MIDI inputs under 2 ms lag time and under 1 ms jitter. If a solution existed, I would pay up to $1,000 for a solution of getting MIDI (Both USB and Old style MIDI) into the computer under 1 ms lag and jitter. 
 
I have found no posts here or on gearslutz talking about Sonar with Alyseum (There are post about Alyseum on gearslutz). I also have not found any current suppliers of PCIx MIDI cards.
 
Note: RME is not a solution for me because I have a good working A/D D/A solution and don't want to mess that up. I don't think they have any MIDI only solutions.
 
My current path is to try the $100 Alyseum 2 in/out model and test if it works. But I thought a post here could be worth a try.
 
Idea's ? solutions ?