To eliminate the drum map, you can assign the output of the MIDI track directly to AD2. This will bypass the map, but leave it in the project. To remove the map from the project altogether, choose 'Drum Map Manager' from the MIDI track output, and delete the map.
But if you actually bumped the MIDI Prepare Using buffer all the way to 2500 from 250, try 500 as recommended earlier; too high can also be problematic.
Did you compose the MIDI for this drum track yourself, or is it from a 3rd party? Are all the parts in a single clip? Is it hard-quantized? What's the project tempo? Sometimes changing the tempo or moving hard-quantized notes a tick or two off the grid will eliminate problems with dropped notes that don't respond to increasing the Prepare Using buffer.