If the note is played on your controller then then it's not unreasonable to have those notes land anywhere before or after the place it will belong. You then quantize the track and the notes will be put in the correct place.
If your driver is not reporting the offset correctly then the timing can be out from what you are playing along with.
Do a loop back test to see if your driver is reporting the correct offset.
Patch the output back to an input and re -record a drum part. example shown below.
Notice how in ASO mode the re recorded track is early, This would be why you would have this issue... your hearing your backing tracks ahead of time so you put the note there.
Track 1 MIDI drum part
Track 2 Audio frozen to session drummer ( note it lines up perfectly)
track 3 ASIO re recorded track normal buffer
track 4 ASIO re recorded track highest buffer
track 5 WDM mode - notice WDM is real late.