What Dave said is probably what is happening to you.
Are you using the new Tascam drivers ASIO that they released in Feb? Big improvement. There was also a good one in Dec 2014
If you doubt your system is not syncing you can perform a loop back test.
Play a Drum audio track out and record it back to a new audio track. Zoom way in and see if they line up.
Here's a screen shot of my US1641 and you can see in ASIO mode the track was early.. there's nothing else in the project. Then in WDM mode it was the opposite and real late.
Track 1 midi drum track
Track 2 frozen Session drummer , see they line up perfectly
Track 3 loop back recording at Tascams Normal asio buffer
Track 4 loop back recording at Tascams highest buffer , notice it's the same.
Track 5 loop back recording in WDM mode.. now we are late,,
I need to test this again using the new drivers , these shots are a couple of year old.