So when you say your having latency issues, Is this when you turn on input echo?
With my us1641 the best I could get was about 14 Ms but at that setting it would stutter. So I always used the "normal setting which was then around 22ms.
But I never would "hear" latency because I always monitored from the interface.
My problem was with using VST's and Effects. My system would just crap out if I tried anything complicated. I always blamed it on my computer which was older and only a duo core 4 Gigs of RAM.
I had to stay with my in laws for a week and I installed X3 on this HP i7 , 16 gig of RAM/ 7200 RPM drive mega computer and I had the same experience...So I got fed up and drove over to a music store and bought the Focusrite 6i6 and have never looked back. Tascam's drivers are just not great.
But to tell the truth, the RTL of the Scarlett is not a heck of lot better than the Tascam, it's the stability that's hugely improved.
Edit:
I see we where typing at the same time.. If your tracks are not in sync then there's something gone weird, I never had that issue with my Tascam.