Timeking, you never were using ASIO. You've been using WDM all along. Forget ASIO4All, get the newest ASIO driver for your interface and give it another try. You might still find that WDM does work better, but it's also possible that ASIO will do a better job. It depends on the vendor's competence with one versus the other.

ASIO4All is a thin wrapper and actually quite efficient, adding only a miniscule amount of overhead. More likely the dramatic difference was, as suggested by Mr. Bone above, the result of smaller buffers.