My ASIO Buffer Size is set to 128 for recording, and when I move to start mixing, I bump it up to 1024, which then allows some of the more robust plugins to do their thing without dropouts and such.
Some plugins are only meant to be used for mixing/mastering, due to the way they process, either by being CPU hungry, or by using look-ahead processing - which adds a bunch of latency.
So, try bumping the ASIO Buffer Size up to 1024 for mixing, and see if that helps you.
From MY perspective, for the rest of time I will be adjusting ASIO Buffer Size, back and forth, from 128 for tracking, and 1024 for mixing/mastering. It's just the way it is, it seems. :)
Bob Bone