What ARE your current Sample Rate and bit-depth settings, and your total roundtrip latency?
I suggest trying for a total roundtrip latency of at or slightly less than 10 milliseconds. I saw you are reporting around 11 ms, but it is not clear if that is input, output, or total roundtrip.
Also, what sort of plugins are present in projects where you are having these troubles? (32-bit, or 64-bit ones that use look-ahead processing or loads of CPU?)
What are your settings for Playback I/O Buffer Size and Recording I/O Buffer Size? If they are at the default of 250 k, or 256 k, try setting those to 500 k or 512 k.
Bob Bone