Hi Daniel. I usually set my audio interface at 128 to 512 to allow for minimal latency when playing various vst's via my midi keyboard. Once I start loading a lot of plug-ins - including various FX plug-ins, I may start hearing some pops/clicks, which usually indicates I'm stressing things out, despite average CPU usage. I'll would then increase my buffer range to higher values, testing to see whether this helps. Usually it does, but at the sacrifice of latency.
If it doesn't help, I'll disable various plug-ins to determine which, if any, may be contributing to a playback issue.
With Kontact, I explored using just one module and loading up various other Kontact modules via the various outputs. I found that this really taxed my system and I ended up just using 1 or more individual Kontact modules. This seems to fair better in terms of the way my system handled it.
Hope that helps a bit?