Could it also maybe due to significant changes in PDC with some plugins?
I'm not sure how well CbB will cope with a massive change in plugin delay during playback.
For plugins that have a constant latency regardless of whether they're on or off it's probably fine, but for those that reduce latency when they've got nothing to do... maybe CbB has problems coping with this in certain circumstances?
M.