If the patch you're using has time-based effects on it (such as delay or reverb) and those effects are tied to the tempo of your track, you can have problems like this. As you've seen, it also happens with "external" delays. The problem is because the delay is being re-calculated on the fly.
How quickly does your tempo change? Is it every quarter-note for one measure, or every sixteenth-note for two measures, or what? You could try having the tempo change less often (say, only twice in a measure), and that will lighten the load on your CPU. Otherwise, you could try just setting the delay manually (using a tempo/delay calculator, for example), and see if that works better on your system.
Steve