This starts happening to me the more tracks/clips/takes I record. This is why as I work I'll trim things down by creating comps of stuff I think I'll use as finals, put them in a new track and Archive (and Hide) the originals.
Also much of it seems to be due to HDD lag. It's trying to ready a ton of info so it takes it's sweet arse time starting playback and whatnot. I find slightly increasing the Read/Write cache helps (but currently I'm doing everything from a single drive which is foolish... I need to set things up use my second drive but have a lot of cleanup to d first).
And if you aren't tracking you might as well increase your interface buffers as latency does not matter when not recording. This helps things stay reasonably smooth as far as editing/navigation/mixing/etc. The only downside really with increased buffers is playback does not start immediately but other things happen faster/more reliably.
Just how I do things. I create pretty massive, audio based projects so managing what is and isn't necessary becomes kind of crucial after a while.
You could be starting to experience project corruption too which is a good reason to keep a lot of Save As points to return to. I've also started sometimes creating completely new projects and importing comped/edited material into it to continue working if I feel the original project is getting too out of control or starting to corrupt.