Well, only 4 minutes wouldn't explain it. If you do Select All (ctrl+a), does the selection only include the same 4 minutes? What I'm curious about is whether maybe there is data way out there in the timeline that you might not be aware of.
Also, does ctrl+s (save) take a long time too? or only initial load? Is the project file itself less than a couple of MB?
If you freeze the synth tracks, does it load faster?