In my experience it's all about plugins. In bigger projects where I have 10, 20, 30 plugins, it might take 20 or 30 seconds to load, even if none of them are samplers. In projects where it's just audio and no plugins at all, it loads up in about 2 seconds.
Another thing to consider is that it's common for hard disks to go to sleep to conserve power if they're not in use. This adds a 5 to 10 second delay when you first try to access them as they wake up. You can usually hear the whirring sound when this is the case, but not always, so try browsing the drive prior to opening the project in Sonar to verify that it's awake before measuring how long opening a project from that drive takes.