I apologize if this kind of topic is verboten.  Everywhere you see people talking about issues loading sample libraries in their DAWs, but folks don't tend to actually say how much and how long - they just say "long".  I would like to understand if my situation is normal.
 
I am on Sonar Platinum, my DAW is an Intel i7-4820K quad core, Asus X79 mobo SATA 6GB/s with 32 GB RAM and separate Samsung EVO SSDs for sample libraries and OS.
 
I just timed loading one of my projects that has a big template spanning six instances of Kontakt 5.4, having a total loaded RAM consumption of 8.1 GB.  The project "comes up" (the UI is completely up and I can see my track view and multidock fully populated) in maybe 20 seconds.  However, I can see each actual instrument fully loading samples one instrument at a time (I assume this is the so-called background loading) until they are all done.  Loading progresses through each Kontakt instance in the order they appear in the synth rack.  It takes about 2 minutes and 30-45 seconds to completely load all Kontakt instruments in all instances (to where I could hit play and hear no drop out at all because of unloaded samples).  By the way, as far as I know each library has been batch-resaved.
 
So, does 2.75 minutes seem a reasonable load time for 8 GB of samples hosted on SSDs?
Thanks for any comments - moderator please feel free to chuck me in the offender's bin if I committed an indiscretion with this post.
 
Lum

That doesn't seem unreasonable at all...that's a lot of samples. Divide 8 GB by the SSD transfer time and you'll probably end up with the time you measured. Be thankful the samples aren't stored on a USB flash drive...they do about 30 MB/second.
 
As to inappropriateness, no issues there...lots of SONAR users have Kontakt. But this might be moved to a different forum as it's not SONAR-specific.

Per comments from Noel, you give up multiple thread processing when you lump a bunch into a smaller number of Kontakt instances, and you might be better off on your load times to split the template into a bunch more instances of it.
 
I myself don't build massive templates, but orchestral folks have different needs, and for them (and others), big templates may be desired.  The more you load, the longer it takes, so, balance convenience with consequence is about the best advice I can give you.
 
Also, even if you split your OS from your sample libraries, on different drives, if you are loading the world into Kontakt, it WILL end up fighting with itself doing disk access at some point.
 
You may want to consider further splitting up sample libraries among additional drives, to split things up even more, and I suppose the order of loading into a given Kontakt instance would potentially be something to consider, in terms of which libraries from which instance and which drive are optimized by which ones appear in which order in the various instances.
 
Bob Bone
 

Thanks Craig.
 
I guess that's why I am confused or curious... the math would suggest  8GB samples divided by 500 MB/s transfer rate should be about 16 seconds.  I have been assuming the additional time was a matter of overhead with regard to the actual number of samples (individual file overhead) being loaded versus the overall size.
 
So I was hoping someone who often loaded large templates could say, "Yeah, 2-3 minutes is about right for 8GB or... no, dude, something is wrong, mine loads under a minute."  Maybe that's what you're telling me :)

That 500 MB/s figure is theoretical. You mentioned that it was an external drive; if it's USB 2.0, the fastest it can go is 60 MB/s but that's virtually never attained - in the real world, 40 MB/s is more like it. Also, you mentioned separate drives but whether they're 2.0 or 3.0, they're probably being driven by the same controller chip. The USB controller has to wait for the drive to say it's ready before it sends any data, which means wait times especially if multiple devices are sharing the same controller.
 
Furthermore, SSDs work most efficiently when streaming single files. Accessing lots of little files - exactly what you'd find in sample libraries with tons of articulations and hundreds of samples per instrument - will slow things down dramatically. Even without taking the interface bottleneck into account, that 500 MB/s could drop easily to 100 MB/s or less.
 
USB 3.0 drives have a maximum theoretical speed of 640 MB/s, but most top out at around 100 MB/s. Also, you have to read specs VERY carefully - 100 mbps is NOT the same as 100 MBps; 100 MBps is 800 mbps. Not that drive manufacturers want you to read the specs carefully...
 
There's also confusion about what a spec says is the maximum speed vs. whether real-world devices can attain that speed, which is rare and happens only as the technology improves over time. Think of it this way. A planning commission specs a highway to have four lanes, even though the current traffic could be handled easily by two lanes, because the existence of the highway will lead to increased traffic and maybe someday, four lanes of traffic will go down that highway. They're spec'ing for the future.
 
Or in other terms, the engine is my car is spec'ed  to go up to 140 MPH. I've never taken it over 95 

Thanks Bob.
 
Something you said actually just turned this into a Sonar-specific discussion because I have Kontakt set up to not be in charge of multi-threading, deferring that aspect of the DAW operation to Sonar.  I have been under the impression that this was the "best" way to configure multi-threading.  The i7 has, functionally, eight cores and Sonar sees them all (although it always bugs me that it loads up core one more than the rest but I guess some things just can't be done in parallel).
 
But as to the multiple instances versus more/fewer, the behavior I see tells me it has no bearing; only one instrument at a time is background loading, so even within one single instance of Kontakt you can see it background load instrument one, then two, etc. and when that instance is done it goes on to the next, loading instrument one, two, etc.  It would be lovely if they would all load in parallel.  And we know Sonar can handle a great many disc I/O threads in parallel - after all that's what it does.  It may, in fact, be that the issue is there are so many individual files to load that I am flat consuming all the threads available for loading and that it's not a SSD transfer issue, it's a file reading thread limitation.

That's why I am looking for some anecdotal feedback from someone who does load larger templates.  But, seriously, thanks for the thoughts.

Thanks again Craig.  Not sure where the "external" came from.  These are internal SSDs on 6GB/s SATA.

Frankly, I am not super-stressed about the load times.  I usually get up to have a wee or go get a pop when I load.  But this large template issue nags at me.  Most of us "feel" that putting super-fast SSDs into a new system with the fastest SATA pipe available is going to make our load time delay go away.  Either that's just not the case or I have a Clark Griswold switch problem still lurking (maybe I can get Beverly D'Angelo to come over and flip some switches).
 

Steve Lum
Thanks again Craig.  Not sure where the "external" came from.  These are internal SSDs on 6GB/s SATA.

 
Oh, OK. I had this picture in my brain of two separate boxes with drives in them.

Frankly, I am not super-stressed about the load times.  I usually get up to have a wee or go get a pop when I load.  But this large template issue nags at me.  Most of us "feel" that putting super-fast SSDs into a new system with the fastest SATA pipe available is going to make our load time delay go away.  Either that's just not the case or I have a Clark Griswold switch problem still lurking (maybe I can get Beverly D'Angelo to come over and flip some switches).

Take a look at how many samples are actually being loaded, and that's probably the issue. SSDs are absolutely fabulous for streaming, but lots of little files are problematic...although I think that's the same for most storage. Also, are Kontakt's preferences set for streaming from disk or loading into RAM? That will likely make a difference as well. 

So, all this sort of caused me to dig and think a little harder.  I have Kontakt set to override the instrument preload buffer size, choosing 18 kb vs the default 60 kb (this relates to the question Craig was asking about in terms of streaming).  The reason is, with SSDs the DAW can load less of the sample beginning into memory since it will be able to stream off disk for the remainder so much faster (than a mechanical drive).  I have seen people say, "go for it wild man, set that preload down to 6 kb", which is the lowest it can go, but I found with these large templates that 6 kb is impractical and I get dropouts, even with the SSDs ( and I do have two of them for samples, there are actually 3 SSDs in the box... details, details).  18 kb never drops out (yet anyways).
 
But here's the deal and the reveal.  When I say I am loading 8GB into RAM, that's terribly misleading.  That's what Sonar reports for memory consumption.  That's ONLY the RAM being used to cache the PRELOAD portion of the samples.  In fact, the actual sum of the library sizes I am loading in this template - face value as it were - is up in the 40 GB range.  The DAW needs only to load a fraction into RAM because of the SSD streaming, but the loaded instruments have an enormous amount of sample references.  So the upshot is, I believe... that's simply a barnload of samples.  The 8GB Sonar reports is only the RAM used to load that 18 kb preload part (plus whatever other VST RAM I am using for things like Superior Drummer, etc.)
 
So I think this is merely an issue of just a zillion sample files being preloaded.  I think this is where I count my blessings and remember when I used to live in that world Craig was thinking about... the multi-box world connected by midi-over-ethernet with 3 gigastudio licenses running.   Ow... that hurts to remember.

I would still love to hear if there are any other orchestra freaks out there who may comment on big templates, but thanks very much Craig and Bob for taking the time to help me think.
 
Cheers mates.

There must also be quite a bit of processing / reorganising / caching overhead in Kontakt itself during the loading process. I haven't fiddled with settings in Kontakt ... never bothered to research what could potentially speed it up; it's only loaded once and I'm patient ;-)
 
However, I have noticed that Kontakt loads a lot faster after a Sonar crash (had quite a few with X3e, K5 and Octa-Capture) ... so Kontakt must also do some sort caching or likewise which is still available after an "unexpected Sonar termination" ???