Anyone here working with really large templates?

I ran into an interesting problem yesterday!
 
My main DAW is an Intel Q6600 @ 2.4GHz with 8GB of RAM, running Win7 x/64. While I do sometimes run up against resource problems, I can usually get away with increasing latency to mix, I haven't had to freeze a track in ages.
 
So yesterday I imported a MIDI only project for a class I am taking. The MIDI only uses 273 MIDI tracks. There is only MIDI data in a handful, and I had not yet loaded any instruments, but I wiped out my physical memory. The good news? Well it did not crash, but I had to wait quite a while for the swapping to take place.
 
I have, in the past, had projects that used up to 60 MIDI tracks, and I never had to pay attention to how much memory was used, and I suppose now I need to go back and load some of them to see.
 
In the meantime, are there settings I need to tweak to support this many MIDI tracks? And why do MIDI tracks use so much memory if they are not connected to instruments?
 
edit: for grins I loaded the same MIDI file into Studio One and it required only about 3.5GB - which is why I am pretty sure I'm doing something dumb in Sonar...
 
Thanks!

I'm not sure - I will check with the instructor. I created several test cases yesterday, and I can post those while I wait for an answer.

let me provide a more detailed view of the situation...
 
I ran a test tonight, loading Sonar till it became unresponsive, unstable, or crashed. The following represents the minimum and maximum memory loads and they correlate quite well with the starting condition, which varied a lot more than I expected. I ran the test five times.
 
I rebooted the computer, memory used was (min) 1.26 GB, (max) 1.98 GB
I launched Sonar, memory used was (min) 1.48 GB, (max) 2.47 GB
then I started adding empty MIDI tracks:
 
25 tracks    1.67 GB    2.63 GB
50 tracks    1.85 GB    2.82 GB
100 tracks    2.21 GB    3.20 GB
200 tracks    2.97 GB    3.95 GB
300 tracks    3.74 GB    4.69 GB
400 tracks    4.50 GB    5.38 GB
500 tracks    5.26 GB    6.19 GB
600 tracks    6.03 GB    6.96 GB
700 tracks    6.80 GB    7.73 GB
718 tracks    6.93 GB    7.86 GB (became unresponsive)
800 tracks    7.42 GB    
900 tracks    7.66 GB (became unresponsive)
 
In comparison, the test project (which I mis-typed earlier) contains 718 tracks!
If I import the MIDI file into an empty Sonar project it uses 7.36 GB (min) and crashed at the max case
If I drag the file into Sonar with no project loaded it uses 7.18GB (min) and 7.88 GB (max) at which point it is very (*VERY) sluggish.
 
I'm not sure what other information might be in the test file, other than a few odd bytes of actual MIDI data.

So the question has become, for me, why can Studio One manage a large template, but Sonar can't. I still think there must be a setting I need to change.

I am going to submit this to the bakers, along with the actual test file. I have not heard back from my instructor yet as to whether or not I can post it here.
 
Thanks for any and all thoughts!!

I don't know where your memory usage numbers are coming from. I just loaded an orchestral template with 83 tracks, 16 buses, and 9 instances of VEP: <350 mb memory usage. Just out of curiosity, I created a project with 800 midi tracks.Total Sonar memory footprint: 204mb. So ... About .2mb memory usage per midi track(an empty Sonar instance is ~ 50mb). You have something that is sucking-up memory in your template and it is _not_ midi.

Bill can you say what is in those files?

Just for haha's I did an experiment with MIDI tracks.  With a blank project my memory usage was 8.0 GB (I had other things loaded prior to this).

 
 
 
 
 
I then added 1015 MIDI tracks containing 3 bars worth of MIDI data.  After 1015 track my memory went up to 8.7 GB.  
 
" />
 
 
 

Hey Bill 
 
I opened them with no problems.  The 800 MIDI tracks opened and I went from using 8.1 GB pre-launch to 8.7 GB memory used,  (I had other stuff in use on computer prior to launching, which explains why I had 8 gigs of RAM in use already)
 
My results were similar to the original results I posted previously.  I am not getting the same results as you.

So either I am doing something really dumb, or there is something very strange about my computer.
 
This is good news, really, means there is hope, but I have to do some digging! If anyone has suggestions I am all ears... and thanks for checking those files out for me!

I don't get how you could possibly describe 718 midi tracks as "not a lot, really", even considering many would be triggering sample libraries on different machines. I have a really hard time thinking what kind of project would need this. I don't usually go above 100 tracks total (let alone just midi) and if I'm honest it'd be lower if I didn't keep so many ideas and backup tracks around. I can't imagine it's actually easier to navigate that many tracks than it would be to just set up the tracks when you need them. You can only really program or play one at a time anyway. Genuinely interested what kind of class this is and what the philosophy is behind such a gargantuan template.

Loaded sonar, Ram usage: 1.8Gb
opened 800 midi tracks.mid, Ram usage at : 2,1Gb

Closed sonar, and started it again, Ram Usage : 1.9Gb

opened 800 midi tracks.cwp, sonar responded a bit slow (not responding ) for a min
then it asked for E:\DOWNLOAD\memuse\Audio, does not exist ( its where i had the memuse folder/files)
would you like to create it ( i select no ), then it ask me to locate wavedata Directory
( i select Cancel), then i get a missing plugin (console emulator Bus) i hit ok.
then it report missing midi ports ( dakota Port 1) i hit cancel.

Project is now open, Ram usage: 2.5Gb

Closing sonar and open it again... Ram Usage: 1.9Gb


My question is rather, why on earth do u need more than 16-32 tracks of midi, that i don't understand 

My PC is old, but it runs good enought.

Maybe the issue can be related to a Graphic card as well
i did a test creating  ( insert multi midi tracks ) 96 at bulk, up to 580 tracks

What i noticed was: when i reached above 400+ tracks , they started to add slower and slower

Well fun to try out ...., but will i ever go beyond 16 midi tracks ! almost certainly not..

oops - left out a minor gotcha... in order to make keyswitched instruments a bit easier to use, I end up using two MIDI tracks per... one for the notes, one for the keyswitches. I've toyed with a single keyswitch track for multiple MIDI tracks, but there's just too much over-lap, and I am too lazy to re-assign them<G>.

I'm going to see if I can easily export the track list from my most recent attempt at a master template, if I can I'll post it.

I've seen 400-track projects for video game soundtracks, so yeh, track counts can get up there.
 
800 boggles my mind, though. That would be 10 tracks for each player in an 80-piece orchestra! Granted, they represent different articulations, but most instruments can't even perform 10 separate articulations. How many tracks does a triangle require?
 
Could this memory consumption issue be somehow related to Kiosk Project's thread about excessively long MIDI files? In that case, something in the file was making it appear to be a much longer sequence than it really was.

working backwards...
 
Bit - I thought there might be junk in the project that started this little adventure, but the files I posted had no data in them. They were not long because in fact they had no length.
 
As far as 10 tracks per player... that's conservative<G>. I have multiple libraries that cover the same instruments, and ideally I'd love to have them all at my disposal all the time so I can decide if this library or that library fits the piece better. And while I don't need 10 string libraries, I'd very much like to add at least two more (more than two really, since Spitfire sorta divided their strings up.)

Even now I have KHS Concert Strings 2, KHS Spotlight Strings, Project Sam Orchestral Essentials, and Presonus Orchestral Instruments all covering strings, plus some oddball string effects libraries. It adds up fast! There is no question that both individual track per articulation and key-switched articulations both have issues (and by issues I mean problems), but dang, having had the opportunity to work with a large-ish template it is absolutely the way to go. I was able to play the same part into Addagiato, Mural, LASS (OK, yes, this guy has more libraries than I would think I need, but...), and anyway it was dead simple to audition the differences. Yes, I would like to be able to work that way, with maybe a few less choices<G>!
 
Soundblend - Key-switching can conserve the number of tracks required, but it causes more problems than it solves. The biggest issue is that key-switches are not chase-able. So you have to rewind to get the articulations that you want. I have a kludge where I convert key switches to CCs and use CCs to select articulations, but it is a LOT of work to set up for each instrument, and I haven't decided if it is really going to work for me yet.

There is always the load the instruments as you need them approach. It does address some problems, like memory, CPU, track count, etc, but it adds so much tome to the process, and I am kinda tired of working that way!

The good news is that I seem to be the only person creating gargantuan MIDI files, so there may be hope yet until I can afford a fire breathing DAW and half a dozen slaves, and all the libraries to fill them<G>!

I suppose that I have been know to use multiple articulations as an effect, but it is not something you would normally use if you are trying for a realistic mock-up.

So you guys have been a big help, and I did some digging last night!
 
The first thing I discovered is that the only part of my copy cakewalk.ini that is populated is the dump requests. That seems odd. Odder still, the timestamp in the file is March, 2002. Have I got an ini file problem, a corrupted copy perhaps?
 
And if so, other than completely removing and reinstalling is there a way to get a current copy of the file?
 
And if a complete re-install is required are there any gotchas I need to be aware of?
 
And should I suspect the validity of aud.ini too?
 
I did do a re-personalize, or whatever they call it - no change, which makes sense because I found the master copy, and it too is dated 2002 and empty except for dump requests.
 
So I re-ran the latest (X3e) patcher... still no joy.
 
If it matters I am running the x64 version, and I did have X2 installed prior to upgrading to X3. None of the earlier versions ever made it onto this OS build.
 
Thanks a million... I really need to solve this problem so I can get back to work.

I had not tried renaming them - that's next I guess. I did try both SHIFT and CTL-SHIFT, and the results were the same for both. That's why I have a wee suspicion that the master copy might be the one that got clobbered.
 
Thanks though for the tips earlier in the thread, that's what motivated me to try!