NTFS file compression

Last week I asked a question in this forum about the wisdom of using file compression on my sample files and the unequivocal view was that if NTFS file compression is used it brings additional load to the CPU in loading a file and therefore slows everything down. Short answer - don't do it - bad juju - dumb idea.

OK, that seemed logical until I encountered a work colleague who insisted that this view no longer holds true because of modern processor capability/capacity in conjunction with a smaller file footprint on the hard drive for compressed files. The substance of his argument seems to be that the the hard drive rotation time to read the file is the limiting factor now, not the de-compression time required by the CPU. Compressed file = smaller footprint on the disk = less disk read time.

I tried to think about a way to test this at first hand to satisfy myself and in the absence of any good measurement tools at hand, I decided that I would take a 140 MB WAV file and import it into SONAR - measuring the time it takes to load. I would then copy the 140 MB WAV file - rename it and apply NTFS compression and repeat the process, timing it again.

Guess which one loaded faster? Yes, the compressed version! Uncompressed load time is approximately 5 seconds, compressed file load time is approximately 3 seconds.

Hmmm... OK, but that may not be the full story. Do compressed files load faster only above a certain file size? What is the truth here? What tools can I use to explore and understand this better. Or maybe I don't have to explore..... What is the wisdom in the forum?

Thanks.

ASUS P5W DH Deluxe Mobo
Intel Core Two Duo 6600 2.40 GHz
2Gb Corsair RAM
Vista Ultimate (32 bit)
74 GB Raptor C: drive
Dual 150GB Raptor drives in RAID 1 configuration for project files
150 GB Raptor drive dedicated to sample storage
NVIDIA GeForce 7600 GS video card
2 X Philips 190B monitors
SONAR Producer 6.21

I think your experiment has some severe limitations. loading a single file no matter how big it is is not representative of what manipulating audio data is about.

Now lets say you're editing 10 audio tracks each being different .wav files on your hardrive and you make a simple edit across all 10 files like cutting 1 bar of music out the middle of your song. Would your experiment still hold true?

Or how about streaming samples which can sometimes be much larger than your experimental 140meg file.

I reckon keeping files uncompressed is the way to go at least until someone can show me some well documented experiment results to the contrary

Loading a file while the CPU has nothing else to do is not really a good experiment. That may apply well to sample sets that are loaded into RAM, so they only get accessed when the project is opened. In that case I'm sure you could save a couple seconds opening the project.

Things will change once you're trying to stream to and from the disk real time while asking the CPU to process plug ins. Performing file compression and decompression is certainly going to take CPU cycles away from the plug ins. Of course if you're project is not using everything you've got, then maybe you can spare the cycles and save some disk.

IMO, disk space is so cheap that why would you bother?

NTFS allows compression per file, per directory, per drive - it also doesn't compress when the data won't comrpess (it does it on a cluster by cluster basis).

Parts of XP - eg. the \windows\system32\dllcache directory - are compressed on a standard XP install. One reason that the whole \windows directory isn't compressed is that this would mean the boot loader needs to be able to handle compressed files to start loading, increasing its complexity. The NTFS driver handles the compression, so until that has been loaded, Windows can't access any NTFS-compressed data.

Many thanks to all who have responded to this post. What prompted this train of thought is the fact that the samples
associated with my soft synths have grown to the point where they exceed the available space on the 150GB Raptor drive I have set aside for this purpose. (dfh Superior alone needs 35 GB) Whilst it is generally true that disk space is cheap, I
already have four drives in this computer and adding a fifth presents some practical problems in terms of physically
accommodating same. Raptor drives whilst wonderful performers are not the cheapest drives either and they don't come
bigger than 150 GB.

I acknowledge that the test I have done is crude, but without any meaningful test software it is at least a start. In the
particular circumstances I have outlined above, streaming samples direct from disk is probably the issue of most interest,
but how might this be tested in an easily measurable and scientific way in order to remove perception as a variable.

For the record, the first test was done with both files in the same folder on the same drive which is well defragged, so I
know that at least in this circumstance, a compressed file loads more quickly. But does that translate to better
performance streaming samples from disk? That seems a leap too far without a better testing regime.

Thanks for your comments bitflipper, that all makes perfect sense. It does seem to me that nobody has actually done a
recent review of the benefits or otherwise of NTFS compression in an audio environment as described above. If I am to make sense of this I need some good tools for accurate measurement. Any suggestions for appropriate utilities?

Cheers

Midimanz

ASUS P5W DH Deluxe Mobo
Intel Core Two Duo 6600 2.40 GHz
2Gb Corsair RAM
Vista Ultimate (32 bit)
74 GB Raptor C: drive
Dual 150GB Raptor drives in RAID 1 configuration for project files
150 GB Raptor drive dedicated to sample storage
NVIDIA GeForce 7600 GS video card
2 X Philips 190B monitors
SONAR Producer 6.2.1

One possible wrinkle in this scheme is that wav files typically don't compress very well. You may not get the big savings in disk space you're hoping for - maybe only a 5% reduction in file size.

Generally, the better the compression ratio, the slower the process and more cpu cycles required.

The 140meg wav file is one you created? If your disk space is sampler's samples, realize that most of them are already compressed in their own format and I'd expect recompressing/decompressing them again will only serve to add more work for the cpu. For example, EZDrummer files are already super compressed and 'tuned' for the program. Same deal with SampleTank.

ORIGINAL: bitflipper

..maybe with the way that Windows wants to compress them ;)

Yeh, maybe if Windows turned them all into MP3s, that would save a lot of disk space!

Ehhm..It's all lossless compression algorithms there in that chart.

I never said that Midimanz should compress that drive but I just had to respond to the "wav files typically don't compress very well" .. no one said MP3 or other lossy compressions..maybe you didn't look at that chart..see how good RAR is doing for example?

The CPU's big hit - of course.. i would never use that for multi track "realtime" playing.