I think the kind of think TAFKAT does is useful, but you also have to know enough to interpret the results. For example, if a PCIe card doesn't outperform a USB 1.1 interface, that's a problem :)

For many desktop applications a lot of this is splitting hairs, but if you're using amp sims live with a laptop, then latency becomes a very serious issue. You want to overcompensate on latency for safety, so with a high performance interface, you'll get lower latency and still have a comfortable margin.

It's also worth noting that not all interfaces report latency accurately, and his interface tests show this. For example, for some reason Line 6 was always able to be very precise, other interfaces less so.

Where tests like that are helpful in that you may be interested in 1) seeing how efficient Sonar is, 2) such as seeing how your DAW's load balancing compares to other DAWs (the dreaded core 1 is spiking the others are nowhere near 100%), 3) determining if there's another DAW that will allow you to squeeze a few more plugins/tracks into your mixes (this would be helpful if you had a fast PC but were still maxing out your CPU during large mixes). It could also help companies like Cakewalk in identifying where there can be room for improvement, by someone else doing free performance benchmarking for them.


It's completely legit, and if you read about their testing methods and how they distribute the plugins...I think it's a fairly unimpeachable method of comparison. If you also read his criticism of Noel's testing method, his points seem entirely valid. I'd love to see Noel's response to that.

What a test like that doesn't show is how well you interact with your DAW, it's workflow, and it's features. These are clearly the most important aspects in picking a DAW. Sonar still some has some things that drive me nuts (limited routing options, the terribly huge, non-resizable mixer, no varispeed, horrible notation) but overall, it's still the best DAW choice for me. I've yet to find the perfect DAW, and every one out there seems like it's good at some things and not so good at others, and overall Sonar has become my main choice. So to those who point out that the benchmarks don't address this aspect, I agree with you 100%. 

But that doesn't mean anyone should just blindly ignore the test results because Sonar didn't come out on top. At that point, people are turning DAW preference into a psuedo-religion by ignoring the science. I think we can all be a little more rational than that. 


I honestly don't think the people saying the test is completely irrelevant would be too upset if Cakewalk came out and said, "hey, we improved Sonar's efficiency by up to 25% in large projects by making adjustments to how we handle load balancing" or something like that as a result of benchmarks like these.

Hi Craig,
  It occurs to me that one may not feel a need for 105 instances of convolution reverb yet the same person may also have a keen interest in placing a convolution based, console emulation process on each and every track and bus in their projects.  I imagine stuff like that will be added to the DAWbench tests someday.

 Hi Dean, 
  What I have found on my system is that the portion of the actual round trip latency that is not reported varies with the sample buffer I am using. So I have a selection of manual correction amounts and I use each when it pertains to arriving at actual sync under any given circumstance. 




 best regards, 

mike






written in firefox, formatted in chrome

fbb - "I turn it on, I can tell which input is which, and it works. That's plenty enough for me thanks. "
 
God, that made me laugh..   
 
Thank you, fbb. 

slartabartfast


Look, everyone has the right to have an opinion, but not every opinion is right. It probably does not matter to most of us in using our software for our usual tasks if there is a small difference in audio efficiency between Windows 7 and Windows 8. But there needs to be an objective way of testing that difference, if MICROSOFT is going to be able to evaluate their software. If we just use the opinions of Windows customers, and if the acid test is "Does it do what I usually do?" for most users, then MS can turn out a Windows 9 that just manages to play MP3's without skipping and will be able to satisfy 99% of their user base. Serious audio is being largely ignored already, and there is very little public information about how seriously MS is taking it. 

If somebody is not doing serious reproducible testing of high end applications and making results public, there is not much hope that newer OS versions will improve all that much. The testing reported on Brandon's blog is very superficial. Not to say it is inaccurate or done without honest intent, but it is in no way either exhaustive, comprehensive or reproduced on multiple platforms. It is an N=1 study, of the same significance as a single case report. You can do a more useful test by installing both windows versions on your own systems, and doing a head to head comparison. 

Personally I don't know how valid the results are. Perhaps one can say safely they are valid for the system used to get the results. I have no idea if one can interpret the results  to apply to the larger world of computers. To even try to do that the tester would need at least a large enough sample of various machines with a good sampling of audio and MIDI interfaces as well as basic configuration for displays and so on.  Only then can a statistic have any meaning.   

What is been offered as a statistic is based on a population of at least a very few instances. That is not valid for statistical analysis.  

But than it is interesting but only in that context.  

js516
  
Can't say that it was "oh my god its so much faster" but I did notice that the machine is more responsive with Windows 8, and as I said, I can use smaller buffers that I could not before.  So the claims that Windows 8 is worse than Windows 7, in my case, is not true at all.

Problem with this kind of thing is that a fresh install always feels more responsive.  Just rebuild your Windows 7 machine and you'd get that effect.


This is the very reason we do need benchmarking.

The main point is that my reported latency went down by half of what it is under Windows 7

 
This matches up with that John was saying - one person's combination of hardware, drivers and software may not give the same results as another's.  So in your case it's a positive, even if it's not for someone else using a different DAW, hardware and drivers (like on the Gearslutz thread).  
 
But this is no surprise in my book since it seems just like everything else in DAW/PC land...
 

FWIW, Unless you like testing... you'd be well advised to sit out the first 6 months or so of Win8.

 
Jim, that's far too sensible an outlook to be posting here.
 
One should always mess up a perfectly working system in order to migrate to a new OS at the earliest opportunity.

re: benefits of bench testing
I bought a Stealth PC from Vin almost 2 years ago and shipped him my Focusrites and he bench tested and configured the system.
The result- i have been unable to crash the Sonar ,not once.No glitches or dropouts-nothing-absolutely reliable.
(but then i`m still on 8.5.3 and only recording 16 tracks live @24/48-hope i`m still welcome here)
To me,that was worth every single extra penny and my personal opinion is to buy or upgrade to the best dedicated pc you can afford from Jim Roseberry etc or Vin if you`re down here and have them personalise it for you (bench test?)and remove all margin for error and conflicts and save a lot of grief and visits to the forums. 
with respect
tony

hey mike,
i should come over and see if you can find me an acceptable X2 GUI to convince me to upgrade . 
cheers

Bitflipper, I am not a disgruntled former anything Mate !!

Hi, Vin. Sorry if I mischaracterized you. But this is your first post here since 2008, and you did write on another forum that 

"I will no longer be participating over at the Cakewalk forum tho , there are some entities there that I really cannot tolerate , plus the underlying vibe of the forum is not one that I want to be part of."

I interpreted your long absence as the basis for my "former" adjective, and your unhappiness with this forum as being "disgruntled". My apologies if I exaggerated, and I hope I haven't become one of those entities you can't tolerate!

I had actually posted this thread originally to get a feel for whether I should upgrade from xp to win 7 or 8. I appreciate all the responses to that original topic but was hoping someone from cakewalk might chime in since the gearslutz thread included a chorus of voices that implied that cake tends to exaggerate performance benefits of each new windows OS. And many of those posts were about as slammy as you can get. 

Is the fact I joined the forum in 2008 relevant or is it just coincidence that people left then?*






*No need for the head boy to answer that.

Funkybot


Where tests like that are helpful in that you may be interested in 1) seeing how efficient Sonar is, 2) such as seeing how your DAW's load balancing compares to other DAWs (the dreaded core 1 is spiking the others are nowhere near 100%), 3) determining if there's another DAW that will allow you to squeeze a few more plugins/tracks into your mixes (this would be helpful if you had a fast PC but were still maxing out your CPU during large mixes). It could also help companies like Cakewalk in identifying where there can be room for improvement, by someone else doing free performance benchmarking for them.


It's completely legit, and if you read about their testing methods and how they distribute the plugins...I think it's a fairly unimpeachable method of comparison. If you also read his criticism of Noel's testing method, his points seem entirely valid. I'd love to see Noel's response to that.

What a test like that doesn't show is how well you interact with your DAW, it's workflow, and it's features. These are clearly the most important aspects in picking a DAW. Sonar still some has some things that drive me nuts (limited routing options, the terribly huge, non-resizable mixer, no varispeed, horrible notation) but overall, it's still the best DAW choice for me. I've yet to find the perfect DAW, and every one out there seems like it's good at some things and not so good at others, and overall Sonar has become my main choice. So to those who point out that the benchmarks don't address this aspect, I agree with you 100%. 

But that doesn't mean anyone should just blindly ignore the test results because Sonar didn't come out on top. At that point, people are turning DAW preference into a psuedo-religion by ignoring the science. I think we can all be a little more rational than that. 


I honestly don't think the people saying the test is completely irrelevant would be too upset if Cakewalk came out and said, "hey, we improved Sonar's efficiency by up to 25% in large projects by making adjustments to how we handle load balancing" or something like that as a result of benchmarks like these.

Exactly!


Much props to Vin for all of his contributions... I can see why he hasn't posted here in so long lol

bitflipper


The GS posters even have at least a bit of a case for accusing Noel (whom they inexplicably keep calling "Neil")  of being a Microsoft sock-puppet. He does diplomatically sidestep talking about Metro, even though I'm sure he dislikes it as much as everyone else. He's said that future builds of SONAR won't even install on XP, a foolish and unnecessary move that only makes sense if you assume he's taking orders from Redmond. 

Even though I am skeptical that Windows 8 offers anything over Windows 7, I have to respectfully disagree with your statement regarding the drop of support for XP.  This is an operating system that was released 11 years ago.  Yes it was stable and while Vista was the only upgrade path, people were right to maintain support for it.


But just how much resource do you give to supporting 4 x 32-bit operating systems + 3 x 64 bit operating systems (if you don't count XP x64, which was only an OEM release).  And how far back do you support?  How many beta testers do you force to use an out-dated OS just so you can test it?  IMHO, it's entirely reasonable to drop support for an OS 11 years after release.  Was anyone developing for Windows 3.1x 11 years later?


Many of the Brand-name computers I deploy at work don't even have XP drivers available.  There comes a time when you draw a line in the sand and move on.  If you want to see Sonar progress, you have to cut ties with the legacy OS.


Having read Noel's article, he certainly does not "sidestep" the Metro I/F.  He points out the limitations of trying to develop a Metro audio app.  Implying that he's "taking orders from Redmond" is a little insulting, but the truth is - every Windows developer in the world has to ensure their software will work with the software coming out of Redmond.