All "dedicated" boxes seem to be written on Linux nowadays. Check the "Steam Machine".
> but since Linux is the worst mainstream OS for audio that would be a bad choice. That's a heck of a statement, "worst" meaning?
I totally disagree the audio engine is probably the best, drivers for it are pretty good nowadays. It has the worst support perhaps but DAW's for Linux are popping up all over the place for the reason I've just explained.
I don't see any point in creating an OS, an existing stripped down OS is what you need. Linux is perfect for the job the kernel can be recompiled to more or less suit anything.
You could strip down Windows but you still have the HAL to contend with, and Windows fundamentally flawed design for audio (which apparently might get fixed in Windows 10 one day).