Totalmix is awesome once you get it figured out. A few things to remember that are not obvious...
All Inputs are reflected into software on your computer/Sonar. What I mean by this, you don't select an "output" to affect what goes into the computer. Just select any input and the controls you see are what is going into the computer sans FX unless you have loopback on which I don't recommend to begin with.
So to set up to record in sonar it's as simple as selecting the inputs you are using and setting the gain/phantom power/etc. and set your input on the sonar track to that input and you're all set.
To monitor via headphones (no latency) connected to the RME unit, or to use the main outs... Select the output you want to use. Click on the headphone output you want to use, once you click on it all the other settings/levels you see are what's going to that headphone output, your headphone mix. Then select the main outs, you may have to click the "assign" button to assign output channels to the mains, once you do that when you select the mains output what you see on the other channels is what is being sent to the mains.
Click back and forth between the headphone output and the mains output. You'll see the other faders change as it changes from what is mixed into the heaphone output and what is mixed into the mains output. They can be completely independently mixed and have nothing to do with what goes into the computer. What goes into the computer is selected by selecting each input channel and manipulating it.
Hope that helps. gabo