OK. At the bottom of the Software Playback channel you're using will be something like 'AN1/2” in white.
The Totalmix channels and outputs you have available will be different to me, because I've 12 ins and 12 outs in the interface itself plus two phone channels and the digital ADAT etc. channels as well.
Now, at the bottom of the Software Playback channel you're using will be something like 'AN 1/2' in white. The numbers may be different, but they'll probably be the main analogue outs that go to the monitors.
If you click on the white letters a menu will appear. This lets you select where you want the signal from that channel to go and you can select as many entries in it as you want. Active routes have a small tick beside them. One of those entries will point to the headphone outlet you're using. Select it and the Software Playback channel fader will now control the volume being sent to the headphones output. I suggest turning it to zero and forgetting about it, controlling the volume from the headphone output instead.
Unless you untick the entry for the main puts they will also be active.
Generally I leave software playback channel faders at 0 for the ones I use and at the bottom of their travel (-infinity) for the rest. I have software playback 1/2 routed to the UFX AN 1/2 outs, three running to hardware outputs so I can process audio from the computer through hardware and back in again and one software playback channel configured to the phones output.
I also set up a DAW output bus that goes direct to the software playback channel that feeds the headphones, with Sonarworks and Waves Nx inserted on that DAW bus, which is the one I use for headphones. That's if its just me doing stuff. If anyone else is involved things can get more complicated :-/
Totalmix presents a huge number of available channels and available routings, it's designed to accomodate pretty much anything anyone in a studio of any size might require of a digital mixer such as lots of individual foldback and headphone mixes . I find it helps to get your head around it if you take a look at the routing diagrams in RMEs' manual and draw your own diagrams of the routings you want and channels involved then work through them in the software until things work.
I've been using Totalmix for a while and sometimes still find new ideas for what it can do.