Sharke, I totally agree with you. I'm held back from doing lots of mobile recording in Linux because it's hard to control my UCX without TotalMix. I love the idea, but I really must have TotalMix, so I use Windows.
My UCX also has FX processing internal (I'm thinking the BabyFace may not). That is so cool, to be able to EQ and Compress every channel separately as well as use Reverb and never leave the device. The latency is amazingly low that way. I don't dare guess how low.
Really the latency introduced by the device is getting in and out of the OS, not the conversion itself. At least for the RME. The actual conversion to digital and back has really no audible latency that I can hear. (Again, I'm talking about staying in the RME Device and going directly out, without going through the computer - once the computer is introduced, all the usual latency stuff appears).
BTW, Sharke, have you installed DigiCheck? It's an absolute must have. Check it out!! You can record with it too!