bitflipper
It could be due to the fact that ... the Fast Track Ultra ... is optimized for 96/24, whereas another audio interface may work better at a lower sample rate.
Dan Lavry has said as much. An interface designer has to pick a sample rate to design around, and might well choose 96KHz as that optimal rate. I would have expected a prosumer-class device like the Fast Track Ultra, being targeted at hobbyists, would have been optimized for 44.1. But who knows what their reasoning was?
Now that you mention it, I remember reading that article by him. There's a PDF of a paper he wrote floating around the forums somewhere iirc.
My latency is quite high when I set it to 44.1. IIRC, it's around 8 ~ 9ms at 128 samples. I checked their site to see if there was an optimal sample rate but I couldn't find anything.