9 ms round-trip sounds about right with 64 samples with the US-20x20...there are USB safety buffers that add several milliseconds. FWIW that's about 4 ms more than Thunderbolt on a Mac, for a lot less bucks...in any event, for me 10 ms is the crucial dividing line for real-time playing. Playing guitar below 10 ms feels good; the feel becomes progressively worse about 10 ms.
Also note that the US-20x20 uses class-compliant USB 3.0 drivers, i.e., it communicates directly with the host rather than through chipset drivers. I believe this is what makes it more or less independent of the chipset (remember that with Windows 7 and before, there wasn't native OS support for USB 3.0 so you needed to install a chipset driver for your USB 3.0 chipset). I don't know whether a custom driver could lower the latency further; that seems like a possibility. However the tradeoff might be some chipset incompatibility issues.