Hi Tom,
The "X factor" with audio drivers is each one's hidden-safety-buffer.
All audio interface drivers have a hidden-safety-buffer.
A larger safety-buffer can help avoid drop-outs/glitches under less than ideal circumstances (ie: high DPC Latency)
But it comes at the cost of higher latency
Most audio interfaces don't allow you to adjust the safety-buffer (thus why I call it "hidden").
Some audio interfaces (Focusrite, Behringer X32 series, and the new MOTU ABV units to name several) allow you to adjust the safety-buffer's size. This allows the end-user complete flexibility to decide how to use the safety-buffer.
In the case of the MOTU Ultralite AVB, that 4.9ms is the result of using the smallest safety-buffer size... and the smallest ASIO buffer size (which is 64-samples).
Don't throw the Babyface out with the bathwater!

The ASIO I/O buffers *AND* the safety-buffer (combined) yield the unit's total latency.
Babyface: - Hidden (fixed) safety-buffer is a littler larger than the MOTU's (at the smallest size), but it can use a smaller ASIO I/O buffer size (48-samples)
MOTU Ultralite AVB:- Safety-buffer can be adjusted to a little smaller than the Babyface, but the smallest ASIO buffer size is larger (64-samples)
Total round-trip latency of the Babyface and Ultralite AVB is identical (4.9ms at 44.1k).
Both allow you to run substantial loads (glitch-free) at 4.9ms total round-trip latency.
Note: The computer has to be able to keep up with the load...
Both are great performers.