MickeyD - your TOTAL Roundtrip Latency will still be high enough to cause some latency issues. The value you reported (5.6 ms) is the value for input latency or output latency.
Running at 128, MY input and output latency values are each 4.7 ms, BUT my TOTAL Roundtrip Latency is 9.3 ms.
While what you reported is part of the equation, but it is the total roundtrip latency that needs to come down to around 10 ms or a bit less.
Bob Bone