What do you mean by 2ms latency?
a 128 sample buffer at 44.1kHz is 2.9ms and 128 sample buffer at 48kHz 2.6ms.
Are you getting clicks and pops at 128 samples?
Or are you talking about roundtrip latency and running at 32 samples?
Or are you at 96kHz and speaking about a 256 sample buffer?
It's hard to have a perspective when the only thing to consider is "2ms".
It could be that you are at the very edge of performance capability and the 64bit driver is simply choking before the 32bit. Or, you could be out in a comfort zone where it should work great and this could be an indicator of a very bad driver.
It's hard to consider where you stand without some perspective... and it will be easier to understand if you state the latency in terms of total round trip latency or in samples so that we can understand how aggressive your settings are.
best regards,
mike