When running at higher sample-rates, many audio interfaces (including the Roland units) double the minimum ASIO buffer size.
The short answer, you won't be able to go below a 96-sample ASIO buffer size when running at 96k.
RME USB units are known for low round-trip latency... and rock-solid drivers.
The Fireface UFX's smallest ASIO buffer size (when running at 96k) is also 96-samples.
Note that the UFX yields ~4ms total round-trip latency at this setting.