For the same buffer size the latency is lower at higher sample rates. The reason is very simple.
Assume you are using a buffer size of 64 samples.
At 48K this corresponds to a latency of 64000 / 48000 = 1.3 msec
At 96K this same buffer corresponds to a latency of 64000 / 96000 = 0.6 msec which is half that at 48K