Its most easily understood by an example.
Latency is always caused by buffering. Lets say you have a 512 sample buffer size.
At a sample rate of 44.1KHz this translates to a latency of 11.6 msec.
However assuming the same buffer size but a sample rate of 88.2 KHz this translates to half the latency which is 5.8 msec.