The buffer size you use isn't the deciding factor although it's part of it when going for low latency.
The measured round trip is the crucial figure. You maybe able to run at 32 samples with no problem but the round trip figure will sometimes be higher than someone elses card running 64 samples. My old E-Mu for example would only go down to 96 samples but I could get similar latency figures as I do with my current USB device running at 48 samples buffer.
The only way to get the true picture is to measure the point the signal goes into your interface to the time it comes out the other end of your chain.
Not every one needs that low latency which can sometimes be at the expense of audio stability and increased CPU usage, for example if you generally use external hardware monitoring it won't bother you at all.
A low enough figure so you can track a guitar say through whatever amp sim you use to get your tone or play a keyboard through your favourite synth without the latency interfering with your performance because of the time lag is usually low enough for most people, although many people treat it like an Olympic sport and want to have the best figures for 'show and tell'.
Again that figure will depend on your sensitivity to the time delay. For me anything between 6-9 ms is OK anything below that is excellent and anything above that and it starts to throw me.