I never change my buffer. I set it at the lower medium end and just leave it there. On the Tascam and Scarlett it's the same which computes to about 24-27 ms RTL. I think this is 256 ms but Scarlett shows this as 7ms and the Tascam calls it Medium lowest.
The reason is I don't use real time effects while recording, and at this setting I don't have problems with recording midi. Only time I change it to 3 on the scarlett is when tracking digital drums,, but even at that setting I still hear an echo so for me there's no point changing.