I typically record with my ASIO Buffer Size of my interface set to 128. I could go lower, but since I don't hear any lag at that setting, I leave it there and it just works, wonderfully.
When I finish recording and move on to mixing, I then raise the size of my ASIO Buffer Size to either 1024 or 2048, so that I can load up a bunch of effects, some of which require a large buffer to process signal correctly. Since I am not trying to record anything at this point, the large buffer will take a few moments to start playback, but that doesn't matter - having enough buffer size for the effects processing is what matters at that stage.
If you are still having audio quality issues running with a buffer size around 128 when recording, I would suggest you perform a simple test - hit the letter 'E' on your computer keyboard, and that will toggle a bypass of all effects. If your audio quality issues vanish, then one or more of your effects is likely NOT meant to be used during recording, but instead is meant to be used during MIXING, where a super large buffer setting will allow that effect to work properly. (After testing, hit 'E' again to toggle off the effects bypassing).
So, IF the above demonstrates issues with one or more of the effects - look for 'leaner' effects to be used while recording, and then swap them back out during mixing, for the more robust ones. Effects that add a bunch of latency or otherwise chew up a lot of CPU are the ones to watch out for - like a convoluted reverb, or linear phase types. (Perfect Space, Boost 11, etc.)
Bob Bone