I keep my buffers at the maximum all the time. With my current interface, that's 2048. My previous interface could go to 16384, which was excessive so I never put it higher than 4096.
I get away with this because latency is not a concern. My process was developed long ago when computers just weren't powerful enough to bring latency down low enough, and still serves me well.
I monitor live audio input through the interface, so there's no appreciable latency.
I do not play soft synths in real time. I do, however, use a lot of soft synths and samplers and always play every part in real time on a keyboard. I manage that by using an outboard synthesizer that I can hear in real time. After recording the performance, I'll then re-route the recorded MIDI to a soft synth or sample library.