thanks for all those great replies. Just to be clear, when you say: "But when mixing, I raise it to 1024 to reduce CPU load and save it for the plugins."  I'm just trying to get a clear picture of what you mean by "mixing." Do you mean you let all the track play and, as it's going, your record (or automate) volume changes to any track or group of tracks?
Or, do you mean something else by "mixing."
Thanks.
LNovik

Excepting the creative stuff that has to happen first, the typical process of converting those sound and lyric ideas into a complete song is: record, mix, master. My reference to mixing above was to the second phase of the process. There's a ton of great info on the web on this subject, but I would highly recommend two books by Mike Senior. They're not cheap, but they are definitely worth the money.
 
Recording Secrets for the Small Studio
https://www.amazon.com/Recording-Secrets-Small-Studio-Senior/dp/0415716705/
 
Mixing Secrets for the Small Studio
https://www.amazon.com/Mixing-Secrets-Small-Studio-Senior/dp/0240815807/
 
Long story short, though...
 
Recording is capturing your musical performances on "tape" (digital today, but I sometimes miss my old JVC 4-track reel-to-reel).
 
Mixing is combining each of the individual performances (drums, bass, guitar, keys, synths, vocals, etc) into a cohesive whole. That's when you use most of your heavy CPU plugins (Linear phase EQs and compressors being hogs, for example; OK, they're typically used in the mastering phase, but I'm just making a point).
 
So, when tracking (recording), where you want "instant" feedback of your playing, you set the buffer small (128 or 64). This gives you low latency, at the expense of tying up the CPU processing the small buffer.
 
When mixing, where you don't want glitching in the audio you're listening to, you set the buffer larger (2048 or 1024). This gives the CPU a LOT more time to spend on processing plugin code instead of interrupts.