There is no rule, except for the rule that larger buffers = greater latency. If latency doesn't matter, larger buffers are generally a good thing. Equalizers get more accurate, samplers place lighter demands on your disk drives, and the CPU is able to handle more tracks and effects. If latency doesn't matter, go large.
I don't know about VSL, because that stuff's above my pay grade, but I can tell you that larger buffers in Kontakt make a big difference if you've got the RAM to do it.
As for audio I/O buffers, mine are always set to the maximum value supported by the driver, which for my current interface is 2048 samples. My old interface offered 4096, so that was what I used until that device died earlier this year.