Hi. Are you using the Focusrite ASIO drivers? I believe that while recording, or when projects are reasonable size, you could get lower. Try going down to 256, or even 128 during the recording phase.
You do not need it to be that low for mixing. For that stage, you can / could up the buffers to as stable as needed (1024 and up, if necessary).
Best of luck.