Latency only affects real-time input from the user. This is an important point that is often overlooked. It will not mess with existing tracks, offline edits or really anything except what you're playing or manipulating in real-time. Even then, a soft synth you're playing will still record the notes at the time you hit the keys rather than the moment you hear the sound.
So you can comfortably up the buffer size/latency for editing even if you plan to track more later at lower settings. Or, yes, you can ignore the warning. I've always done that.