Well....
It has to be listened to, as part of the job is spotting errors, it turns out (I didn't realise this initially).
However, you can scroll through the wave and easily identify breaths visually, so the initial removal step can be quite quick. I've found that on a half hour wave, I might miss two or three breaths, and might delete two or three things that look like breaths but aren't. So I've got that part down fairly quick and error-free now.
Second step is just playing the thing back and tweaking any timing issues that appear, and making a note of any differences with the text.
I think I can get it down to being 2.5 to 3 times as many work hours as the length of the original material. Not knowing anyone else who does this, though, I've no idea whether that's good or bad.