So oddly enough, when I add the soft synth as "Simple Instrument" it has the bad latency. I usually don't use this option as I want the more detailed control that the other options provide. In this case I just wanted to do a quick take to mirror an electric bass part (to see how bad its intonation was!) so thought simple instrument would be fine. So thanks to the last comment here I tried adding the instrument the way I normally do with "Midi Source" and "First Synth Audio Out" and tada - no latency - just like I'm used to on my system.
I looked at the original problematic track before deleting it, and it was not added as an audio track with soft synth inside. So not sure what the problem is but at least I have a solution!
Thanks all.