Update: Thanks Mario, but I checked all those things before posting.
I think I found the problem.
I just disabled all the plug-ins (only 6) on the busses and it worked.
Then did a one-by-one and found it to be the TS-64 Transient shaper!
I just mute this and the latency goes away.
Does anyone have any idea why?
Not really much point in having great plug-ins if you can't use them!