Hello James, and welcome to the forum!
You can't be completely rid of latency, but it is possible to get it down below 5ms, though you probably won't be able to run very many softsynths before the audio starts getting distorted that low. 10ms is more the norm, I think. I from 10ms to 20ms is playable but beyon that you really run into what you've already described.
What you need is a really good audio card combined with excellent audio drivers. M-Audio has worked really well for many people.
Also, tweak out your machine so it is not running much background stuff while you're trying to play live.