Latency is unavoidable. All you can hope for is to reduce it to a point where it's no longer a distraction. Depending on the instrument, 5 to 10 milliseconds is usually acceptable.
Most of the latency is caused by buffering data coming in and going out of the computer. It takes a finite amount of time to fill a buffer, based on your sample rate, which determines how fast data is moving, and the size of the buffer. Raising the sample rate therefore reduces latency, as does making the buffers smaller. On top of that, many plugins add their own latency to the total.
So to get the latency down low, you can do any of the following:
- reduce buffer sizes
- increase the sample rate
- avoid using high-latency plugins
Of course, there's always a catch-22. If latency is reduced too much, your CPU won't have enough time to process data, and you'll get dropouts. So there's always a limit to how low you can get your latency before the computer runs out of steam.