You should be able to tell if it's processor related or not by looking at the CPU usage meter on the control panel. As soon as one or more of the cores start peaking into the red, you're going to get pops and clicks.
The latency delay could be related to plugins that use "look ahead" technology, for instance the Concrete Limiter. I would try turning off your effects one by one to determine if any particular ones are causing the latency.