The process of identifying and removing latency-causing hardware and/or software is one giant pain in the rear, however these issues are well-reported on the web and are usually solvable.
I had a laptop that had massive spikes occurring quite frequently and were up in the 2800 range. These had a disastrous impact on sound quality, and it rendered the laptop unusable for any music performance or production, despite having a solid-state hard drive, optimized memory (services carefully selected/disabled as needed) and 4GB of memory. I was also using an external audio interface.
The issue ended up being a hardware issue where the BATTERY for the laptop was somehow causing these massive spikes. The INSTANT I removed the battery, the problems went away.
The point is that there a a number of causes to latency issues, but that solving these issues is generally a linear process, and that there some tools and tips available to detect and assist in resolution.
1. Make sure you have as much memory available as you can
2. Make sure you carefully evaluate loaded applications (antivurus, other startup loaded aps), for potential disabling during music performance/recording sessions.
3. Check latency, using free tools such as DPCLat and LatencyMon.
4. Carefully look at the Windows services configured to run on your system, and if needed try to pare down some of the services that don't need to run during music sessions. Things related to network services and print spooling come to mind - there are lots and lots to go through. Be VERY careful not to disable something needed to run Windows properly. If unsure, look up the service names on the web to determine whether or not you can safely disable - and also to find out what some of the services do.
5. Use an external audio interface to have A/D and D/A conversion processed through dedicated hardware, as opposed to making your computer's CPU do all the work.
6. Look on the web for tips on optimizing computers for audio processing. There is a lot of information out there on this.
Good luck with getting things set up and working without audio problems.