My pc case is a rackmount so I can't put the big ones in it.
I bought a coolmaster 120mm and the rad is too big for the space available.
Any suggestions on what i can use? The cooler I'm replacing is the Noctua NH-L9x65 92 x 92 x 14mm, 92 x 92 x 25mm and my cpu temps was still up at 76C and it would start throttling.

A portion of the issue may simply be air flow to/through the rack and how things are loaded. If units are below the PC the heat from them will also rise. Dust on heat exchangers can also contribute, but if the rack output is hot, it is not getting proper air in/through.

I am loathe to say this, since a DIY solution is not recommended, but enclosed electronics are often liquid cooled to a remote heat exchanger outside of the unit. If you can put the exchanger outside the rack, a pre built with long enough hoses may work, but those hoses are rarely very long, and I would not try to modify them.

I would start with scrutinizing air flow and sources of heat in the rack first. The current cooler "should" be adequate if you can replicate its intended working environment.

I was a network manager, until I retired.  We had racks of equipment and mostly closed back racks.  We had plenum air (raised computer flooring). If the cool air at the bottom is both not enough and not cold enough the equipment in the racks could/would over heat and the further up the rack the higher the temperature.  
 
I suspect its even worse in a control room or wherever the equipment rack is in most recording studios. Very few have raised floor plenum air, just the ambient air of the room.  It must be cold enough for the equipment or whatever you do to the PC for CPU cooling isn't really going to help.

There was a plate on the top cover (140mm) I removed the plate and added a fan to push air out, the fan on the back draws air in, so I think I've dealt the airflow issues. Temps have come down but I'd still like to get them down a bit more.

If the intake air is not cool then no amount of fans is going to substantially help.
 

I bought a case fan , this time having it draw air into the machine instead of out. Temps have reduced significantly, even under a project with heavy load. It's Noctua 140mm runs at 1500 rpm and my temps are now36C at idle and 53 under a heavy load project with lots of plugins and 44 tracks. The max temp of the processor is 61C so this is great.
 
Funny thing is Latencymon still says my computer will have probs with realtime audio even though my test track was run at 256kb so looks like I could still make improvements but don't know what, or know if i need to after this test.

Latency Mon can be very picky and will overload you with data.  But it is a good tool. 
Exactly what is it telling you?  One thing I found, on my systems, is to disable the Internet when recording and that reduces or reduces the chance of it interfering (and causing spikes in Latency Mon).
 

ok let me try that, I think I've had the internet on every time I put latencymon on.

If you're using WiFi, disable the WiFi adapter... and run Latency Mon.
WiFi adapters often have poorly written drivers (cause high DPC Latency).

fireberd
Latency Mon can be very picky and will overload you with data.  But it is a good tool. 
Exactly what is it telling you?  One thing I found, on my systems, is to disable the Internet when recording and that reduces or reduces the chance of it interfering (and causing spikes in Latency Mon).
 

Case in point: I just happened to be running a check-up with LatencyMon when my AV decided to run a scheduled virus definition update.  My PC suddenly slipped from "all good" to "not so good".
 
Watch out for that internet, and especially scheduled tasks!
 
So my point is, even though my internet was on, and latency was all green, that can suddenly change when something starts using the internet in the background.