My band is recording our first demo, and anyone who has had to mix extreme metal knows how difficult it is to get everything to cut through. We actually have a female vocalist who sounds great at practice and when I recorded her parts. When I went to compress, EQ, and level her track with the song, I noticed certain parts of her scream were getting lost or not sounding defined.

Anyone have any tips? I have done this before with male vocalists and though I have never been 100 percent happy, I have never had this issue. Any tips would be greatly appreciated.
 
Track currently has these effects:
 
Sonitus Compressor>CW Parametric EQ>Reverb (Very moderate)

This is by no means pro advice and is probably very wrong but I used to just go right into the track and turn up the parts that were to low in the mix (split the parts up and turn up with volume/gain automation). This was for female metal vocals as well. I found when they were singing lower pitched parts it lost a lot of the oomph. I think it's just the physical nature of the female voice. The shrieks cut but they start losing power when doing lower, growly parts. I always managed to get things evened out pretty well without any effects. You might want to try panning the guitars ever so slightly because those frequencies might be competing with the vocals in the middle. I'm not very knowledgeable about that process though so again I could be wrong. Cheers.

Apply a volume envelope to the vocal, getting all the quieter sections sections to ride just above the music, then apply compression to taste.

And yeah, I'd imagine it would be a lot better to use automation nodes instead of chopping up the track but I didn't know any better back when I was doing this stuff. In fact I'm not even sure my ancient version of Nuendo worked that way.

Great tips! I am going to go home and try it. If I use automation nodes or a volume envelope, does it work before or after the compressor I have on the track?

volume automation before the comp will influence how the comp reacts to the vocals, which you should use to your advantage by pushing the scream so that the compressor is consistently working it. after the comp will be less useful but isn't necessarily wrong. but typically the goal is even levels post-comp so i'd use a clip gain envelope (if memory serves) which is pre-everything.

my guess is you are looking for a lot of "white-noise" in your scream, a good way to do that is double the vocal and either spike the mids or bandpass pretty aggressively so that the mids are prominent, compress the crap out of it and lay it under the original track to taste. play with the freqs since it is a female vocal but i think you are looking for something around 1.5-2k. cut other tracks in the range you like so it comes through better. electric guitars tend to sound fairly terrible (vacuum cleanerish) in that range so it could work out pretty good.