"Standalone...guitar lovely, but in a mix..."
Try playing the mix with all muted cept guitar. This may go quicker w 4-6 busses steada individual trax. Guitar lovely, yes or no? If yes, you captured the lovely you heard on capture/setup, if no...something's been lost and you have to figure out what.
Unmute vox. Guitar still lovely, yes or no? If yes...unmute the next most likely competitor...keys if present, bass if not. You know where the general freq overlaps occur. When you find the intruder, start w its fader. Bring it in looking for two things, where the masking begins, and, if possible why. Whats going away in guitar, or stepping on it from another source?
Some usual key elements to a lovely guitar are:
1. Low end, the chunk from making a big cone move big air.
2. Phat mids.
3. Upper mid ring.
4. Attack, pick noise, chicka chicka (on rhythm)
5. Sparkle.
7.Nature of the fuzz if overdriven. Only you can define the proper mix of crunch and fizz.
8.Post...verb, delay, mod. I've had guitars that worked a ton of different ways till I lost the verb tails in a mix, which killed it.
Once you know whats going away, and who is causing it, you're on the road to a pair of objectives. First, priorities... maybe the offender is more important to the message than the guitar is. Second, hazy outlines what to do about the conflict. Placement in stereo spread, dynamics on one or both, eq, timing, what takes priority when, how many element's is the listener being asked to focus on at a time? Who is the boss, measure by measure, who can carry the piece alone at that point, indispensible, and is that part virtuoso enough to carry the whole tune or is evolution required? Is this measure a solo, solo w backing, a duet, (back and forth, or blended) a gang fight, team on team, (blended or back and forth), or a climax, all in, all they got?
If your in the bridge headed for climax, say, guitar bass and drums back n forth w vox, and the vox are too strong, all it might need is the tiniest bit of click/chicka from guitar to get back the balanced tension, and you might just get that carving a few db outta the 5k range on vox for more pick noise on guitar, and lose some nasty vox artifacts in the process, and at the same time, that might kill the mood too.
With guitar and vox both popular cuz they tag peak human frequency sensitivity, something that's worked for me is to pull unpleasant vox honk 700-900, and see what guitar I can narrowly slot in the gap...how can I tailor distortion and/or verb tails plus dry guitar in a niche where vox isn't working? How can I max that edge with additional small increments of pan or verb differential? Real important for me to keep a close eye on how I did this, cuz I might just want to swing focus back and forth as the message develops.
One more common one to look at, guitar verb.They often set their own, and it can fight w the common room you set the rest of the piece in very easily, especially if you are adding room to an amp's spring or plate. Recipe for mud there.
How do you know? Two verbs mushing out, or guitar crunch negatively interacting w vox? Two options...get lucky, or isolate these individual components by seperating them and A/Bing. You effectively have unlimited trax, clone buttons, eq, and mutes, once you find the broader areas of contention w busses. Then you have unlimited tools to address the conflicts.
In fact, the infinite is the enemy, too many options, unless you harness and group them with a systematic approach, planned in advance, and executed without getting lost in deep space. A hardcopy checklist is my goto, the first time I end a resolution session without finishing the original plan or solving the problem.
Good luck!