I've done a bit of recording of metal guitars over the years and there's a few things I've learned (sometimes the hard, expensive way)...
I'm going to be repeating a few things that you already know and stuff that people have linked to but there might be something in here that might twig something and have you go "hey...... let me try that!" Apologies if I'm retreading too much stuff you've already tried!
First, what you may think is a great guitar tone by itself could sound like garbage in context. A few people have mentioned guts coming from the bass or kick drums - that's certainly a valid way to do things. You'll find especially that when you start layering guitar tracks, the mud build-up in the low end gets a bit crazy and can stomp all over your other instruments. The same goes for other frequencies. The tone might sound super heavy scooped but will you hear it in the mix? You have sizzle but is it fighting with the hats and cymbals? One of the first big productions I ever did was a great learning experience. I got THE guitar tone in isolation, but in the mix it sounded like complete rubbish and I couldn't work out why. It was either overpowering or gone entirely and that's because of the clashing frequencies and all that was left was sort of empty.
My advice there is to get your drums and bass EQ'd to sound good first. These things may need tweaking later, of course, but at least ball park dial them in first. Then bring in your guitars. Pan as you like (I prefer just 2 tracks of hard panned guitars but I've worked with 2 on either side and rarely there's been ones in the middle too).
Listen carefully to what they're doing with the rest of the mix. When you have them cranked, can you still hear the bass? Are the kicks getting lost? If you drop the level a bit, do they still sound good or are they thin? This is telling you that they need a lot more EQ work if things are disappearing and you can't seem to find the right level.
Be prepared to aggressively cut the low end to get rid of the mud, or if that's taking too much away from the tone, run them all to a buss or aux track and throw a multiband compressor over it and set one of the bands to only trigger on the mud frequencies, so everything will still sound pretty thick until you cross the line with the big chugs. I'd suggest EQ first before diving into anything like this, however.
How is the high end? If you're using amp sims, they're notorious for having this fizzy plastic sound. You can fix that by aggressively rolling off the high end, much like a mic'd speaker would do. Not getting enough cut? Throw some more mids at it. By itself this will sound throaty and thin, but in context I'd wager you'll get a lot more definition. Even high gain guitar rigs can suffer from that fizzy high end syndrome, especially if you're relying on the distortion coming from pedals or the preamp section rather than belting the output stage, something you can't really do so much in a small apartment.
And if you DO crank it, are you getting comb-filtering or phase cancellation off of other things in your room? Your environment can play a big part in what the mic actually hears.
Still not getting the definition you'd like? That brings me to the second point: back your gain off a bit, or layer 2 sounds with different gain settings.
Super heavy guitar tones sound great when you're on stage and ripping out, but in a mix, especially when you're layering a few tracks together, all of the articulation tends to get lost. Dial back your gain and then layer a few takes of the tracks together to fatten it back up again. Again, in isolation it'll sound wimpy, but layered and in context it'll make a big difference. Missing some guts after that? Do a layer of backed off guitar and then do another one cranked, but have the second track much quieter than the first one. Repeat for "guitar 2" (ie: another "backed off" and another "cranked" track) and pan left and right. Hugeness awaits!
Obviously it's crucial to get this tight, and it needs to be played well to make it work, but these tricks will give you a lot more definition and guts in context. Refer back to EQ / Multiband compression if you have problems with mud after this.
Finally, you can most definitely get some great tones out of amp sims, but they're not real amps and don't respond like one. If you approach things like you're recording a real amp (aside from using a Kemper, or AxFX or something) then you're gonna have a Bad Time. Treat them as what they are: understand that because something says "JCM800" on the patch that it will likely sound nothing like a real JCM800, but if you like the tone then who cares what it's supposed to be modelling? Trying to emulate someone's rig with a plugin is a lesson in futility. It'll get you part way there but take the sim as its own unique entity.
I'm sure Ola would have some killer advice though - he's an approachable dude and you'll always find him around the place offering advice to people looking for tones. I'm not huge on some of the tones he pulls personally, but what he does, he does extremely well and it works great for the style he's going for. Hit him up!
Anyway, I'm sure I've reiterated a lot of stuff that was already tried/said, but hopefully someone finds something useful here. Good luck!
