Amp Sims have a tendency to be too "perfect" or hot at higher frequencies. A LPF between 4-7K before compression is another item that may help.

So much of tone is personal preference so is impossible to give you a slam dunk reply.

Another learning tool is to take settings to extremes one at a time so you can easily recognize what is really going on. Swap order of modules in chains, etc.

mettelus
Amp Sims have a tendency to be too "perfect" or hot at higher frequencies.

Not sure what you mean here. FWIW, Most amp-sims suffer from aliasing, which becomes apparent and annoying with higher gain settings.

I'll certainly run compressor plugins on bass, whether recorded using an MXR bass preamp, Sansamp or using a software amp sim. Which seem to perform better than most of the guitar amp sims for me.

I've done it with guitars as well, often the BozDigital 10dB compressor which I find sounds pretty good with guitar.

The guitar compressor pedal I use is a Pigtronix Philosopher's Tone, mostly for playing glissando or after a Qtron envelope filter which pushes out some high gain peaks that need evening out a bit before they hit other pedals or the amp. I put a Keeley set pretty mildly in my bass setup to control filter peaks and resonance as well.

I use amp Sims frequently because it is convenient. I'm under no illusions that it will sound close to a real amp with real Mics.

As far as trying different plugins... try them all. In every order. With all settings. I'm serious.

The more you play the better you will get. It is sometimes interesting to try to duplicate other sounds, but personally, the results I enjoy the most are the ones I invent because I like the sound.

I researching gain structure for TH3, a day latter you appeared in my recommended videos on Youtube. Was helpful to see you getting to grips with it.
 
I enjoy just messing with a clean loop I've played and letting it play while I adjust the settings, then going back with the new settings to see how it feels, for another take, then rinse and repeat

I just got a chance to watch Geoff's video above. Since I just got the TH3 upgrade myself, I noticed the input levels in that video are basically what I am seeing, i.e. I cannot drive TH3 above about 50% (in the yellow) before things start to blow apart (which is about -6dB in on my interface). Very mild volume levels will induce sustain feedback from PA speakers 15' away.
 
One thing that did come to mind with what Geoff is pursuing is have you looked at EQ matching (iZotope Ozone or similar) to look at the differences in the mics? For things that are identical otherwise, it will yield some insight into what each microphone is doing. As far as an amp sim goes, that is also a cheezy way to better replicate a recording (amped or otherwise) to what is coming out of the amp sim. Another method which I use as an intermediate step is to get midway between the amp and speaker (or run each in one channel on headphones). This will often get "close enough" alone from tweaking the amp sim alone, but there is often a few subtle +/- 1-2 dB areas floating around that an EQ matcher will point out.

In Overloud's Markstudio, changing thie mics seems to to make small noticeable difference (I keep finding myself on the ribbon mic)

Real ribbons on real amps are great for recording loud guitars. I believe the California 121 is the Royer 121 ribbon in the Overloud simulators.

KamikazeI inclined to agree with TLW, I don't see what would be so special about stomp box compressor apart from it's portability and ease of use on a stage. Which isn't the scenario here. If I want a fast compressor before a sim, it seems I have some better tools in sonar for the job.
 
Being players often do put stomp boxes before an amp, I don't see why it would care if it's from an external VST, or a purpose built stomp that comes with the modeled amp.

I didn't say that you can't use a general purpose compressor before the sim.  My point is that if you are looking to emulate a classic tone that was achieved with a specific stomp box, you'll probably find it easier to get that tone using the stompbox emulation rather than a general purpose emulation.  If you are looking for something different then, by all means, you'd want to try a different compressor.  Again, there aren't any rules and there isn't any "better" other than what you think sounds better.  
 
The real world LA-2A, stompbox compressors, etc. are analog circuits that implement the theoretical concept of compression and as such have imperfect audio characteristics beyond their theoretical purpose.  The available controls differ and that is going to affect what compression characteristics you can dial in.  The detector circuit design is an obvious example of something that is going to differ drastically and affect the response (a big part of the allure of the LA-2A in the first place though that might not be the response you want for a specific guitar tone).  An emulation of a battery powered stompbox isn't going to have the 50hz/60hz noise that an accurate LA-2A emulation will have.  That's why emulations exist rather than everyone just using something like the Sonitus compressor for everything.  I'd suggest trying all three all over the place to understand the difference (a stompbox emu, a studio emu, and a "purer" compressor).  These concepts apply to any other effect -- not just compression. 
 
Of course, all of this depends on the accuracy of the emulations.  I wouldn't be surprised if some emulations are just purer compressors with controls slapped on to work like the "emulated" hardware.   
 

I hadn't said anything about emulating a classic tone though, just getting a better one. My question was also about improving the tone of my bass and guitar before hitting a an Amp sim.