What I found is things like Hi Hat and such work fine if you match them with the correct sfx sound files. In Session drummer I loaded the Yamaha Birch kit and it responds to my Yamaha DX drums controller info.
But most of the samples are not like this.
Sometimes I'll even record the audio output of the brain, after I edit the midi track, as it seems to have the best "natural" drum sound of all. You just have to like the sounds the brain has. But I end up mixing from multiple sources.
My go to drum set up has been TTS-1 for Kick
Session drummer 3 and the Yamaha Birch kit for the rest.
Then I use all sorts of different snares including the DX brain.