Here's what I've tried.
1st thing I did was to load a mixdown I've just received in both Studio One and PT and use a single Waves plug-in in each application, so that I could simply use the same factory preset for both the VST and RTAS version.
RESULT: files didn't completely cancel each other, and I could hear and see the frequencies dancing on the meter starting at around 2 k. But we're talking about lower volume stuff - peaking around -50 db.
2nd I loaded a stereo instance of Superior Drummer in both PT and S1, picked up a midi loop and bounced.
RESULT: difference was much more obvious this time - frequencies registered everywhere on the multimeter and as loud as -30, even if the actual audio files were much quieter than the mixdown I tried in my first test.
I assumed that this was maybe due to the fact that the same midi sequence would not necessarily always trigger the exact same samples every time it played back through SD.
But I did the test quickly in Logic using the Audio Unit version and rendered two copies of the same loop - the two AU bounces completely cancelled out.
I then compared the Audio Unit bounces from Logic to the Audio Unit bounce I had done in Studio One. The difference was just as obvious as w/ VST or RTAS.
Finally, I went back to Studio One and bounced both a VST and a Audio Unit instance of Superior - the cancelled each other perfectly.
So I would be tempted to think that the difference may have more to do w/ the host than the plug-in format. Obviously, this little improvised test doesn't take into account that one app may have a 64 bit engine or such. And it doesn't really tell us if RTAS sounds better or worst or if it's only the PT engine.
Unfortunately, I don't have FXPansion's VST to RTAS adapter so I can't try to bounce a VST in PT. But I'm tempted to believe they'd be pretty similar.