Hi,
Just wondering if you ever worked out a solution to your issue above?  I recently upgraded to Sonar 8.5.3 64 bit (from 32 bit) and am experiencing the same symptoms with Waves GTR3 v7 (just opens up in a small grey box when running as a plugin).  I've disabled x-ray, closed and re-opened the plugin, updated video drivers, used shell2vst.exec - all to no avail.
Was wondering if you had some other suggestions.  Thanks!