mettelus
I just tested remove silence and it does not perform a bounce, so the file size is unchanged. Simple test to prove to yourself:

1.  Create a project, insert the same discrete audio clip from the media browser (I grabbed an 8 bar loop and inserted it once at bar 3, then again at bar 30 - clip used was "DMS CPC PT2 05 E MINOR CHORDS 138 BPM.wav").
2. Check the audio folder, two discrete clips (these are identical, both 2401KB).
3. Select both clips and bounce to new clip (creates one file, size 24.122MB).
4. Save As (checking "copy all audio with project", and the 24.122MB file moves).
5. Run Remove Silence (I used default settings), clip size changes, but no bounce occurs (audio is still 24.122MB).
6. Save As (but this time check "copy all audio with project" and "Create one file per clip").
7. This save as renders new audio files (one per clip) (clip 1 = 6387KB (due to the silent tail), clip 2 = 5517KB (as at the end of the track so has no tail).

Again, "Remove Silence" unto itself does nothing to the underlying audio (no bounce occurs). What does change file size is the "Create one file per clip" in the save as dialog (render does occur for this).
 
Just to be clear on the original point for folks this is confusing - a "clip" is a window into an underlying wav file (what you are telling the DAW to use). Manipulating this window does not change the underlying file unless it is bounced/rendered, which is a "destructive edit" (the clip will no longer have access to the original file). "Destructive edit" in this case does not actually delete the original file (it is still in the audio folder), but does assign the clip (window) to a different file.