Joey, if you're asking:
1. if it's a waste of time
2. to trim the part of the track where there are no vocals
3. to reduce any possible unwanted noise that might become part of the sound floor of your track,
that really depends on what your background noise is/might be. I've had tracks where I trimmed as you described after hearing noise I felt was too imposing when I listened back. Other times where I just left it as is.
But I'd say I do trim it most of the time. As an experiment, I gave 2 versions of the same vocal track — one with the non-vocal space trimmed out and one with the quiet room left in — to a very experienced mixing engineer to compare, and he said he didn't think it would make any difference in the mix.
But he did say this as well: if the song would have been a very quiet, single guitar with vocal, that trimming might actually be worse depending on what kind of ambiance was ultimately desired, as the sound floor if trimmed out, might sound "on and off" and odd.