Of course there are many ways to approach it, including turning on and off power on various pieces, or using a mixer, or having a good audio interface with some software routing that allows some software switches to be available on your PC screen.
After years of knowing about it and thinking "some day" I ended up finally getting a Big Knob. It's excellent. It handles switching around from four different input sources and routing out to three different monitor sets, plus routing for 2 headphones and even a mic and talk back feature if you someone recording with mics in a booth.
Anyway, the question was posed if there was a solution, and I'm just giving a thumbs up to the Big Knob which does that really well.