Are you running the Saffire as your main computer audio output? If so, I have a hunch that it is more an issue that the Saffire is trying to accommodate varying bit-rates/bit depths for playback. It will accommodate requested changes (not instantaneously), but cannot do all at once.
On my machine, I use my Saffire for DAW work only for the most part and let the onboard Realtek handle "Windows stuff." This does require a second sound system, but alleviates multiple sources trying to use the Saffire in (potentially) different playback modes. Anytime a "browser" gets thrown into the mix, ASIO doesn't seem to be a happy camper.