It's not your imagination, sharke. The illusion's just not that good to convey small movements accurately. In the real world, we can do it surprisingly well, but that's different. That's sound coming from multiple sources and reflections and being sensed binaurally, not sound coming from just two discrete point sources.
Furthermore, we are rarely listening to speakers in the precise center where the illusion is best recreated. More likely, we're listening at a distance, off to one side, or we're using headphones/earbuds that do away with an connection to physical reality altogether.
Despite all that, small changes are still useful in mixing. For example, panning harmony vocals 5% off center gives a sense of width that you're missing when they're all panned center. We can't tell they're 5% versus 8%, but we can sense that they're wider than a single point source. Assuming the listener isn't that typical teenager with the one 95-cent earbud. But then, I mix for the 1% - if one person hears it correctly, that's the one person I mixed it for.