The reality of software is that it can go on forever - "just
one more feature, just
one more week of testing" - so companies choose targets and create milestones for what they'd like to see in place in terms of features and testing, and by what date. If they make the milestones and the betas are stable, the product gets released. If not, it gets pushed back.
Also, retailers put pressure on manufacturers to have products in the pipeline by certain dates that correlate to sales activity. Christmas is an obvious one, but there are others. Pro users put pressure on wanting certain features NOW that a competitive program might have. To them, once a year is nowhere near often enough.
As to upgrading, that decision is entirely up to the customer. If an upgrade contains compelling features that would be useful and justify the cost, then you upgrade. If not, then you don't. And there's not a one-to-one correlation between features and upgrade cost. For example, if you didn't use pitch correction, never comped, had no VST3 plug-ins, but did need a solid virtual drum program, then X3 was worth it because the upgrade was the same price as Addictive Drums. OTOH I think some people would have upgraded just to get Melodyne, given the entire upgrade was only $50 more than Melodyne Essential and you got the cool comping, AD, VST3, etc.
As to X4's drop date, it's after X3, but before X5. I'm quite certain of that