There's a lot of odd stuff being interjected around this thread. Playing in time with something else is a skill like any other, and for a lot of untrained drummers it's a skill they never have to learn because their band follows them. So, you put a click on (or a guitar part, or a sequence of percussion or anything else) and they can't play in time with it - they rush, drag and correct which results in an uneven time feel, or just just drift off entirely. Like any other instrument out there, the majority of drummers suck and haven't done the work. The same is true of guitarists, the same is true of singers.
Nearly any drummer who's even remotely what I'd describe as "good" has put in serious hours with a metronome learning to play in time. That's just part of learning to play, because it's very hard to actually develop any kind of consistent time feel on the drums without an external reference. The drums are a very physical instrument, and there are a lot more things in terms of co-ordination and balance that can affect your timing in a way that you won't experience as a guitarist or keyboard player. If you have to go from a cymbal on your left to a cymbal on your right in a certain amount of time it's a very different movement to moving from the snare to the high tom.
As an analogy, imagine driving to work. It's easy for you to do it in time because you do it every day and you have a clock that keeps you in time while you're doing it. Now imagine having a fixed amount of time to get from any point in your town or city to any other point, not being allowed to arrive early or late and having to pick your speed to make sure you arrive precisely on time. That's what drummers do every time they move around a drum kit, and if they practice that without external reference then they'll probably get it wrong sometimes - usually on fills, or when coming back from a fill to the groove.
The other aspect is that many drummers actually can't play in time with themselves. Drumming typically consists of playing ostinatos somewhere on the kit (usually right hand, for a right handed drummer) and then playing syncopated patterns against those ostinatos (bass drum and snare, in a pop/rock context). If you're not actually able to land your notes in precise unison with the ostinato then you have another problem, because most drummers tend to carry the time in their ostinato, but the rest of the band carry the time from their bass drum / snare pulse. So if a drummer doesn't have very good technique or precise co-ordination then you'll again have trouble landing notes precisely on the click when playing grooves.
It's all actually really quite hard. The physical and co-ordinational bar for playing the drums consistently is much higher than on the guitar or keyboard, while the requirement to play consistently is considerably higher-priority. Does it matter if your guitarist has slightly loose time? Not as much as it matters if your drummer does.
The reality of the matter here is that when guitarists suck it tends to be less obviously their time that is the problem. Crap guitarists play out of key, flub chord changes, have poor tone, get their volume wrong... all manner of stuff. If you find one who doesn't do all of that stuff but who has slightly loose timing you tend to give them a bit of a break. Drummers, on the other hand, have to get it perfect or be named-and-shamed, regardless of what else they do right.
I studied at music school with an engineer who used to run us through a test: He'd give us a click at 120bpm for sixteen bars, let us get comfortable, then drop it out of the cans for four bars. If we came out the end of that still in time with the click he'd bump that up to eight bars. Then sixteen. Then thirty-two. The all-time record was held by a Korean guy who managed a massive SIXTY-FOUR bars of staying in perfect time with a click he couldn't hear. That's more than two minutes. I think my record was eight bars... But this stuff is something you can learn if you want to, and if you want to record regularly as a drummer you'd really be advised to put the time in. Go grab Gary Chaffee's book "Time functioning patterns" and George L. Stone's "Stick Control" and practice them front to back with a metronome, at every tempo from 30bpm up to your maximum. That ought to do it.