If the drummer feels the need to hear the click track they will inevitably drag behind it. The drummer that can play to a click track does it by getting the click to disappear and to stay away because it stays obscured OR the click track is done with eight notes so that the drummer can hear the intermediate notes.
Ah, a more objective approach, and I think very close to the heart of the matter. I do think there's maybe also a simple physiological reason that might be involved - the timing mechanisms in our head that allow us to speak without stuttering (singing is easy, speech is clipped) are not equally well developed in all of us. Put on headphones with a slight delay, and EVERYBODY will start to stutter, and get hopelessly lost rhythmically.
So I think that's somehow involved , but by no means implies that our 'normal' vocal stuttering is directly the 'culprit' in this case, just trying to make a connection that I think could very well turn out to be a valid one.
Some drummers might just lack the talent to switch off their 'internal' earphones, pick up the wrong beat, and go out of synch.
Anyone thinks this is exaggerated, fire up sonar, put on the headphone, plug in your mike and put in a nice delay and start a speech, and I wish you good luck. You never knew you had it in you.
D-d-d-d-d-dats all, folks, without trying.