Yes the actual sound can help when playing along and using a pre-made groove to jam with can help some.
TBH I never 'hear' the click after the count in, I probably sense it in some way but I'm never consciously 'aware' of it. The only time I hear the click apart from that is when I hear the first beat after I've already realised I've fluffed the take anyway.
Western music dictates measures, bars and beats as strict time based divisions adhereing to a tempo even without a click. 'Con Rubato' hardly ever appears as a required interpretation/playing style on any score for popular music.
To me playing 'loose' and 'grooving' are two entirely different subjects.
Much of the subject on this particular debate as well will be for those wanting to add-in some feel post-event and there are some good pointers already here on how to do that, so those producers can add a 'feel' to their stuff that might not already be there, when they are using methods to arrive at a groove and being able to actually play it out isn't an option for them.
Adhering to a click shouldn't and doesn't limit any ability to groove the only thing it does is keep the tempo strict. The only reason not to use one is if you are using tempo changes to bring particular emphasis to certain elements of a song structure where being glued to a specific tempo might prevent that. Tempo to my mind is a seperate element from 'the groove' which occurs within the structure of a measurable time-base whatever the tempo.
It's strange for me even to try and discuss it because normally I'd just do it. I rarely if ever think about it or attempt to analyze it.