I use a q tip all the time. That way I do not get any build up. Yes you have to be careful.
When I shower I try to get a little warm water in the ear and then clean the ear with a q-tip as part of my drying off routine.
I have noticed that when I go swimming it takes a while for the water to get out of my right ear. Not sure what that is about.
Anyway. Glad you can hear again.