Disable Time Ruler Zooming

I wonder whether there is a way to disable zooming in the time ruler?
I like the left/right scrolling in the time ruler, but as I am not so skilled with mouse moving I often get up or down accidentally and that zooms the track view which is annoying. I would prefer to have only the simple left/right scrolling active (by the way I have the same problem with Melodyne).