It depends entirely on the task at hand.
For mundane chores or routine assembly work, I will generally have some music playing.
If I need to be mentally focused, then yes, music can be distracting.
One caveat to that is if I am drawing or painting. Then, carefully selected music can be very beneficial for dialing in the proper mind set.