Depending on the level of fragmentation, you are going to move everything on your hard drive to a different place, including parts of the OS, and in the process, data from target locations can be "erased" and held in buffer, while new material fills its spot, before being itself written to a new area.
You are accessing many, if not all areas of the drive, and the heads are in constant motion throughout. From the position that a parked head has zero chance of a head crash, defrag ops represent the opposite end of the probability spectrum.
Defrag routines are mature, and largely robust, I don't recall any recent problems with mine, but I have lost entire drives, and many files, over the course of decades, and anytime I move quantities of data, including major copy operations, I back up first.
Its your data. Nobody cares about it as much as you (except the NSA, and they already have multiple copies that you aren't allowed to access). It's akin to emptying your house into your backyard, so you can reorganize it piece by piece throughout. If you are comfortable parking your wife's diamonds and your collection of gold coins out by the swingset with the rest of the Tupperware bins, till you have the room they are destined for cleaned and organized, have at it. I prefer to avoid risk, build up good karma, and save edgy maneuver till I really need it.