I believe you're right. AFAIK there's no other way than to do it gap by gap, manually.
But I think that dragging the clips is faster than deleting the spaces. Starting from the end would lead to
having chunks of several clips you could move together. If it's all on one track it shouldn't take long, five-ten minutes.
If the size of the clips allows it would be useful to set Snap To Grid accordingly.
Unless there is a CAL-function for it. I've never used CALs so I know nothing about them.