This happens to me sometimes. If I unfreeze and refreeze, it's usually OK.
For me, the clip is always late and not early.
Just for the record, I often freeze and instrument, drag the audio to a new track, and then delete the virtual instrument to save resources. I would not do this with a critical, foundational instrument, but if I have a percussion sound, or short instrument line, etc., I don't mind deleting the instrument since it would always be easy to redo it if I needed to.