I can relate to this.
I always start songs at a slower tempo than they eventually wind up, but by the time I've recorded the final tracks, I've settled on a tempo.
I chalk this up to the fact that when I'm writing the song, I'm still working out the changes and therefore more hesitant and less confident both in the song and my playing of it. I may have finished it as a writer before I've learned it as a player or arranger. I also kind of write and arrange as I go.
My shifts have gone as high as 10BPM or more.
I account for it by just cranking up the tempo by 5 BPM over whatever I choose at the start, and it never feels rushed to have done that by the second day of working on the tune, so, good news.
The suggestions others have had to do it at the beginning rather than the end of your process may be good ones!