Okay, so here's the updated weirdness that is happening, along with excruciating detail on the project. This project use a 16 track template, all audio tracks. I added a MIDI track for the drums, so we have a total of 17 tracks. I record scratch bass on one track & 2 rhythm guitar tracks, and the auto-muting function works just fine during recording. All of these takes were done as straight beginning-to-end takes, then I would go back & punch in to fix any mistakes or make changes, which is where the auto-mute was evident, using comp recording.
Then I add a melody guitar, using the looping feature so that I can try some variations on what I'm doing with the melody. So far, so good; auto muting during comp recording works fine. Once I'm happy with the arrangement I go in and record the drums to a click, and again, so far so good. Then I record permanent bass & rhythm guitar tracks. It's all still good. Then I go in to record the permanent melody guitar and the issue pops up: I can always hear the most recent take, which should be auto-muted when I hit the record button. After monkeying around with it a bit, I post the original question here at the forum. Several folks offer up suggestions, I double check everything they suggest, but no joy.
So yesterday, I really dig in and experiment, and this is where it gets even weirder. The melody guitar is on track 8, and I have four other tracks with guitars (in addition to the 2 rhythm guitars) that do only 12 measures of these little harmonized flourishes. I recorded some nonsense parts on each of those four tracks as a test, and the auto-mute worked just fine. I do the same experiment on the bass track & the two rhythm tracks; auto mute works fine! I go to some empty tracks & record some random stuff with both the loop function and just recording, stopping & recording again, and the auto mute works fine. Only on track 8, it still fails to mute the most recent take automatically. Ultimately, I test all 16 audio tracks, and auto mute during recording works on all but track 8. I have opened up every menu I can find, and all of the tracks appear to have identical settings. So... I quit, lol.
When this problem popped up I only had one little 4 measure section on the melody guitar that I kept flubbing, so I just let it go for a day, went back the next day, manually muted the take lanes, played the part right, and got on with the rest of the project.
I had hoped to figure out what caused it, so I could avoid the issue in the future, but I'm going to attribute it to the ghost in the machine, call it a one-time bug, and forget about it. Thank you to all who offered suggestions.
Oh, here's the other weirdness: at one point near the end of the song, which has a very funk feel to it, you can hear what sounds like someone doing a little "uh" type of grunt. It's rhythmic, and at the end of a cool little riff. Problem is, no one but me was present during any of these sessions, and I didn't do it. If I solo each track out, trying to find it, the "uh" is not there. But if I solo the two rhythm guitar tracks you can hear it. And it does not sound at all like a guitar sound (finger scrape, pick hitting a muted string, etc) but definitely sounds like a soulful dude going "uh." I'm leaving it in, 'cuz it fits & I'm too weirded out by it to try to fix it, whatever "it" is, lol.
Thanks again, everyone. I'm off to lay some guitars down for a couple other tunes. Hopefully the ghost in the machine will not play any tricks on me with those.