(Updated) Comp recording acting like sound-on-sound?

right click on record button to open preferences & change

You could try going into the actual preferences [under edit or file ] and change it there. Even if it already says comping change it to something else and then back again.

Have you made sure Exclusive solo isn't on ?
 
Have you expanded the lanes to see if some have become unmuted ?

could you possibly have a hidden track somewhere in your project?

Okay, so here's the updated weirdness that is happening, along with excruciating detail on the project. This project use a 16 track template, all audio tracks. I added a MIDI track for the drums, so we have a total of 17 tracks. I record scratch bass on one track & 2 rhythm guitar tracks, and the auto-muting function works just fine during recording. All of these takes were done as straight beginning-to-end takes, then I would go back & punch in to fix any mistakes or make changes, which is where the auto-mute was evident, using comp recording.
 
Then I add a melody guitar, using the looping feature so that I can try some variations on what I'm doing with the melody. So far, so good; auto muting during comp recording works fine. Once I'm happy with the arrangement I go in and record the drums to a click, and again, so far so good. Then I record permanent bass & rhythm guitar tracks. It's all still good. Then I go in to record the permanent melody guitar and the issue pops up: I can always hear the most recent take, which should be auto-muted when I hit the record button. After monkeying around with it a bit, I post the original question here at the forum. Several folks offer up suggestions, I double check everything they suggest, but no joy.
 
So yesterday, I really dig in and experiment, and this is where it gets even weirder. The melody guitar is on track 8, and I have four other tracks with guitars (in addition to the 2 rhythm guitars) that do only 12 measures of these little harmonized flourishes. I recorded some nonsense parts on each of those four tracks as a test, and the auto-mute worked just fine. I do the same experiment on the bass track & the two rhythm tracks; auto mute works fine! I go to some empty tracks & record some random stuff with both the loop function and just recording, stopping & recording again, and the auto mute works fine. Only on track 8, it still fails to mute the most recent take automatically. Ultimately, I test all 16 audio tracks, and auto mute during recording works on all but track 8. I have opened up every menu I can find, and all of the tracks appear to have identical settings. So... I quit, lol.
 
When this problem popped up I only had one little 4 measure section on the melody guitar that I kept flubbing, so I just let it go for a day, went back the next day, manually muted the take lanes, played the part right, and got on with the rest of the project.
 
I had hoped to figure out what caused it, so I could avoid the issue in the future, but I'm going to attribute it to the ghost in the machine, call it a one-time bug, and forget about it. Thank you to all who offered suggestions.
 
Oh, here's the other weirdness: at one point near the end of the song, which has a very funk feel to it, you can hear what sounds like someone doing a little "uh" type of grunt. It's rhythmic, and at the end of a cool little riff. Problem is, no one but me was present during any of these sessions, and I didn't do it. If I solo each track out, trying to find it, the "uh" is not there. But if I solo the two rhythm guitar tracks you can hear it. And it does not sound at all like a guitar sound (finger scrape, pick hitting a muted string, etc) but definitely sounds like a soulful dude going "uh." I'm leaving it in, 'cuz it fits & I'm too weirded out by it to try to fix it, whatever "it" is, lol.
 
Thanks again, everyone. I'm off to lay some guitars down for a couple other tunes. Hopefully the ghost in the machine will not play any tricks on me with those.

You probably have...................but did you try starting it in safe mode?

Hey Z1812, that totally worked! I opened the project with Sonar safe mode, tried recording a little loop on track 8, waveform appeared, all takes were muted as new ones were recorded. Then I dumped the test recordings, shut the project without saving changes, opened it normally & tried the experiment again. Everything works as it should!
 
I can't believe I didn't know about the Splat safe mode. Thank you!

As far as the "uh" goes, I've had similar exeriences. Almost surely the result of 2 or more tracks merging at one point creating an odd sound.

Ears are good at playing tricks on you.

One such instance resulted in a "happy accident" which I tried to recreate.

To narrow it down you need to solo a suspect track while soloing the rest one by one.

There's a known phenomena in orchestrated music called Combination Tone where 2 or more instruments playing different notes collide forming sounds not produced by either one of them.