Comp mode IS supposed to do that automatically but I have not done any extensive recording in that mode. It may however hinge on the setting CJay mentioned as well.
In X2 I actually had some problems with previous takes being auto muted even with that setting engaged though. This seemed to be a quirk with using the "R" key to start recording instead of clicking the record button in the Transport Module. Basically sometimes the previous take would mute and other times it would not.
The R key does seem to be working okay here with X3 but as I said I have not yet recorded extensively with Comping mode because I did not understand it initially and opted for the more familiar Sound On Sound option and manually muting the previous takes as I worked.
My point (good god Beepster... do you EVER get to the point?!) is do some experiments sitting at the computer. Try beginning record transport in Comp mode using 1) the Record transport button, 2) the "R" key and then 3) what I'm assuming is some kind of control surface that you have been dragging into your live room.
Try these options multiple times as the "R" key issue I mentioned in X2 was intermittent so it may be in this case as well. Perhaps there is a way to map your control surface to whatever works the most reliably.
Again... comping mode IS indeed supposed to mute the previous take by default however take a look at the Preference option CJay mentioned because that may alter the default behavior.
Cheers.