I am not sure that can be removed, as I believe it is a core feature. I cannot recall which update it was released in offhand, but the intent was for a quick input option on-the-fly (it is not hardware, so will not show in your devices). I am thinking you unplugging your controller was coincidence with you noticing this.
As far as your project goes, when you reopen them with the controller plugged in, they map correctly to their saved state, correct?