ASIO is far preferred, trust me. I have a couple M-Audio MIDI controllers, and if I recall correctly, the drivers they came with were also ASIO. If I had either plugged into my computer via USB, I immediately got conflicts (it seemed I could not just choose it for MIDI, it wanted the ASIO spot too

). The only way I got around this was to route one via MIDI cables into my audio interface (i.e. unplug it from USB completely). I am not sure if you are experiencing the same issue. Do you have that option to connect it via MIDI cables to your interface? My system is still set up this way, and I have MIDI assigned to my audio interface's MIDI, not the controller itself.
Another option is to disconnect/reconnect the USB and see if this will reset anything. For me, this did not help.
I would also recommend sending M-Audio an email, and ask them specifically "How do I use my Axiom as a MIDI controller only without the ASIO drivers being loaded?"