Try using a lower case "x" instead of the capital; I tried this and it worked as expected:
number("0xAB12")
I also tested this and received a "0" as the response:
number("0xAZ")
I agree that this should return a value of "NA" which is value of nullnumber or at the very least throw a number conversion exception. Please write this up as a ticket.
Sam
PS: I tested this on my 15.0 SP02 P6 instance of MII