you didn't know and you can't change the past, so try to stop worrying (easier said that done, i know). stop taking the xanax and talk to your doctor about safer alternatives.
xanax is a class d drug and during pregnancy you really want to only take class a and b drugs (sometimes c if it's needed).