Your husbands sperm goes to your stomach, your baby is in your uterus. There is no connection between the two. Your baby gets its nourishment through the umbrillical cord, and the placenta filters out the toxins, your baby will not be hurt.
Anyway, sperm is almost pure protein, so it might even be good for the baby