No, it isn't something you should feel stupid about or anything. The flutter you felt may not have been the baby moving, but just gas movement or whatever.....
If it was the baby moving, then you are going through what most moms go through: during the day the baby gets rocked to sleep by your movement so she doesn't move as often. Also if you have a busy lifestyle, you aren't going to notice half as many movements as you would if you were thinking about it all the time. The baby could be moving, but isn't that big either, so sometimes he may not be able to give you a poke that you can feel yet.
At night you are sleeping and sedentary, so the baby isn't being rocked to sleep, so he moves more.....but then you're asleep and his kicks aren't quite enough to wake you. Most of the time they won't.
If you are really worried, drink a glass of milk and then lie down for about 30 minutes. You should feel your baby move quite a bit.
You can always contact your doctor if you are worried and need peace of mind. You don't even need an US, just a heart rate test will assure you that everything is ok