Don't know what to do (sad)

I don't know if it is and I'm annoying him as I always seem to.....We have been married for nearly 5yrs. We are both 23 an have 2 children. My situation has been progressively getting worse. He practically never goes to be with me he either falls asleep on the couch or stays up even if I ask him to come to bed. We have sex like once a month usually initiated by me and I know for a fact there is nothing wrong with him sexually. He gets annoyed with me pretty half the time I speak to him. He's never abusive though and doesn't drink because of our beliefs. We both study so he does a 1/4 of the house work. He forgets our anniversary and my b'day, once our son knocked his tooth pretty badly we drove to the hospital he didn't even get out of the car, during my second labor he read a magazine and said later that every woman goes through this. He doesn't like me being upset with him so during the day if I'm not ANNOYING him he will act happy and if I show that I'm upset he will suck up and if not immediately forgiven will get pissed off at me and not talk to me. On the most recent mother's day he did not even mention it let alone do something special. I feel isolated and alone and don't want to tell anyone because I don't know if I am to needy. Oh and he doesn't like going anywhere with us (me and kids) it's boring for him so as a result we usually don't do anything as a family. I just don't know what to do...... :'(