First off, you should not feel badly. All of us with children have had the exact same experiences, and we've all been there. Plus, in every church of any jurisdiction or denomination, there are always people who feel they have to educate young parents on what to do.
If I were you, I would ask your priest what he thinks about the situation. Different priests have different thresholds of irritation regarding crying children, and it'd be nice to know where he is with that. I know I don't mind hearing a crying child in the congregation---it means the children are coming to church, which is where they should be, after all---but I also respect the parent's decision if they need to leave temporarily.
Plus, you may want to ask your priest what should you do if it is in one of the moments of the Liturgy where you're not 'supposed' to be entering or exiting the church---what would he have you do?
Finally, you also have to use your judgment. If the child is being a little cranky and only needs a little distraction before s/he settles back down, and if it's otherwise OK with your priest, then stay there. If it's obvious the child is worked up about something, then maybe retiring to the narthex to investigate is in order---but you should also come back in or otherwise the child will learn that s/he can get Dad and he to leave whenever he acts badly.
So, just call your priest and ask him. Then, when a busybody butts in to give you 'helpful advice', you can indicate that you have discussed this matter with Father already, and done exactly what he wanted!