Am I right to be mad?

I think you overreacting a little bit. If he couldn't talk to you then maybe the doctor was in the room while he and his ex was in the room and they were talking about important information. Or he and his ex could be discussing important information about their child. If he cares about you like you think he does then you have nothing to worry about. He will probably always be in contact with his ex because of the fact that they have a child together.