Should I remove him from my childs life?

My husband left me for a much older woman in October and basically revolves seeing his son around times that are convenient for other woman. he recently told me his gf thinks we talk too much so dont contact him even about our son, doesn't matter if he's sick. Just not at all. He was seeing him twice a week plus e/o weekend but it was too hard so now e/o weekend and even then hw shortens the time. He nevereven calls our son.. Ishould I just remove him now before he hurts my son or should I keep giving him chances?