So you have two choices.

You can leave him, and you can just accept that you will live with it for the rest of your life.

What I don't get is why it matters to you? Why you are so put out by this?

He isn't. Why are you?