The thing is, the person you M'd is also the person who was an angry SOB with no patience and held so much anger towards you. It is nice that he is being transparent, touching; what is he doing to work on his issues?
When he was so angry at you and turned to OW, why was he angry and why did he not talk to you? Conflict avoiding is common, and it takes paractice to overcome it.
Why did he blame you for his unhappiness? FWW did that with me too. Is he learning to own and process his feelings, and to talkwith you about conflicts?
I'm hoping with time it can fade and we will be better off than we were.
The pain fades with time. Change for the better requires understanding, insight, and practice to re-enforce new habits and responses in place of old ones.
You might spend some time thinking about why you stayed with a spouse you did not like for 5 years. When I did this the answers were an important part of my healing and become the better person I wanted to be.