I think sometimes we unconsciously replace feelings of shame with a different emotion that is easier to deal with, particularly one that's outward directed. Anger is one alternative that is common.
Recovering from infidelity does involve learning to deal with very uncomfortable emotional states for a pretty long time. Sometimes my feelings track BS's apparent emotional state and sometimes they come as a result of the personal work I'm slogging through.
Something I've found that helps is any amount of meditation practice. Even if it's just a few minutes its helping me develop the ability to give myself a mental break.
There are a lot of resources on the internets that are free and can help you start a meditation practice. One book that I'm learning a lot from about why meditation works from a physiological perspective is Buddha's Brain by Rick Hanson.
This is a bumpy ride so it's good idea to start assembling a tool kit of wholesome self-soothing techniques.
Strength to you from a fellow EvolvingSoul.