Now that the D is started and STBX is looking like he will turn nasty, I am worried that she will tell her H everything, he will relay that to his brother (my STBX), and my STBX will be able to use this to hurt me in the D process.
Am I just spinning my wheels here, or is there anything in particular I can do to do some damage control?
It is what it is.
Even if your STBX found out everything you said to your SIL, I doubt it would carry any weight in terms of the divorce proceedings. Judges don't care about that sort of drama. It's normally very dry, just about who makes how much for the purposes of alimony and child support, and also about full disclosure of assets so that things are divided fairly.
The process is so stressful. Hang in there.
Did you tell her in an attempt to destroy your STBX? Or were you instead merely confiding in her as is the established nature of your relationship?
Yes, you should tell your atty everything. However, it is also true that judges really don't give a shit about anything disgusting & salacious. They really do only seem to care about division of assets & custody, and the less you involve them in that the better.
If it is nothing like this, the worst that will likely happen is that you've created unnecessary family drama.