I recommend stating that your attorney has advised you to avoid personal conversation, and to remind him that he's there to interact with his son.
The attorney line will allow you to more easily stonewall him when he tries to converse. Just keep repeating it as necessary, and do not get drawn into any conversations. "Remember, you're here for Billy. I'm not supposed to have personal conversations not directly about Billy."
Repeat that line to yourself before every visit, too. Do not let yourself get drawn into any conversations, even those he prompts by saying, "Slut and I are getting engaged" or "Slut meets my needs so well, we're so happy." If he starts actually harassing you with direct insults, you should consult your attorney, but for generally triggering remarks that aren't directly abusive, just don't respond, and he'll engage with you less and less.
Small children should be told that mommy and daddy are going to live in separate houses from now on, but they both still love him. If he asks why, say, "Sometimes parents have to live separately, but you'll always live with me and we both will always love you." Don't promise him that his dad will always come to see him, because his dad might well quit visiting eventually.
When your child is old enough to understand & if he asks then, you can tell him that his father's adultery broke up the marriage, but don't rub it in or editorialize. Just state the most basic fact needed for him to have clarity.
[This message edited by morningglory at 2:38 PM, Thursday, June 23rd]