Unless they are teenagers or there is neglect or abuse happening they don't get a choice. What ever the schedule is, this is what occurs.
I would caution listening to their wants as they maybe trying to manipulate the situation or playing you off against your ex. Or they maybe concerned about leaving you alone or possibly missing out on the fun stuff you do while they are away.
I don't discuss my plans with my children while they are at their fathers for this very reason. Also so they don't inadvertently share that info with their father it is none of his business.
Ensure they know you will miss them and that is ok, but their other parent misses them while they are with you as well.