Your kids are 18 and 16. I'm not sure how young the third child is. But,if they are close in age to the older kids, then all the more reason to stop allowing him full access to your home.
If he can't have them where he us staying, then HE needs to make plans to actually pick the kids up,and take them somewhere to spend time with them. He left. It is his job to arrange time with the kids, not yours. There are plenty of free,fun things he can plan for them.
He isn't full of shame. He's having a great time. He's on his own,living very close to his younger girlfriend. He gets to see her whenever be wants, and then you let him come home,make him dinner, dance the pick me tango, and he's enjoying every bit of it. He has 2 women acting as if he's a prize.
You need to set some boundaries. He left. He doesn't get to come,and go,as if he still lives there. He left. He needs to sit with that decision. He needs to live as the single man he wants to be. Close up the bakery. Tell him you want the keys to the house.
Of course, legally, it's still his house,and if he refuses,there's nothing you can do. But he wanted to leave. He can't have it both ways. And you need to stop making this so easy for him.
Also, that he still continues to speak to her,tells you all ou need to know. He's gone. If he returns, it's not because he suddenly remembers how great you are. It will be because he tried out the new shiny toy,and she wasn't as fun as she was when it was an affair. Don't be his Plan B.