I am confident [my] ex had no problem with self love... he was his favorite person in [the] universe.
Self-love, and being in love with yourself, are two very different things in my mind. What it sounds like you are describing is a narcissist, which is the sort of the polar opposite of self-love.
When we discuss self-love, what we are talking about is self-respect, integrity, healthy boundaries, humility, confidence, morals...
"Self love" means that you want to be the very best person you can be, despite your human failings, and that you focus on being someone that you can respect, and that others can respect. You can sleep at night, knowing you've done all you can, every day, to be a decent person to yourself, others and the world around you. Self-love lacks selfishness and entitlement, because the need to feel validated is fulfilled from within ourselves, not from others.
Self-love is NOT thinking you are hotter, that you are smarter, that you are better at everything and better than everyone else. It does not mean you want to fuck yourself when you look in the mirror. It is not feeling entitled or more-special or more-deserving or better than others. Feeling any of these things is the exact opposite of self-love. A person with self-love is incapable of feeling "more than" others because the need to feel that way is non-existent, they literally have no reason to feel that way. In the same way that a person who just stuffed themselves with food won't feel hungry, a person who stuffs themselves with self-respect and self-value will never need to try and get those things from others by feeling entitled and special.
A person who:
* Respects themselves and others
* Has healthy boundaries and enforces themselves
* Has integrity and self-respect
* Understands their own needs
* Lacks selfishness and entitlement
is, in my opinion, simply incapable of having an affair. Any one, let alone all, of the qualities listed above, would prevent that person from disrespecting themselves or another person in such a way.
Just as an aside, narcissistic behavior often stems from self-hate, or maybe more accurately, self-desperation and neediness. Most narcs are created from one of two situations. Either they are told they are never good enough no matter what they do, or they are told everything they do is perfect. The ones who are told they aren't good enough are constantly looking for the praise they never received, and so they see themselves as better than everyone and everything so that others will tell them how wonderful they are. They are incapable of feeling good about themselves, and so they create a persona that they feel everyone should love and praise.
Those who were told they are amazing "no matter what" are handed a feeling of entitlement that they didn't work for or deserve. Strangely enough, they too also are incapable of "truly" feeling good about themselves, because they never actually learned how to achieve something by working for it, so any praise they get feels hollow to them, yet at the same time, they require it since they lack the tools to earn the feelings of worth for themselves. In other words, even those "who are in love with themselves" are often the ones that hate themselves the most.
(Sorry for the T/J)