That explains a few things

Here's another interesting tidbit from The Science of Happiness about why we stray:

"If love is not involved, the lure of the unfamiliar is often much more important in sex than the quality of the interaction... There are indications that in humans, the inborn amount of dopamine receptors has an influence on a person's sexual promiscuity. These, at least, were the findings of Dean Hamer, as researcher at the National Cancer Institute in Washington, who determined that there is a direct relation between a certain variant of the gene for a D4 receptor -- a receptor for dopamine -- and the urge for erotic adventures. Thirty percent of all men are supposed to have such a 'gene for promiscuity,' and over time they have 20 percent more sex partners that the average man... His finding does indicate that the extent of this urge for novelty must be at least partly inborn."