Green’s Dictionary of Slang

hobby horse man n.

[hobby horse n. (2)]

[early 17C] a womanizer, an adulterer.