Green’s Dictionary of Slang

cauliflower n.4

[? Fr. choux, cabbage, used as a term of affection]

(US) a person, a fellow.

[US]J.C. Neal Charcoal Sketches (1865) 37: Let go, watchy! — let go, my cauliflower!