Green’s Dictionary of Slang

the penis; cit. 1691 is double entendre, the play is on both one’s wife’s lovers and their penises.

[1691		‘A New Medley’ in Ebsworth Merry Drollery Compleat (1875) 347: For there where other Gard’ners have been sowing / Their seed].
1890–1904		Farmer & Henley Sl. and Its Analogues.
1980		Maledicta IV:2 (Winter) 193: The third and most numerous group of fruit and vegetable synonyms comes from analogy with the corresponding feminine terms, so that the gardener tends the garden or cabbage patch.
2010		(con. 17C) W. Ings ‘Trolling the Beat to Working the Soob’ in Int’l Jrnl Lexicog. 23:1 62: By the nineteenth century garden had also come to refer to genitals, thus [...] gardener (the penis).