Green’s Dictionary of Slang

an insult hurled at a second-rate, incompetent coachman.

1859		G.A. Sala Twice Round the Clock 116: He can drive neither to the right nor to the left, nor backwards, nor forwards [...] A sarcastic saloon omnibus driver behind jeeringly bids him ‘keep moving,’ accompanying his behest by the aggressive taunt of ‘gardner.’.
1864, 1867, 1870		Hotten Sl. Dict.
1873		Sl. Dict. 173: Gardener an awkward coachman; an insinuation that he is both coachman and gardener, and understands the latter branch of service better than the first; ‘get on, gardener,’ is a most insulting expression from a cabby to a real coachman. Men who in small families do the coach, garden, and general work, are sometimes called ‘teakettle grooms,’ or ‘teakettle coachmen.’.
1894		Morn. Call (S.F.) 23 May 11/1: Slang rom London Streets [...] An inexperienced Jehu may be either a ‘gardener’ or a ‘whack’.