Green’s Dictionary of Slang

(US) nonsense; also as excl.

1940		F.S. Fitzgerald ‘Pat Hobby, Putative Father’ in Pat Hobby Stories (1967) 86: Aw — bananas.
1960, 1975		Wentworth & Flexner DAS.
1989		H. Rawson Dict. of Invective (1991) 34: In the plural, the word means ‘nonsense,’ often as an interjection, bananas!