Green’s Dictionary of Slang

(US) to leave quietly.

1842		Flash (NY) 3 July n.p.: New York Wants to Know [...] Whether Ad—s really did caterpillar when he read the Flash.
1849		W.C. Hall ‘Mike Hooter’s Fight with the “Bar”’ in Spirit of the Times 10 Nov. (N.Y.) 452: ‘Let’s licker first,’ said Mike, ‘an’ if you don’t all caterpillar, you can shoot me’.
1902		L. Mead Word-Coinage 188: I caterpillared [HDAS].