Green’s Dictionary of Slang

(US) to regard with scorn, dislike.

1963

in T.I. Rubin Sweet Daddy 119: They’d fish me, I’d fish them.