Green’s Dictionary of Slang

two by four n.1

(US) an insignificant person.

[US]S. Ford Torchy 20: I sized up Piddie for a two by four right at the get away; but I’ve been keepin’ him jollied along just for the fun of it.