Green’s Dictionary of Slang

dog n.4

[? the animal’s inability to speak]

[early 19C+] (US) nothing; usu. in phr. never say dog, to stay silent.