Green’s Dictionary of Slang

color n.

In phrases

get one’s color up (v.)

(US black) to lose one’s temper.

1961

C. Himes Pinktoes (1989) 166: He had a profound respect for colored ladies when they had gotten their color up.