Green’s Dictionary of Slang

1. (US) used as a euph. for a brothel [presumably based on a specific brick-built building in a single city, but also used generically].

1833		Ely’s Hawk & Buzzard (NY) 14 Sept. n.p.: [This] Col. had better pay up his old score Boarding in S — street [in Brooklyn] before he begins a new one in a four-story brick house oin Fulton-st.
1842		Whip & Satirist (NY) 8 Jan. n.p.: A very handsome three-story brick house [...] looks very suspicious on account of the number of pretty females that are always to be seen seated at the windows.
1842		Whip & Satirist (NY) 11 June n.p.: New Haven Wants to Know What Mr T.T. the ex-merchant [...] is doing so often at the brick house at the corner of Temple and Chapel sts. Look out Tommy or you will be caught.
1850		Life in Boston (MA) 7 Sept. n.p.: Moll McQuade, alaias ‘Bald head Moll,’ now lives in the brick house corner of An and Richmond streets, with Hen Dean, a darkey, and Angenette Piper, a white gal [...] Moll deriving a small income from her intercourse with mokes.
1860		N.E. Police Gaz. (Boston, MA) 18 Aug. 7/3: The clerks in a certain freight office [in Rochester, NY] had better [...] keep out of a certain brick house close by.

2. (US milit.) a psychiatric institution [‘probably derived from the kind of building in which is housed the general hospital for insane soldiers at Washington, D.C.’ M’Govern 1906].

1906

C. M’Govern ‘Soldier Sl.’ in Sarjint Larry an’ Frinds n.p.: brick-house:—Insane asylum.