Green’s Dictionary of Slang

fag end n.

[SE fag, to droop, to decline, to flag]

1. [17C+] (also butt-end) the last part or remnant of anything.

2. [19C+] a fragmentary part of a speech or conversation that one might overhear, just as it tails off.

3. [mid-19C+] (also fag) the butt of a cigarette or cigar.

In compounds

fag-end man (n.)

[1910s–20s] a man who collects cigarette ends from the pavement.

In phrases

pick up fag-ends (v.) [sense 2 above, but pun on sense 3]

[1910s+] to listen in to other people’s conversations and attempt to comment upon them or join in; esp. as don’t pick up fag-ends.