Green’s Dictionary of Slang

casual n.2

[SE casual clothing]

1. a working-class youth who dresses in casual clothes, often designer labels, but whose accent and lifestyle remain proletarian.

[UK]Guardian Guide 17–23 July 28: Fighting men have a fine tradition to live up to: Mods v Rockers, Punks v Soul Boys, Casuals v Other Casuals.
[UK](con. 1979) N. ‘Razor’ Smith A Few Kind Words and a Loaded Gun 207: Rude-boys, smoothies, and casuals.

2. a football hooligan who adopts such a style.

[Scot]I. Welsh ‘Wayne Foster’ in Acid House 82: They fuck casuals. No interested in the fitba, these cunts.
[Scot](con. 1980s) I. Welsh Skagboys 194: Lexo fae the casuals wis telling us this this QC fucker [etc.].