Brought up on football from the late 70's and going away throughout the 80's I found it was sensible to not wear your colours for fear of a slap. This, coupled with the fact I have always liked a drink and it meant getting into boozers was always easier. I always wore a badge though. A bit like me, nice and discreet.
I always have a Charlton badge on when out and about. I wear a Charlton woolly hat during the coldest months. And I may even where my Lee Bowyer scarf if the wind is particularly biting. I bought the Charlton duvet set at the Wednesday match. It only cost a cock & hen. The missus wasn't too impressed, but she soon relented when I agreed to buy her a new set of saucepans. Ha!
I always wore a red & white scarf over to the Valley, regardless of the weather. My visits to away grounds probably totals about six in the last 20 years. Always thought it prudent to go in plain clothes. Nowadays the home game scarf is only worn when it is cold. Not something that's happened so far this season.