I'd venture to guess that "most people" that think that way do not live in professional sports cities. I lived in suburban Chicago for 11 years and now here in Houston for the past 6 years -- jerseys are a common thing for adults of all age ranges. It is very common on gameday to see both men and women with Texans and Astros jerseys. I wore my Blackhawks jersey quite often when I lived in Chicago, even on non-gameday sometimes. NBA jerseys have not been common for adults 25+ in either city.
The real funny thing is that you do not have to be a fan of the team in your city to be seen sporting a jersey from another team. Case in point, my wife, who is from Chicago, is a lifelong Packers fan. She wears her Green Bay Packers jersey every Sunday during football season. Surprisingly, I have seen quite a few Packers fans in our area. And, of course the irritating Cowboys fans with their jerseys and paraphernalia are ALWAYS around wherever you go.