It really does smell awful and fair or not, in most people's mind, it changes the perception of the event from normal partying to something else. Combine that with loud explicit music and you pretty quickly make it an environment that a good number of paying fans are not going to like.
I certainly can't stand it. I know plenty of people that smoke or take edibles and I have no problem with it, but when I smell it in public, it immediately makes the area seem ten times trashier to me. Of course usually the people smoking in public are trashy, so that's probably a big part of it.