Because the "Worldcon Must Change to Suit Me!" crowd would never set up such a crazy situation as the members actually deciding what to do themselves (members obviously being too stupid to make up their own minds for themselves), they have great difficulty believing that anyone else ever did it that way. Thus there must be a Ruling Cabal (call them the "Secret Masters of Fandom") who really run things, so you just have to find those people and subvert them.
Alternatively, if you really are so stupid to have set up an organization whose mere members get to decide how things are run and where their conventions should be held, then those 4000-5000 people should just go away and never come back and give their property to Someone Else who can Do Things Right, which in this case means throwing away all of the things that those 4-5K people enjoy doing and Doing Something Else because a different, larger group like doing them.
There is nothing wrong with Anime Expo having 50K anime fans enjoying themselves, or ComicCon having >100K enjoying themselves, or for that matter Comiket having something like a quarter-million people as I recall. Why do so many people want the World Science Fiction Convention, which primarily (but not exclusively) celebrates written science fiction and fantasy, not comics, anime, movies, television, ballroom dancing, baseball, trains, regency-style dancing, costuming, or any of a number of different popular activities to die right this second to validate their own personal hobby entertainment preference? Why do they, to quote someone who recently Got It, want to "appropriate the cultural identity of thousands"?
Worldcon is a club. It's a club with between four and five thousand members, but it's a club, and it's run very similarly to how a club of only about two hundred people — the approximate attendance of the first Worldcon — might be run if organized by a group of opinionated know-it-alls who trust nobody but themselves to run things. And the members, on the whole, like it that way. Yes, the club might eventually go out of business when nobody shows up for their meetings, but until then, what is so wrong with the members running their club the way they want to run their club for the benefit of the other members of the club?