thoughtfulAs far as "not being invited because we don't like you" goes: A lot of these people really Don't Get It. In my book, a fan-run SF/F convention, even up to a Worldcon (remember, I co-chaired one, so this is the philosophy I brought with me in 2002) is a party we hold for each other. It's not a commercial venture except insofar as we have to cover our costs. (If I had enough money to do so, I'd underwrite the costs personally, because of how much the community has meant to me since I found my way to Fandom in 1984.)
My metaphor is that Fandom is a Pot-Luck Dinner. We have lots of acquaintances, we all like to eat, and we decide to hold a big pot-luck dinner where we can share our favorite dishes and socialize. We have so many friends and friends-of-friends that none of us owns a barn big enough to hold everyone, so we have to rent the community center. None of us is wealthy enough to do that on our own, so we ask everyone coming to not only bring a dish, but also to kick in part of the cost of renting the hall, plus the tables, chairs, etc. We're not running a restaurant, and we're not making a profit, just covering the cost of putting on the event. Some of us volunteer to schlep tables and chairs, others volunteer to wash dishes, and so forth. Everyone brings something. That means some of the food is stuff I personally like, and other stuff I hate. But that's okay: I eat what I like, and leave the rest for those who like green bean casserole.
Somewhere along the way, we got the idea of voting among ourselves for what the best dishes were. ("Best Appetizer," "Best Main Course," "Best Dessert") And we started holding this big pot-luck in different places so as to share the fun with our far-away friends who couldn't necessarily make the trip to Our Fair City.
Well, now we've got people who started coming to the pot-luck, paying the share of the hall rental, and are angry that we've been choosing things they personally hate to eat, and have decided that they want to knock over all of the tables with food they dislike and insist that the rest of us eat that stuff that they personally like, because they say so. It should not be a surprise to them that the rest of us start saying, "I don't think we want to invite you anymore; you're making the rest of us very uncomfortable by your anti-social behavior." They respond with variations of, "I paid my cover charge to your restaurant, and you're responsible for feeding me things that I like, and to not serve food I don't like!" and they don't understand why that response alternately baffles or infuriates the rest of us.
Possibly these people who take the attitude of "You'll ruin your turnout and none of my fans will show up" are mistaking a pot-luck dinner for a for-profit buffet restaurant. The latter needs to cater for the tastes of everyone willing to pay; the former only needs to cater to the desires of the people who they like and with whom they want to associate.
One of the cultural disconnects of "winning the culture war," as Moshe Feder once put it, and of SF/F becoming so much a part of popular culture, is that you have a lot of people attempting to apply the commercial model to a fundamentally non-commercial venture. When you point out to these people that "This isn't our business; it's our hobby, and we're not in it to make a profit," they look at you blankly, because they can't conceive of anyone ever doing that much work unless they were Getting Paid.