I know exactly what you're going through, as I want to throw a party for everyone. It's even more of a problem for me because I'm actually anti-somebody (pirates), and can't be seen inviting them. As long as you're on the ball and can ban trolls, you're okay. There's the obvious threat of a party-crash, where tons of unaffiliated alts descend upon your channel and spam ASCII dicks into it faster than you can ban them, though. But that's rare.
However, in character it gets to be an issue for safety reasons. While it's easy to just dismiss the OOC trolling thing, IC there is a real and present danger from just inviting anyone who wants to show up. For example, my character doesn't see you ready to ban trolls. She sees your character inviting the likes of dangerous folk. Illegal slavers, rapists, murderers, Sansha, etc. She sees very real threats to her well being, and she's also aware that she is not allowed to bring her own protection.
Without any way to defend themselves, and without any assurance that dangerous people won't be allowed in - my character takes issue with Lasa's events, because they aren't something she can feel safe in. It would be negligent of Katrina to show up to an event like that, as it would be putting herself in a severely disadvantageous situation.
It's not a personal attack on you or your events. It's just my character being paranoid.
I suggest, strongly, that you have holograph projectors and such installed at your events and ADVERTISE that you do. Those who are paranoid can attend your event via holographic virtual reality, and those who are more gutsy can show up in person. That way, no harm no foul.