That travelling to planets takes quite a long time? For more, read here.
Instead of looking at it as a case of exclusion, why not figure out a way to include those who decide on the "path less traveled" in other ways? When the world puts up a "No Slavers Allowed" sign, why not put up your own "Only Slavers Allowed" to counter it?There's always alternatives.