Maybe I'm saying something that's already been stated or agreed upon, but it seems to me like we simply have a case of a demographic that is very largely skewed.
For instance, if you take into account the fact that Commoners should dramatically outnumber Holders in the Empire (something along the ratio of like 10,000,000:1) then you'll see the numbers more like this:
For every 10,000,000 Commoner families, there is 1 Holder family. That one Holder family will have anywhere from 10,000 - 100,000 slaves. Which means you have roughly 50,000 slaves per 10,000,000 people in the empire. Which is 0.5% of the population being slaves.
So, in a population of 21 trillion, the numbers would look like this:
"Free" population: 21,000,000,000,000 (21 trillion)
Slave population: 105,000,000,000 (105 billion)
To correct this, you simply have to elevate the number of slaves per Holder family, or the number of Holder families per Commoner family.
If we elevate the number of Holders per 10 million people to something more like 10,000,000:5 and keep the same ratio of slaves. Then you end up with something like 250,000 per 10,000,000 people which is 2.5% of the population being enslaved. So your numbers would be:
"Free" population: 21,000,000,000,000 (21 trillion)
Slave population: 525,000,000,000 (525 billion)
So on, so forth until you reach the desired ratio. The key is to determine what you raise: the number of slaves per Holder family or the number of Holder families per Commoner family?
No matter which path you take, you can arrive at the desired numbers; it comes down to what picture of the Empire you want to paint. Do you want a vastly Commoner Empire with few Holder families responsible for literally millions of slaves each or do you want a more spread Empire with slightly larger numbers of Holders and only thousands or hundreds of thousands of slaves each?