You are right about that, and I think in context it was meant as more of a joke? Because I remember a discussion somewhere in the canon that the counts were limited to 20 armsmen bodyguards as a hedge against them trying to create private militias that could threaten the power of the emporer.

But I would defer to people who are much more expert on the canon than I.