Me and most others

The "most others" here is a heavily self-selected group of people who don't want to compromise on any of their values and treat any effort to grow as a threat.

All of this to say, it's fine if you say "Yes, we are small and I want it that way because if it gets any bigger we will be surrounded by people who do not uphold the same values we do". The problem is that you're arguing "We are only small because of $random_reason (network effects/evil capitalists/not enough funding/etc)", as if "being small" was determined by external factors and not something that you can control.

That's the point of disagreement. I think we can control this and we can bring more people here, but it's just that you don't want to do it if means sacrificing your ideology.