It only grows due to testosterone exposure during pre-natal development. It can grow a little bit in AFAB individuals who take testosterone but if you're AMAB and it's fully developed it's as big as it'll get for you.
Your perspective is more correct, because it starts out tiny and gets bigger. Clitoris is the default.