Height is about 80% genetics, 20% environment (nutrition and general health). Grandparent's genetics are a factor since that is where the parents inherited their genes.

A child inherits two copies of every gene from their parents, one copy from each. Your family could be like mine, where my grandfather was quite tall and my grandmother was unusually short, suggesting my grandfather had double pairs of genes for height and my grandmother had double pairs of genes for... not height. Their children, my mother and her siblings, then had an even assortment of tall and short genes.

Her brother married a woman whose parents followed the same tall father/short mother pattern and they had three sons: two of about the same height as the parents (even tall/short gene mix) and one who is about 20 cm taller than the rest (few or no short genes).

My parents had a big height gap, mom 163 in cm, dad 193. So the average is 178. My sister is 178cm, I am 175. I guess that's why I expected my daughters to be the average of our heights. My brothers are 188 cm. So all of us are in between the heights of the parents but there's a lot of "in between".

My mom's parents were actually both quite tall though, her mom my height, which for back then was really tall, she (my mom) told me she kept expecting to get taller than her mom like the rest of her friends and it never happened.

And my dad's parents were both quite short, about 165cm and all their kids as tall as my dad, so obviously there are other factors. I have no idea why my dad and his brothers were so tall. Like nearly a whole foot taller than both their parents.