MacOS and docker is asking for trouble. It requires all kinds of hacks and awareness of where things are being built or mounted, which architecture is being emulated because it runs in a VM, memory limits, swapping, and so on. It's no surprise build times are a problem.
I imagine your build pipeline is for aarch64, you have made modifications to numpy itself (build flags or something), you're using a special distro in the pipelines, or the rebuild is unnecessary. For me numpy has always been a simple "pip install numpy" or "poetry install" when not on nixOS.
My guess is that if you used non-emulated Linux, your local build issues would be reduced and also sped up. But I can't say that with any certainty because I don't have nor require nvidia in my Linux rig, nor do I know your setup very well.
You do however fall quite nicely into the criteria I shared. Were you outside of that, python would probably be much much easier (aka no non-pytjon deps).