So, let me preface this with saying that I am quite familiar with what you're talking about. My partner has published two books, I used to design book covers before AI killed that, I know many writers, etc.

Extensive worldbuilding has its pros and its cons. Having a good understanding of the world that you're writing in will help you keep your story consistent. Your characters will be more well-rounded, because even if they don't directly reference your worldbuilding, they should be acting with the world's constraints in mind. You won't have to make as many snap decisions that you have to remember to account for later, less plot holes to fill. It's not about "nobody is going to notice", because if you have solid worldbuilding, it'll flow through book and they'll see it without realizing it.

The problem is when it starts getting in the way, like you're describing. You say you feel stuck perfecting the setting, and making sure everything makes sense. Forget about other people, why is that important to you?