Yeah, there's a certain risk for rolling with your own engine, but if you start the project with the idea of having a custom engine you probably know what you're doing and have taken into account the complexities of having a custom engine. IMO if you're a group of small experienced devs having a custom engine is not really a show stopper, if you're a junior the project probably isn't even getting off the ground.

But changing the engine mid-project is almost always a huge decision and more often than not a killing blow for most projects. Depending on the stage of the project you're guaranteeing adding a year or two to your development. It's better to accept the limitations of the existing engine and compromise on the vision rather than swap engines in hopes of realizing the vision that got refined during development.