Each VM can be sized appropriately for the demands of the container. With docker desktop, you can't have a container use all of your system cores without making the VM have access to all of your cores all the time always. One of the biggest benefits (imo) of running containers on a Linux workstation is that if you don't define a CPI limit, a container can use all the compute/memory on your system. You just can't do that with Docker desktop. This also affects multi threaded container builds when you're using buildkit.

Being able to spin up a vm to build a container with all cores accessible to it, and then run the actual container with a smaller number of cores would make container builds so much faster.

EDIT: I've looked, and it appears that podman desktop also does 1 big VM, rather than having 1 VM per container.