“Software encoding” is just kinda a colloquial term for running on the CPU. Because the alternative was dedicated hardware for the task.

As for the Vulcan part I have no clue how that works. Does vulkan have an Open equivalent or something?

This Reddit comment says it’s more a compatibility layer.

The Vulkan AV1 video encode just wraps the native fixed function calls in a compatibility layer that makes it easy for ffmpeg users and libavcodec integrators to support multiple vendors and OSes. It does not add a new Vulkan-compute based encoder that enables AV1 encoding on GPUs that don't officially advertise it. So expect it'll work with Nvidia Ada/Blackwell, AMD RDNA3/4, Intel Arc, and other hardware that officially support for AV1 encoding via their respective APIs.

If you're already happy using av1_nvenc, just use that. At best, av1_vulkan should perform about the same. At worst, it may add slight overhead and limit options/features compared to the latest native API on a given platform. However, Vulkan Video seems to be well-supported for Nvidia.