Now you have two package managers. The problem is that Nix needs to create a folder structure that matches what your language package manager wants to see.
Even if the language package manager goes away, languages have slight differences that can't be totally abstracted away. C/C++ have .h and .c files. JS has its own thing, C# has a .dll. Each language has difference opinions on how symbols get resolved.