It's not that it's a bad idea, it's more that it's basically impossible at this point.
Bazel can't use Cargo, CMake and so on for downloading & building packages because it is trying to do cross-language builds the right way - a single build graph that includes everything. No other third party package managers/build tools actually exposed their build graphs - only commands to perform builds, which are too coarse for Bazel.
To put it simply, its necessary if you are doing things at the scale of Bazel.