Time. Your build environment can be potentially compromised. Takes more tools to build then install increase maintance, chances for failure to update, and increase attack surface area.

Removing software can be much harder if not installed by a package manager as well, which can make upgrades a PITA

That said yeah I do it from time to time. Build in one container, install in an other, and you mitigate some of that concern.