Like many have said, the main disadvantages are:
- Any software installed outside of your package manager is difficult to uninstall or maintain.
- Compilation time can be an annoyance.
The main advantages are:
- Removal of unused features.
- CPU specific optimizations.
If you're interested in compiling your own software maybe consider using a distro whose package manager allows to compile stuff, Gentoo is the obvious choice, but Debian based distros can also do that.