Not to mention the time spent. I'd rather spend that time and effort doing other things.
I've compiled plenty of programs, even the kernel. But that's a means to an end. I've hunted down drivers and manually resolved dependencies to get them to compile. I've run on weird old platforms (Minix on a 286, but I don't think I got the Ethernet drivers working there). Sure it's interesting, but eventually I'd rather actually use the thing.
If you really want to compile your system, go do Linux From Scratch.