People have done it. The problem is the huge binary blobs for hardware. Its difficult without the hardware company themselves giving you the specs to make your own.

One of the reasons fairphone is supported on many alternate platforms.