Oh, see, unlike on x86 where you have the ACPI to detect hardware with minimal device quirks (still a lot of them), everything else doesn't have that. Well, except some Qualcomm chips, but their implementation sucks and basically only works reasonably with Windows and Windows Phone. So you need a device tree blob (DTB) to tell the kernel where everything is. But enabling all of the drivers in a single kernel build makes it not fit (the partition for that is traditionally quite tight), so you make different kernels per device.

AND, on Android in particular, lots of features need device specific configuration for all of the small stuff like the proximity sensor and the cameras (a LOT more complex than webcams). This + the need for OEMs to insert their own spyware and the already existing tradition at the time to make device specific images made the decision stick around. There's GSI, which basically forces the OEM to write drivers and all of that with a stable-ish API to make universal images possible, but it results in a system with lots of tiny inexplicable problems that slowly make you loose your sanity in my experience.

How postmarketOS handles it is that there are basically meta packages per device that depend on the kernel package appropriate to the device (sometimes for a whole platform or SoC, having multiple DTBs inside for each device) that flashes itself to the appropriate partition via a post installation hook, as well as all of the config files for apps that need device specific stuff and don't already have it upstream (like camera apps).