Ok, I did test it.

1. I set up DNS server on my PC using Technitium.
2. I add zone a.local and add record with device A's IP.
3. I add zone b.local and add record with device B's IP.
4. On device A (Android 11) I go to Setting » WiFi and set DNS to PC's IP.
5. On device B (Android 13) I go to Settings » WiFi and:
   • Change IP to static
   • Change both DNS1 and DNS2 to PC's IP.
6. I ran some file served over http on both devices.
7. I visit http://a.local/ on device B ❯ A's Files accessed
8. I visit http://b.local/ on device A ❯ B's Files accessed

Everything works.

Note:

• it took DNS server about 1min to add the records (idk why).
• It seems devices caches NAME_NOT_RELOVED so if it failed it will be cached as such for at least a few minutes.