Sounds like 4chan rolled their own server-side page cache. It probably made more sense for 4chan to do that then, back when the internet was still pretty new.

Nowadays there are a few packages that perform a similar function, like Varnish. Varnish picks up requests from the internet and where possible, it returns the response as static HTML from its own cache.

There are other types of caching at different stages of the request lifecycle that can reduce the gap between dynamic and static website performance.

Lemmy is built as a separate web app and back end that communicate via APIs. It makes it easy to support apps and alternate web interfaces, unlike Piefed which only started adding an API reasonably recently and doesn't have a fraction of the eco system that Lemmy has.

But I think an important part is that Lemmy is explicitly customised for users. While bots are getting the same logged out version, the intent of Lemmy is not to be scraped by bots but it's to be custom for each user. You're supposed to log in, subscribe to communities you like, block people you don't like, and have posts you've already seen hidden (if you set this in your settings).

Serving a custom page to each user based on their settings, blocks, and subscriptions isn't really suited to static pages.