You can't impersonate search engine crawlers, because the server can verify that it's really a search engine crawler

See this from google: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests

Here, duckduckgo: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot

A lot of article websites whitelist Google's crawler, right? Can't that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though