I think they use the same thing that web crawlers use. If Google's crawler couldn't access the content of the page (or could only access a limited amount of content), it would likely rank far lower in search results