Well behaved bots should see the origin in the headers and only crawl those.
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Well behaved bots should see the origin in the headers and only crawl those.