NOTICE:
This content is in the "archives" of a retired blog called Gadgetopia. It has been moved to this subdomain as it is no longer considered relevant to the site. It is being hosted here for a indeterminate period of time. Its existence at this URL is not guaranteed, and it may be removed at any time. If you would like to refer to this content in the future, you are encouraged to save it to your local file system.

Some content from Gadgetopia was moved to the technical blog on deanebarker.net

The Googlebot and Subscription Sites

Originally published by "dbarker" on 2008-09-11 13:47:00Z

: Registration/subscription sites: You are apparently allowed to detect the Googlebot and allow it to bypass your login page and crawl content behind it. This means that subscription content will appear in the Google index, but people will be prompted to login when they click a search result.

Google News does include sites that require users to set up usernames and passwords to access content. Since crawlers can’t currently fill out registration or subscription forms, nor do they support cookies, we need to be able to circumvent those pages in order to successfully crawl those sites. The easiest way to do this is to configure your servers to not serve the registration or subscription page to our crawlers (when the User-Agent is “Googlebot”).

It also means, of course, that you could change your User Agent string and bypass the login page for any site that does this.

Via a good discussion on Reddit about Experts Exchange and how they do some funky things with Google.

Just for giggles, I downloaded the User Agent Switcher Firefox extension and confirmed that when you change your User Agent to impersonate the Googlebot, all the answers appear uncloaked on Experts Exchange pages.