The AboutUs Bot
The main job of the AboutUs Bot is to generate basic initial pages and analysis about websites. The bot pulls initial page data once when a page is first created. Website analysis may be pulled multiple times but is cached to prevent continuous access by the bot. We want the bot to be well behaved, if you are seeing otherwise please contact us and let us know.
The AboutUs Bot User-Agent string contains the following:
- AboutUsBot/VERSION (PURPOSE; http://www.aboutus.org/Aboutus:Bot; firstname.lastname@example.org)
- AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; email@example.com)
The current AboutUs Bot version is Harpy.
Blocking the AboutUs Bot
Using a robots.txt file, you can choose to not have the About Us Bot access your website. This doesn't mean that we won't create a page for your website. Our members still have the opportunity to contribute their own content describing your site.
To prevent the AboutUs Bot from accessing your site in the future, please include the following lines in your /robots.txt file.
- User-agent: AboutUsBot
- Disallow: /
The AboutUs Bot will also honor a rule like this in your robots.txt file:
- User-agent: *
- Disallow: /
However; this rule will prevent all well behaved bots, including Google, from crawling your site.
For more information about robots.txt read our robots.txt article.