Glossary/Robots.txt
< Glossary
edit Robots.txt
Robots.txt is a file that website owners put on their web servers to give instructions to robots, crawlers or spiders: computer programs that travel around the web, looking for specific kinds of web pages and content.
It resides after the domain name - /robots.txt - i.e. http://www.aboutus.org/robots.txt
For more information on robots.txt, read:
