Difference between revisions of "Learn/404-Errors-Drive-Visitors-Away"

Revision as of 00:01, 8 October 2010

What does a "page not found" message mean?

Have you ever typed a URL in the navigation bar only to receive a "404 error message" or learn that the page no longer exists? It can be frustrating and feel like a waste of time, especially when repeated attempts produce the same results. It's not good for site visibility and it certainly isn't good for a customer trying to find your page. It's similar to a customer crossing town to visit your business and finding the "closed" sign on your door. Will the customer return? They might if you have a niche business, but if you're selling something that can be obtained elsewhere, they probably won't return. The same can occur with your website.

When and why do these messages occur?

"Page not found" or "404 error message" is the standard response to a server request for a site that is either dead, broken, or no longer exists. In other words, when a server looks for a specific URL and can't find the requested site, the server will send a message indicating the page cannot be loaded or opened. Link rot is a slang term for the same occurrence.

While there are many reasons for a link to be broken, it is frequently due to some form of blocking such as content filters or firewalls. Links may also be rendered inactive when the server hosting a page stops working, or relocates to a new domain name.

How to avoid a 404 error message?

The following simple rules will stave off a 404 error message or page not found notice when searching for your URL.

• Redirect on-site rather than take your page down. When someone arrives at your page they should find something useful even if it's simply a message inviting them to to visit your new homepage or contact you for more information. Finding "something" instead of "nothing" is the way to build trust among viewers.

• Redirect from the host - Another option is to check with your hosting company to see if they offer a 404 redirect. When a user enters the URL for your former page, they will automatically be redirected to your index or home page. It's the easiest way to ensure that a potential customer is never lost due to a page not being found.

• Submit new pages to search engines - It's a good idea is to resubmit new pages to search engines for reindexing. Although it's nice to inform visitors that a page they're looking for no longer exists, it's even better to have the new page with all the information the visitor requests when they perform a search.

• Keep an eye on your page listings - Do yourself a favor and check regularly to make sure your site listings aren't taking potential clients to 404 pages. Go to a search engine like www.altavista.com and type www.yoursiteURL.com in the search bar.

If the search results index a page that no longer appears on your server:

go to the search engine's submissions page and submit the new URL for the page that covers the information of the old page, and make note of the name of the page that's no longer available, and then rename one of your existing pages with this old title. This way you can ensure that people who search for information on your site will be taken to the correct page until the new page is indexed -- and ensure you don't lose potential sales because of missing pages.

5. Get organised with updates

Keep notes: maintaining a Website is a big job, so don't rely solely on your memory. Keep a notebook or digital log full of your site changes to help refresh your memory and ensure you follow up changes with the search engines.

How do I prevent robots scanning my site?

There is no easy way to prevent all robots from visiting your site. However, you can request that well-behaved robots not visit your site by adding these two lines into your robots.txt file:

User-agent: *

Disallow: /

This asks all robots to please not visit any pages on the site.

Things to be aware of

• You need a separate "Disallow" line for every page, file, or directory that you want to exclude.
• You can't have blank lines in a record. Blank lines are used to separate multiple records.
• Regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Lines like "User-agent: *bot*", "Disallow: /tmp/*" or "Disallow: *.gif" won't work.
• Everything not explicitly disallowed is considered fair game to retrieve.
• The robots.txt file is a polite request to robots and not a mandate they have to follow. Robots that scan the web for security vulnerabilities, email address harvesters used by spammers, and other malicious bots will pay no attention to the request.
• The robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to examine. If you want to hide information, password protect the section instead of trying to rely on robots.txt to hide information.

Examples