Most people who have a website of any description want to make sure they are doing all they can to lift its position in the search engines. When I say search engines, of course I mean Google primarily. This is the search engine the majority of people use, so it is vital to ensure you understand how a site is searched and indexed in the search engine itself.
But not everyone is aware that you have some degree of control over how Google’s crawler (otherwise known as Googlebot) does this. Luckily Google has created a section on their site at https://code.google.com/web/controlcrawlindex/ to give you all the step by step information you need in this respect. Being able to control how the Googlebot crawls over your content could be extremely useful when you are tweaking and changing your website.
The step by step process is broken down into easy bite sized stages. The section on controlling crawling is particularly interesting, especially if you want to stop the Googlebot from getting to parts of your website you don’t want it to reach. Examples are also given to try and demystify the process for webmasters who are not familiar with this. You’ll certainly learn more about the robots.txt file by reading the information they give you. On some occasions you may not need it and in others you will need to tweak it to gain the control you want.
Depending on the nature of your website and how you want it to be crawled, you may not even need any of this information. But it is certainly good to have it at your fingertips so you can control the indexing of your site’s pages in the future if need be. This would especially apply to any new sites you have in mind.
Previous Post: How to add ‘Deja vu’ with Google Re-marketing