Occasionally I go over to Googles’ Webmaster Tools to check out my sitesmaps. Today I checked out the Webseo sitemap and was knocked off my chair when I had a look at my robots.txt file.
Search engines, mainly the big ones like Google, Yahoo and MSN (Bing) look for a bit of code in the meta area of your header code to find directions on what you want indexed (crawled) and what you dont want indexed. I was shocked to see that I told the earch engines crawlers (bots) that I wanted them to crawl everything. I remember I did this about 2 years ago when I was young an niave. I thought that if Iallowed the search engines to crawl everything, I would getting ranked quicker. Nahhhhhh thats wrong!
Now that I am older and wiser, I have learnt that I only want the bots to crawl the content that I want seen in the search results. This could be my posts or even my images. You can get a lot of traffic from the images search, so don’t ignore your images.
So after having a look at the robts.txt for webseo.com.au/blog/ and getting back on my chair I decided to optimize it a bit better. This optimization will be different for everyone, but webseo is a wordpress blog so it was pretty simple.
I didnt want the bots to crawl my Wordpress Admin folder, cgi folder, includes folder, wp-content folder or the search folders. I just wanted them to crawl the content. Heres what my robots.txt file started to look like:
User-agent: *
Allow: /
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /search/*/feed
Disallow: /search/*/*
Disallow: /
User-agent: Mediapartners-Google
Allow: /
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Image
Allow: /
User-agent: Googlebot-Mobile
Allow: /
#User-agent: ia_archiver-web.archive.org
#Disallow: /
Sitemap: http://www.webseo.com.au/blog/sitemap.xml
I than added the User-agent to allow Googles Google Adsense bot, Google images bot and Google Mobile bot to crawl my site. I then thought that here was an opportunity to add another link to my sitemap.
So the message is to make sure you have a robots.txt file on your server in the root folder. Then make sure that you only allow the search engines to crawl the content you want them to index. If you don’t want it to show up in the search engines search results, make sure you stop it in your robots.txt file.
If you have a Wordpress website, fel free to copy my code above. Make sure you change the sitemap url, or else Google will only skip over to my website. Cheers
{ 8 comments… read them below or add one }
can u tell me? i wanted to crawl my blog in the google
so what is my robot.txt
Ali a robot.txt file is a file you place on you hosting server to tell the search engines what files and folders you want them to crawl (read) and then hopefully add to their search indexes. Do you have one on your website?
yes , this is my website, http://alisoft7.blogspot.com
actually, since , i submitted the website in google webmaster. i have faced one problem regarding robot.txt file. “Restricted by robots.txt (27)” i have see this error.
so how should i remove this error.
Secondly, mostly, i have deleted my categories labels from my blog. so. is this creating me in problem?
can u solve my problem
Ali I looked at your robots.txt file and it is very basic. A robots.txt file should tell the search engines what you want them to see and what you dont want them to see. In your case I would do a search on Google for robots.txt “blogspot” You will find lots of information on how to develop a good robots.txt file. Then you should’nt get any eoors at Google Webmaster.
As far as your category problem, it shouldnt be a problem, but if it is I can not fix it for you. I clicked on a category label and it worked fine for me.
Steve
ok
can i send you my screenshot of webmaster tool
Ali yes send me the screen shots. Send to steve@webseo.com.au
Steve
ok i m sending plz tell me, how should i deal now?
my blog http://www.diyanswerdirect.com/blog hasnt been indexed for two weeks and my site http://www.diyanswerdirect.com has been indexed but never gets updated. Is there anyone who can tell me when google or bing updates there search engines please thankyou lee