Amazon.co.uk Widgets

Follow Us:

Call Now! + (123) 456-7890

How to Create robots.txt

You can stop Search engines to crawler into pages which you don’t want to index. If it don’t index then that page won’t be show on Search Engine

For example

If you don’t want to index Admin Panel then you will put this syntax into robots.txt

User-agent:*
Disallow:/Admin Panel/ 

Now, If you don’t want to index any folder or page

User-agent:*
Disallow:/<URL> 

<URL> will be change with your page URL which you do not want to index in Search Engine

Once your robots.txt is ready then upload it into your root folder

www.domain.com/robots.txt

Now if you want to restrict any page then you can do it by meta tag, you can put this syntax in head

<meta name=”robots” content=”nofollow”><meta name=”googlebot” content=”noindex”>

Remember above syntax will be for those page which you don’t want google to crawler for index in google searching  

  1. What does robots.txt mean?

    robots.txt is a file from which you can control crawler to index your pages into Search Engines

  2. Where is the robots.txt file?

    robots.txt file must be in your root folder. e.g http://yourdomain.com/robots.txt

  3. Why robots.txt is important?

    robots.txt is important when you doing SEO for your website.

user-gravatar
Asad Iqbal
No Comments

Post a Comment