You can stop Search engines to crawler into pages which you don’t want to index. If it don’t index then that page won’t be show on Search Engine

For example

If you don’t want to index Admin Panel then you will put this syntax into robots.txt

User-agent:*
Disallow:/Admin Panel/ 

Now, If you don’t want to index any folder or page

User-agent:*
Disallow:/<URL> 

<URL> will be change with your page URL which you do not want to index in Search Engine

Once your robots.txt is ready then upload it into your root folder

www.domain.com/robots.txt

Now if you want to restrict any page then you can do it by meta tag, you can put this syntax in head

<meta name=”robots” content=”nofollow”><meta name=”googlebot” content=”noindex”>

Remember above syntax will be for those page which you don’t want google to crawler for index in google searching  

Leave a Reply