You can stop Search engines to crawler into pages which you don’t want to index. If it don’t index then that page won’t be show on Search Engine
If you don’t want to index Admin Panel then you will put this syntax into robots.txt
User-agent:* Disallow:/Admin Panel/
Now, If you don’t want to index any folder or page
<URL> will be change with your page URL which you do not want to index in Search Engine
Once your robots.txt is ready then upload it into your root folder
Now if you want to restrict any page then you can do it by meta tag, you can put this syntax in head
<meta name=”robots” content=”nofollow”><meta name=”googlebot” content=”noindex”>
Remember above syntax will be for those page which you don’t want google to crawler for index in google searching
- What does robots.txt mean?
robots.txt is a file from which you can control crawler to index your pages into Search Engines
- Where is the robots.txt file?
robots.txt file must be in your root folder. e.g http://yourdomain.com/robots.txt
- Why robots.txt is important?
robots.txt is important when you doing SEO for your website.