what is the use of robot.txt? - Site Owners Forums

hap53144 · 03-23-2016, 08:58 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
Thanks

03-23-2016, 08:58 AM	#15
hap53144 Registered User Join Date: Feb 2016 Posts: 34	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results. Thanks __________________ Create Any Stunning Animated Graphics with To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. In Just A Few Clicks \| New trick works in To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. for ANY Facebook Ad \| Turns Your Videos Into Sales Machine at To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
what is robot.txt?	rubysethi	Search Engine Optimization	0	12-19-2014 11:27 PM
How to know about robot.txt ?	vanaja	Search Engine Optimization	10	10-08-2014 04:39 AM
how can Create the Robot.txt	pankaj	Search Engine Optimization	3	10-08-2014 03:43 AM
how can create the Robot.txt file	eagleeye dental	Search Engine Optimization	0	09-01-2014 09:40 PM

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)