09-30-2015, 04:33 AM
|
#1 |
|
Registered User
Join Date: Sep 2015
Posts: 35
|
What is robot.txt
|
|
|
|
09-30-2015, 10:33 AM
|
#2 |
|
Registered User
Join Date: Sep 2013
Posts: 887
|
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
Buy Flavored condoms online Online shop for condoms
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
09-30-2015, 11:22 AM
|
#3 |
|
Registered User
Join Date: Sep 2015
Location: Dublin,Ohio
Posts: 18
|
if you want disallow your pages, who does not cache or index in google then you use robots.txt . Google web masters create robots how to crawl and index pages on their website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
09-30-2015, 11:53 PM
|
#4 |
|
Registered User
Join Date: May 2015
Location: Forum
Posts: 2,517
|
Robots.txt is a text file that contain instructions for search engine crawlers. They list webpages that need to be crawled and disallowed webpages for search engine bots.
__________________
█ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | Servers with Free IPMI Setup |
|
|
|
|
10-01-2015, 12:03 AM
|
#5 |
|
Registered User
Join Date: Mar 2015
Posts: 124
|
Robots.TXT is a text file which you place on your site to tell Google bot which pages you don't want to crawl. Example:
User-Agent: * Disallow: /s/ Disallow: /cgi-bin/ Disallow: /myadmin/ Disallow: /admincp/
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-01-2015, 12:31 AM
|
#6 |
|
Registered User
Join Date: Sep 2015
Posts: 75
|
Using robots.txt you can restrict Google and other search engine spiders from crawling your website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-01-2015, 03:07 AM
|
#7 |
|
Registered User
Join Date: Sep 2015
Posts: 35
|
Thanks to all
|
|
|
|
|
10-02-2015, 02:14 AM
|
#8 |
|
Registered User
Join Date: Sep 2015
Location: UK
Posts: 447
 |
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
|
|
|
|
|
01-11-2016, 01:05 AM
|
#9 |
|
Registered User
Join Date: Jan 2016
Posts: 21
|
Great Article.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | |
| Display Modes | Rate This Thread |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| what is robot.txt? | rubysethi | Search Engine Optimization | 0 | 12-19-2014 11:27 PM |
| How to know about robot.txt ? | vanaja | Search Engine Optimization | 10 | 10-08-2014 04:39 AM |
| how can Create the Robot.txt | pankaj | Search Engine Optimization | 3 | 10-08-2014 03:43 AM |
| how can create the Robot.txt file | eagleeye dental | Search Engine Optimization | 0 | 09-01-2014 09:40 PM |
All times are GMT -7. The time now is 06:12 AM.