01-04-2017, 02:52 AM
|
#16 |
|
Registered User
Join Date: Jan 2017
Posts: 9
|
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
01-04-2017, 02:58 AM
|
#17 |
|
Registered User
Join Date: Oct 2016
Posts: 30
|
Thank you so much...
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
01-04-2017, 03:09 AM
|
#18 |
|
Registered User
Join Date: Nov 2016
Posts: 72
|
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
01-04-2017, 03:58 AM
|
#19 |
|
Registered User
Join Date: Dec 2016
Posts: 112
|
Robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
01-04-2017, 05:34 AM
|
#20 |
|
Registered User
Join Date: Jun 2016
Posts: 218
|
Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-24-2017, 03:32 AM
|
#21 |
|
Registered User
Join Date: Sep 2016
Posts: 226
|
Robots.txt is used to block the cawler from crawling the private areas.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-26-2017, 11:20 PM
|
#22 |
|
Registered User
Join Date: Mar 2017
Posts: 25
|
Robots.txt is provide instructions to the crawlers about the caching and indexing of a website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-26-2017, 11:31 PM
|
#23 |
|
Registered User
Join Date: Apr 2016
Location: pune
Posts: 583
|
Robot.txt is a text file that specify how to inform the web robot about which areas of the website should not be processed or scanned.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-27-2017, 01:28 AM
|
#24 |
|
Registered User
Join Date: Dec 2016
Posts: 530
|
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-27-2017, 10:10 PM
|
#25 |
|
Registered User
Join Date: Aug 2016
Posts: 387
|
Robots.txt is a standard used by websites to communicate with web crawlers and other web robots.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-28-2017, 02:35 AM
|
#26 |
|
Registered User
Join Date: Oct 2016
Posts: 335
|
The robots exclusion protocol is a standard used by the websites to communicate with web crawlers and other web robots
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-28-2017, 02:56 AM
|
#27 |
|
Registered User
Join Date: Mar 2017
Location: Bangalore
Posts: 14
|
Robots.txt is one of the text fil, Which is used to instruct robots, how to crawl and index pages on their website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | o To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-28-2017, 03:30 AM
|
#28 |
|
Registered User
Join Date: Nov 2016
Posts: 330
|
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-28-2017, 09:14 PM
|
#29 |
|
Registered User
Join Date: Mar 2017
Location: chennai
Posts: 71
|
an text file that indicates pages not to be crawled
|
|
|
|
|
03-28-2017, 10:55 PM
|
#30 |
|
Registered User
Join Date: Nov 2016
Posts: 412
|
A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want to be accessed by search engine crawlers. The file uses the Robots Exclusion Standard, which is a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers (such as mobile crawlers vs desktop crawlers).
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Robots.txt | jaysh4922 | Search Engine Optimization | 13 | 07-28-2016 05:10 AM |
| Crawlers, spider & robots.txt? | jackthomas087 | Search Engine Optimization | 0 | 04-22-2014 03:29 AM |
| Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
All times are GMT -7. The time now is 08:05 AM.