03-02-2017, 11:25 PM
|
#16 |
|
Registered User
Join Date: Feb 2017
Posts: 69
|
A file tell web bots what pages and directories to index or not index.
|
|
|
|
03-02-2017, 11:38 PM
|
#17 |
|
Registered User
Join Date: Jan 2017
Posts: 674
|
robots.txt file assist the search engine in crawling.
|
|
|
|
|
03-02-2017, 11:57 PM
|
#18 |
|
Registered User
Join Date: Jan 2017
Location: hyderabad
Posts: 78
|
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
|
|
|
|
|
03-03-2017, 12:22 AM
|
#19 |
|
Registered User
Join Date: Dec 2016
Posts: 377
|
Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.
__________________
erp solutions company | Erp management system | School system software | erp management software  
|
|
|
|
|
03-14-2017, 11:39 PM
|
#20 |
|
Registered User
Join Date: Mar 2017
Posts: 61
|
Allow and disallow the google bot to crawl the website pages.
|
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Check the website robots file | china-jewels | Bing | 4 | 10-11-2015 08:14 AM |
| Uses of robots? | jamesranatte | General Discussion | 2 | 12-11-2014 08:32 AM |
| Robots exclusion protocols? | jamesranatte | General Discussion | 7 | 12-26-2013 12:01 AM |
| what is difference between spider, crawler and robots? | noahwilson | Search Engine Optimization | 0 | 05-30-2013 03:04 AM |
| What is Robots meta tag? | samlko | Search Engine Optimization | 2 | 04-06-2012 12:10 AM |
All times are GMT -7. The time now is 06:10 AM.