02-16-2016, 11:59 PM
|
#1 |
|
Registered User
Join Date: Dec 2015
Posts: 157
|
What is the use of Robots.txt?
Hello friends,
What is the use of Robots.txt? |
|
|
|
02-17-2016, 12:47 AM
|
#2 |
|
Registered User
Join Date: Jan 2016
Location: India
Posts: 1,258
|
Website owners use the Robots.txt file to give insturctions about their site to web robots this is called The Robots Exclusions Protocol.
|
|
|
|
|
02-17-2016, 12:56 AM
|
#3 |
|
Registered User
Join Date: Jan 2016
Location: India
Posts: 271
|
Hi, robot.txt file is used to give instruction to the search engine about, which page to be crawled are which not.
Upcoming events in Chandigarh, Upcoming events in Mohali, Upcoming events in Panchkula, Upcoming events in Tricity
__________________
http://apnitricity.com/ Last edited by aditya12; 02-17-2016 at 12:57 AM.. Reason: spell mistake |
|
|
|
|
02-17-2016, 02:54 AM
|
#4 |
|
Registered User
Join Date: Jan 2016
Location: Hyderabad
Posts: 4,286
|
In Simple words Robot.txt files gives instructions of your website to search engines.
|
|
|
|
|
02-17-2016, 03:24 AM
|
#5 | |
|
Registered User
Join Date: Feb 2016
Location: India
Posts: 21
|
Quote:
|
|
|
|
|
|
02-17-2016, 09:52 PM
|
#6 |
|
Registered User
Join Date: Feb 2016
Posts: 3
|
Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
or Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. Last edited by Hardeep_Singh; 02-17-2016 at 09:58 PM.. |
|
|
|
|
02-17-2016, 11:26 PM
|
#7 | |
|
Registered User
Join Date: Feb 2016
Posts: 35
|
Quote:
|
|
|
|
|
|
02-17-2016, 11:31 PM
|
#8 |
|
Registered User
Join Date: Feb 2016
Location: Egypt
Posts: 15
|
It is a text file. It instruct the crawler which page has crawl or not
|
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Crawlers, spider & robots.txt? | jackthomas087 | Search Engine Optimization | 0 | 04-22-2014 03:29 AM |
| Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
All times are GMT -7. The time now is 06:59 PM.