|
|
12-23-2014, 05:43 AM
|
#1 |
|
Registered User
Join Date: Oct 2013
Posts: 37
 |
Robot.txt?
What is Robot.txt? Why it is used?
|
|
|
|
12-23-2014, 01:13 PM
|
#2 |
|
Registered User
Join Date: Dec 2014
Location: Pakistan
Posts: 16
|
robots.txt file used as directives to search engine crawlers. you can forbid crawlers to crawl specific pages of your site
__________________
Downloadfs|Software free download all |
|
|
|
|
12-24-2014, 02:01 AM
|
#3 |
|
Registered User
Join Date: Jul 2014
Location: bangalore, india
Posts: 66
|
Robots.txt is a text (not html) file you put on your web page to tell search robots which pages you would like them not to visit.
__________________
SEO service providers in Bangalore | best seo companies in bangalore | Web Design Agency Bangalore |
|
|
|
|
12-24-2014, 06:08 AM
|
#4 |
|
Registered User
Join Date: Dec 2014
Posts: 31
|
Yes all answers are some but not exact :-
Exactly robots.txt is a text file which has two parts user agent and disallow In dis allow, we place that file we don't want to crawl and index in search engine. |
|
|
|
|
12-24-2014, 08:42 AM
|
#5 |
|
Registered User
Join Date: Dec 2014
Posts: 58
|
I think they explained you why you should keep it pn your host
__________________
Eric Leroy |
|
|
|
|
12-24-2014, 10:56 PM
|
#6 |
|
Registered User
Join Date: Sep 2013
Location: Delhi
Posts: 344
|
robots.txt is helpful to crawls the site or not by using allow or disallow.
picnic spots near delhi | New year packages delhi 2015 | camp in kanatal Last edited by rajivdevkk; 12-24-2014 at 11:00 PM.. |
|
|
|
|
12-26-2014, 10:37 PM
|
#7 |
|
Registered User
Join Date: Dec 2014
Posts: 22
|
This helps to prevent a particular page to crowl that the user don't want.
|
|
|
|
|
12-27-2014, 12:08 AM
|
#8 |
|
Registered User
Join Date: Nov 2014
Posts: 12
|
Robots.txt is an text file used by search crewl.it tells crewl which part of website to index or which not index in search result.Find More At Custom Robot.txt
|
|
|
|
|
12-27-2014, 02:51 AM
|
#9 |
|
Registered User
Join Date: Aug 2014
Posts: 98
|
It is used to allow or disallow search engine to crawl your pages..
|
|
|
|
|
12-27-2014, 03:00 AM
|
#10 |
|
Registered User
Join Date: Apr 2014
Posts: 3
|
It help to instruct crawler that which page should be cache on which is not.
__________________
http://www.escortsliverpool.co.uk/ |
|
|
|
|
12-27-2014, 03:51 AM
|
#11 |
|
Registered User
Join Date: Apr 2014
Posts: 110
|
Robot.txt file used to block crawler to crawl some content in your website.
|
|
|
|
|
01-08-2015, 11:36 PM
|
#12 |
|
Registered User
Join Date: Jan 2015
Posts: 6
|
Robots.txt is a text file, It is used to allow or disallow search engines to crawl your website or webpage
|
|
|
|
|
01-09-2015, 12:38 AM
|
#13 |
|
Registered User
Join Date: Jan 2015
Posts: 80
|
Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly.
|
|
|
|
|
01-09-2015, 01:06 AM
|
#14 |
|
Registered User
Join Date: Jan 2015
Posts: 1
|
Robots.txt is a text file which we upload on website server to instruct spider the which page or folder should be crawl and which is not.
|
|
|
|
|
01-09-2015, 01:18 AM
|
#15 |
|
Registered User
Join Date: Mar 2014
Posts: 111
|
Thanks for your question and happy to read all friends suggestion
|
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Hybrid Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| what is robot.txt? | rubysethi | Search Engine Optimization | 0 | 12-19-2014 11:27 PM |
| How to know about robot.txt ? | vanaja | Search Engine Optimization | 10 | 10-08-2014 04:39 AM |
| how can Create the Robot.txt | pankaj | Search Engine Optimization | 3 | 10-08-2014 03:43 AM |
| how can create the Robot.txt file | eagleeye dental | Search Engine Optimization | 0 | 09-01-2014 09:40 PM |
| how to use robot.txt? | dean1122 | Search Engine Optimization | 11 | 01-05-2013 02:10 AM |
All times are GMT -7. The time now is 07:13 PM.