What is robots.txt? - Site Owners Forums

ankit009 · 04-13-2014, 10:36 PM

This is a page that gives search engines information about the pages a company wants indexed or crawled. You can find this page by doing to YOURDOMAIN/robots.txt.

rajivdevkk · 04-13-2014, 10:51 PM

Robots.txt file is very helpful to disallow the pages which you don't want to index in the search engine.

picnic spots near delhi | water park in delhi

jetwings · 04-13-2014, 10:55 PM

Its a file to prevent the crawlers to not to crawl the restricted pages in your site.

web3.0company · 04-13-2014, 11:08 PM

Robot.txt file is use for hide web page from search engine, if you don't want to index any page of website you can use this robot.txt and disallow that page....

wowcab · 04-13-2014, 11:12 PM

with help to robots.txt Google find website internal page easily

anuverma1 · 04-13-2014, 11:12 PM

its restrict the robots not crawler your all page only index page they can craw.

ankit009 · 04-14-2014, 03:07 AM

thanx guys, for giving that nice suggestion or answer regarding robot.txt.

meriwetherlewis · 04-14-2014, 03:49 AM

The Robot Different Frequent, additionally known as the Robots Exclusion Protocol or robots.txt method, is a personalized to training working net elephants as well as added net spiders about acquiring all or portion of a site that's otherwise readily readable. Bots are usually employed by yahoo to truly save and classify websites, or by web masters to estimate source-code. The normal differs from, but could be used together with, Site Maps, a robot intro common for web sites.

creativesquad · 04-14-2014, 04:24 AM

The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.

kumar29 · 04-14-2014, 04:36 AM

Robots.txt file is a system file. If you still want to common things it by your own, create a text file using notepad or any other text editor and name it robots.txt. Then upload it to the root folder of your site by means File Manager or FTP.

seo_butterflyth · 04-14-2014, 05:19 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

Shane Bentick · 05-14-2017, 10:56 PM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

04-13-2014, 10:36 PM	#1
ankit009 Registered User Join Date: Apr 2014 Posts: 118	What is robots.txt? This is a page that gives search engines information about the pages a company wants indexed or crawled. You can find this page by doing to YOURDOMAIN/robots.txt. __________________ How to secure the Excel sheet

04-13-2014, 10:51 PM	#2
rajivdevkk Registered User Join Date: Sep 2013 Location: Delhi Posts: 344	Robots.txt file is very helpful to disallow the pages which you don't want to index in the search engine. picnic spots near delhi \| water park in delhi Last edited by rajivdevkk; 04-13-2014 at 10:53 PM..

04-13-2014, 10:55 PM	#3
jetwings Registered User Join Date: Apr 2014 Posts: 17	Its a file to prevent the crawlers to not to crawl the restricted pages in your site. __________________ Software Training Bangalore \| Project Work for aeronautical engineering \| On the Job Training

04-13-2014, 11:12 PM	#5
wowcab Registered User Join Date: Apr 2014 Posts: 33	with help to robots.txt Google find website internal page easily __________________ cabs delhi

04-13-2014, 11:12 PM	#6
anuverma1 Registered User Join Date: Feb 2014 Posts: 541	its restrict the robots not crawler your all page only index page they can craw. __________________ Auditor in India \| Stock Audit in India \| Tax Advisor in India \| Chartered Accountant in New Delhi

04-13-2014, 11:08 PM	#4
web3.0company Registered User Join Date: Dec 2013 Posts: 104	Robot.txt file is use for hide web page from search engine, if you don't want to index any page of website you can use this robot.txt and disallow that page....

04-14-2014, 03:07 AM	#7
ankit009 Registered User Join Date: Apr 2014 Posts: 118	thanx guys, for giving that nice suggestion or answer regarding robot.txt. __________________ How to secure the Excel sheet

04-14-2014, 03:49 AM	#8
meriwetherlewis Registered User Join Date: Apr 2014 Posts: 5	The Robot Different Frequent, additionally known as the Robots Exclusion Protocol or robots.txt method, is a personalized to training working net elephants as well as added net spiders about acquiring all or portion of a site that's otherwise readily readable. Bots are usually employed by yahoo to truly save and classify websites, or by web masters to estimate source-code. The normal differs from, but could be used together with, Site Maps, a robot intro common for web sites. __________________ medical doctor's office new york

04-14-2014, 04:24 AM	#9
creativesquad Registered User Join Date: Apr 2014 Posts: 110	The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages. __________________ How to create menus in wordpress? \| image editing services in delhi

04-14-2014, 04:36 AM	#10
kumar29 Registered User Join Date: Apr 2014 Posts: 18	Robots.txt file is a system file. If you still want to common things it by your own, create a text file using notepad or any other text editor and name it robots.txt. Then upload it to the root folder of your site by means File Manager or FTP.

04-14-2014, 05:19 AM	#11
seo_butterflyth Registered User Join Date: Mar 2014 Posts: 357	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. __________________ WebHostChampion.com - 100% Working and Valid hostgator Web Hosting Coupons

05-14-2017, 10:56 PM	#12
Shane Bentick Registered User Join Date: Apr 2017 Posts: 355	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. __________________ gmc yukon car cover \| yukon xl car covers \| lincoln town car car cover\| jeep grand cherokee car covers

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
robots.txt	wheelbarrow001	Google	2	07-27-2012 04:10 AM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM