What is the purpose of robots.txt file? - Site Owners Forums

sahithya · 05-06-2014, 11:02 PM

What is the purpose of robots.txt file?

Sarahbriggs · 05-06-2014, 11:24 PM

Robots.txt is a text file which you put on your website to tell crawler which pages you would like not to crawl.

You need to create a file and you have to enter that URL which you don't want to crawl. Have a look below.

User Agent: *
Disallow: //admin

Disallow: ///t141187-

Disallow: //file.html

Disallow: //purpose

vickyjohn · 05-06-2014, 11:32 PM

Robots.txt is a file, which user needs to add a webpage that do not want to crawl by the spider or crawler...
eg-
User Agent: *
Disallow: //officeadmin
Type in a notepad file and add it to website root folder ..

employmentpakis · 05-06-2014, 11:48 PM

Robot.txt file is a text file that give instructions to search engine robots which pages you would like them not to visit.

spyindia01 · 05-07-2014, 12:43 AM

A simple text file that stops Google (and other search engines that recognize the file and its commands) from crawling the site, selected pages in the site, or selected file types in the site.

ChristinaCa · 05-07-2014, 02:38 AM

Robot.txt file is a file which contains the webpages which one don't want to be crawled by the crawler.

LydiaAaron · 11-27-2017, 04:57 AM

Robots.txt purpose not index for the any particular web pages in our web sites.

Fozia khadim · 11-27-2017, 05:07 AM

Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robot.txt. Exclusion Protocol tells the robot that it should not visit any pages on the site.

infiniumtech · 11-27-2017, 08:00 AM

Robots.txt is a text file webmasters create to instruct web robots. how to crawl pages on their website.

AmandaCherry · 11-27-2017, 08:16 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

ROYCPO · 02-13-2018, 08:30 AM

Robots.txt is the common name of a text file that is uploaded to a Web site's root directory and linked in the HTML code of the Web site. The robots.txt file is used to provide instructions to the Web site to Web robots and spiders. Web authors can use robots.txt

harshithaasin · 02-13-2018, 10:25 PM

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl.

deepakrajput · 02-24-2018, 08:51 AM

Robots.txt file is set of instructions help to index the website through search engines.

Kentowin · 02-25-2018, 09:53 PM

robots.txt file is the file which instruct search engine which part of your webpage you don't want to get crawl by the search engine spider.

harshithaasin · 02-25-2018, 10:56 PM

Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.

05-06-2014, 11:02 PM	#1
sahithya Registered User Join Date: Feb 2013 Location: Bangalore Posts: 1,192	What is the purpose of robots.txt file? What is the purpose of robots.txt file? __________________ Software Development Company in Bangalore \| FinTech Software Development Company in India \| Gen AI Consulting Company in India \| Microsoft Power Platform Consulting Services \| InsurTech Software Development Company in India

05-06-2014, 11:24 PM	#2
Sarahbriggs Registered User Join Date: May 2014 Posts: 12	Robots.txt is a text file which you put on your website to tell crawler which pages you would like not to crawl. You need to create a file and you have to enter that URL which you don't want to crawl. Have a look below. User Agent: * Disallow: //admin Disallow: ///t141187- Disallow: //file.html Disallow: //purpose __________________ Supreme Suppliers & Manufacturers of Plastic Furniture India - Find modern designer plastic moulded good quality Premium chairs/ Furnitures,dining Chair, tables, furniture. Armless Cushioned Chair \| Fusion Furniture \| With Arm Cushioned Chair \| Plastic Moulded Chair Exporters

05-06-2014, 11:32 PM	#3
vickyjohn Registered User Join Date: Nov 2013 Posts: 199	Robots.txt is a file, which user needs to add a webpage that do not want to crawl by the spider or crawler... eg- User Agent: * Disallow: //officeadmin Type in a notepad file and add it to website root folder .. __________________ Lol riot points steam wallet codes Free games download

05-06-2014, 11:48 PM	#4
employmentpakis Registered User Join Date: Feb 2014 Posts: 51	Robot.txt file is a text file that give instructions to search engine robots which pages you would like them not to visit. __________________ www.ditrc.com, Manpower Agencies In Pakistan

05-07-2014, 12:43 AM	#5
spyindia01 Registered User Join Date: Jul 2013 Posts: 49	A simple text file that stops Google (and other search engines that recognize the file and its commands) from crawling the site, selected pages in the site, or selected file types in the site. __________________ Top Secret Gadgets \| Secret Softwares

05-07-2014, 02:38 AM	#6
ChristinaCa Registered User Join Date: Mar 2014 Posts: 210	Robot.txt file is a file which contains the webpages which one don't want to be crawled by the crawler. __________________ Kids Birthday Party \| Kids Birthday Parties

11-27-2017, 04:57 AM	#7
LydiaAaron Registered User Join Date: Sep 2017 Posts: 113	Robots.txt purpose not index for the any particular web pages in our web sites. __________________ Clerkenwell Offices \| Offices in London Shops in Old Street

11-27-2017, 05:07 AM	#8
Fozia khadim Registered User Join Date: Nov 2017 Posts: 4	Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robot.txt. Exclusion Protocol tells the robot that it should not visit any pages on the site. __________________ Best Destination Wedding Planner in Udaipur Royal Wedding planner in Udaipur Top Wedding Planner in Udaipur Destination wedding planners in Udaipur Top Wedding Planners in Jodhpur

11-27-2017, 08:00 AM	#9
infiniumtech Registered User Join Date: Nov 2017 Location: Surat, Gujarat, India Posts: 36	Robots.txt is a text file webmasters create to instruct web robots. how to crawl pages on their website.

11-27-2017, 08:16 PM	#10
AmandaCherry Registered User Join Date: Nov 2017 Posts: 342	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ Upload 100 Products on Fiverr \| Video Captions on Fiverr \| Data Entry on Fiverr \| Virtual Assistant on Fiverr

02-13-2018, 08:30 AM	#11
ROYCPO Registered User Join Date: Dec 2015 Location: Bangladesh Posts: 267	Robots.txt is the common name of a text file that is uploaded to a Web site's root directory and linked in the HTML code of the Web site. The robots.txt file is used to provide instructions to the Web site to Web robots and spiders. Web authors can use robots.txt __________________ Clipping Path Outsource, digital graphics studio providing image editing services all over the world. Expert in clipping path Service.

02-13-2018, 10:25 PM	#12
harshithaasin Registered User Join Date: Oct 2017 Posts: 306	The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. __________________ Electronic Medical Record \| Electronic prescription \| EMR Software \| Healthcare IT \| Healthcare Informatics \| Electronic Health Records

02-24-2018, 08:51 AM	#13
deepakrajput Registered User Join Date: Feb 2015 Location: CA Posts: 733	Robots.txt file is set of instructions help to index the website through search engines. __________________ Business \| $99 SEO \| Shopping \| Health \| Business Services \| Small Business SEO \| Digital Marketing

02-25-2018, 09:53 PM	#14
Kentowin Registered User Join Date: Feb 2018 Location: Hyderabad Posts: 47	robots.txt file is the file which instruct search engine which part of your webpage you don't want to get crawl by the search engine spider. __________________ Best Institutes in india \| Best Tutors \| Training Institute \| Coaching classes \| Home Tuition \| Group Tuition \| Online Tuition\|Institute Management System

02-25-2018, 10:56 PM	#15
harshithaasin Registered User Join Date: Oct 2017 Posts: 306	Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. __________________ Electronic Medical Record \| Electronic prescription \| EMR Software \| Healthcare IT \| Healthcare Informatics \| Electronic Health Records

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
RSS Feed	rajivwebads	Search Engine Optimization	12	08-31-2012 07:15 AM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM
Don�t exceed maximum file size for robots.txt file	ClaudiaSchayffe	Search Engine Optimization	3	04-02-2012 04:28 AM