robot.txt - Site Owners Forums

Shivam Tripathi · 04-30-2018, 02:04 AM

Hey Friends,

Please anyone can tell me what is main purpose of robot.txt and why we use robot.txt in SEO ?

Because till now i am struggling for this question.so,please share with me asap..

pradeep01 · 04-30-2018, 04:08 AM

The robots exclusion normal, additionally called the robots exclusion protocol or just robots.txt, may be a normal utilized by netsites to speak with net crawlers and different web robots. the quality specifies the way to inform the online automaton regarding that areas of the web site shouldn't be processed or scanned.

drugrehabpoint · 04-30-2018, 04:29 AM

Thanks for sharing

wellliving · 05-01-2018, 04:11 AM

Hi

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

https://www.welllivingshop.com/baby/

multai · 05-02-2018, 03:37 AM

It is the text file it gives the instructions to the robots about how to crawl the web page...

http://www.multaisolutions.com/ecommerce.html

RH-Calvin · 12-16-2018, 10:11 PM

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

04-30-2018, 02:04 AM	#1
Shivam Tripathi Registered User Join Date: Feb 2018 Location: Pune Posts: 64	robot.txt Hey Friends, Please anyone can tell me what is main purpose of robot.txt and why we use robot.txt in SEO ? Because till now i am struggling for this question.so,please share with me asap.. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-30-2018, 04:08 AM	#2
pradeep01 Registered User Join Date: Apr 2018 Posts: 11	The robots exclusion normal, additionally called the robots exclusion protocol or just robots.txt, may be a normal utilized by netsites to speak with net crawlers and different web robots. the quality specifies the way to inform the online automaton regarding that areas of the web site shouldn't be processed or scanned. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-30-2018, 04:29 AM	#3
drugrehabpoint Registered User Join Date: Sep 2017 Posts: 133	Thanks for sharing __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-01-2018, 04:11 AM	#4
wellliving Registered User Join Date: Dec 2015 Location: USA Posts: 743	Hi Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. https://www.welllivingshop.com/baby/ __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-16-2018, 10:11 PM	#6
RH-Calvin Registered User Join Date: May 2015 Location: Forum Posts: 2,517	Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling. __________________ █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Servers with Free IPMI Setup

05-02-2018, 03:37 AM	#5
multai Registered User Join Date: Apr 2018 Posts: 59	It is the text file it gives the instructions to the robots about how to crawl the web page... http://www.multaisolutions.com/ecommerce.html

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What is Robot.txt?	kumar1_16	General Discussion	11	03-29-2017 03:15 AM
Robot.txt?	siva SEO	Search Engine Optimization	17	01-09-2015 07:02 PM
how can Create the Robot.txt	pankaj	Search Engine Optimization	3	10-08-2014 03:43 AM
how can create the Robot.txt file	eagleeye dental	Search Engine Optimization	0	09-01-2014 09:40 PM