What is the difference between Robots.txt and Spider..? - Site Owners Forums

AshokDixit89 · 07-14-2016, 01:03 AM

What is the difference between Robots.txt and Spider..?

congtacvienotc · 07-14-2016, 01:08 AM

Sorry. I don't know

rekhaweb · 07-14-2016, 01:32 AM

But on the other side on robots.txt we commands to instruct crawlers not to visit those webpages or part of the website which we don't want it to index for crawling..

rahulrockers146 · 07-14-2016, 09:51 AM

robots.txt we commands to instruct crawlers not to visit those webpages or part of the website which we don't want it to index for crawling.

codetoils · 07-14-2016, 02:29 AM

Robots.txt is used for specifying pages which should not be crawled by the google asnd we don't want to pass the link authority to them.

sonamsoni · 07-14-2016, 06:38 AM

Difference between robot txt and spider are as follows
robot txt is use for the specification of the webpage
spider is use to crawl the webpages.

gracielahuff · 07-14-2016, 10:09 AM

A spider is a program run by a search engine to build a summary of a website’s content (content index). Spiders create a text-based summary of content and an address (URL) for each webpage.
robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

RH-Calvin · 07-15-2016, 12:43 AM

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Spider is the search engine program that is responsible to crawl through webpage source to provide information to search engines.

SaraSanjay · 07-15-2016, 03:16 AM

Spider is used to crawl the website
Robots is used to tell which page should be crawled and which should not in the website

07-14-2016, 01:03 AM	#1
AshokDixit89 Registered User Join Date: Jan 2016 Location: Mumbai, India Posts: 1,064	What is the difference between Robots.txt and Spider..? What is the difference between Robots.txt and Spider..? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-14-2016, 01:32 AM	#3
rekhaweb Registered User Join Date: Jul 2016 Posts: 28	But on the other side on robots.txt we commands to instruct crawlers not to visit those webpages or part of the website which we don't want it to index for crawling.. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-14-2016, 09:51 AM	#4
rahulrockers146 Registered User Join Date: Jun 2016 Posts: 41	robots.txt we commands to instruct crawlers not to visit those webpages or part of the website which we don't want it to index for crawling. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-14-2016, 02:29 AM	#5
codetoils Registered User Join Date: Jul 2016 Posts: 20	Robots.txt is used for specifying pages which should not be crawled by the google asnd we don't want to pass the link authority to them. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

07-14-2016, 06:38 AM	#6
sonamsoni Registered User Join Date: Jul 2016 Posts: 57	Difference between robot txt and spider are as follows robot txt is use for the specification of the webpage spider is use to crawl the webpages. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-14-2016, 01:08 AM	#2
congtacvienotc Registered User Join Date: Jul 2016 Posts: 12	Sorry. I don't know

07-14-2016, 10:09 AM	#7
gracielahuff Registered User Join Date: Jun 2016 Posts: 79	A spider is a program run by a search engine to build a summary of a website’s content (content index). Spiders create a text-based summary of content and an address (URL) for each webpage. robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-15-2016, 12:43 AM	#8
RH-Calvin Registered User Join Date: May 2015 Location: Forum Posts: 2,517	Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling. Spider is the search engine program that is responsible to crawl through webpage source to provide information to search engines. __________________ █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Servers with Free IPMI Setup

07-15-2016, 03:16 AM	#9
SaraSanjay Registered User Join Date: May 2016 Posts: 551	Spider is used to crawl the website Robots is used to tell which page should be crawled and which should not in the website __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How to build Spider pool for SEO to rank website	Edison Thomas	Search Engine Optimization	4	05-04-2018 03:52 AM
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
Difference between Spider and Crawlers ?	noahwilson	Search Engine Optimization	3	06-17-2013 07:04 PM
what is difference between spider, crawler and robots?	noahwilson	Search Engine Optimization	0	05-30-2013 03:04 AM