What is the difference between Robots.txt and Spider..?
 

What is the difference between Robots.txt and Spider..?

Sorry. I don't know

But on the other side on robots.txt we commands to instruct crawlers not to visit those webpages or part of the website which we don't want it to index for crawling..

Robots.txt is used for specifying pages which should not be crawled by the google asnd we don't want to pass the link authority to them.

Difference between robot txt and spider are as follows
robot txt is use for the specification of the webpage
spider is use to crawl the webpages.

robots.txt we commands to instruct crawlers not to visit those webpages or part of the website which we don't want it to index for crawling.

A spider is a program run by a search engine to build a summary of a website’s content (content index). Spiders create a text-based summary of content and an address (URL) for each webpage.
robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Spider is the search engine program that is responsible to crawl through webpage source to provide information to search engines.

Spider is used to crawl the website
Robots is used to tell which page should be crawled and which should not in the website