[ramphal]

Yes all answers are some but not exact :-
Exactly robots.txt is a text file which has two parts
user agent and disallow
In dis allow, we place that file we don't want to crawl and index in search engine.