07-11-2017, 04:41 AM
|
#11 |
|
Registered User
Join Date: Jun 2017
Location: India
Posts: 15
|
Robots.txt is the root file of the website, where you mention which pages of the website are allowed to be crawled and which pages are blocked.
If your website is not having Robots.txt file, than by default all pages of the website are allowed to be crawled. Robots.txt tag is the substitute of Robots.txt file. It also does the same work, the only difference between Robots.txt file and tag is, tag need to be mentioned in all the pages, whereas file is uploaded only once in the root file. |
|
|
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Threaded Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Robots.txt or Meta Robots.txt??? | santoshkumar | Search Engine Optimization | 10 | 02-14-2017 04:03 AM |
| Robots.txt | jaysh4922 | Search Engine Optimization | 13 | 07-28-2016 05:10 AM |
| Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
All times are GMT -7. The time now is 09:40 PM.