09-21-2022, 11:57 PM
|
#30 |
|
Registered User
Join Date: Aug 2014
Location: India
Posts: 816
|
Robots.txt are the files that will tell the crawler which pages of the website to crawl and which pages of the website not to crawl.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Threaded Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
| Robots.txt | sameerseo | Search Engine Optimization | 26 | 09-28-2012 04:18 AM |
| robots.txt | wheelbarrow001 | 2 | 07-27-2012 04:10 AM | |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
| What Is Robots.txt? | samlko | Search Engine Optimization | 13 | 03-09-2012 02:37 PM |
All times are GMT -7. The time now is 01:28 PM.