03-07-2025, 01:04 AM
|
#1 |
|
Registered User
Join Date: Jan 2012
Posts: 194
|
Benefits of Robots.txt File
What is the Benefits of Robots.txt File in Website ?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
03-21-2025, 02:34 AM
|
#2 |
|
Registered User
Join Date: Dec 2012
Location: Los Angeles
Posts: 208
|
By using the Robot.txt file, you can provide the command to the crawler regarding what webpage is to be crawled. For example, when I was working on a pregnancy website, I used a follow-link in the Robot.txt file on the webpage I wanted to crawl and the no-follow link that I didn't want to be crawled.
Last edited by markstyne; 03-21-2025 at 09:19 PM.. |
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | |
| Display Modes | Rate This Thread |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Is it necessary to use Robots.txt file? | latestnewsheadl | Search Engine Optimization | 4 | 07-23-2022 06:31 AM |
| what is robots.txt file: | primeforex | General Discussion | 0 | 05-07-2018 03:31 AM |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
All times are GMT -7. The time now is 09:19 AM.