|
|
08-28-2023, 04:53 AM
|
#1 |
|
Registered User
Join Date: Apr 2016
Location: South Korea
Posts: 721
|
Robots.txt is a text file created by webmasters to tell crawlers how to crawl pages on their websites. The robots.txt file is part of the Robot Exclusion Protocol (REP), a set of web standards that govern how robots crawl the web, access content, index it, and make it available to users.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | |
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Hybrid Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| How will you create a file in Perl? | Pavan Techn | Cgi-perl | 0 | 06-01-2018 11:09 PM |
| How to Create the Robots txt file. | Nexevo Technolo | 15 | 03-14-2018 09:23 PM | |
| How to create the Sitemap file manually? | sahithya | Search Engine Optimization | 4 | 11-26-2014 06:23 AM |
| how can create the Robot.txt file | eagleeye dental | Search Engine Optimization | 0 | 09-01-2014 09:40 PM |
| How to create user friendly URL in FTP file? | sahithya | Search Engine Optimization | 2 | 08-08-2014 03:14 AM |
All times are GMT -7. The time now is 01:18 PM.