11-06-2016, 11:14 PM
|
#6 |
|
Registered User
Join Date: Nov 2016
Posts: 372
|
1. Identify which directories and files on your web server you want to block from being rawled by the crawler
2. Identify whether or not you need to specify additional instructions for a particular search engine bot beyond a generic set of crawling directives 3. Use a text editor to create the robots.txt file and directives to block content 4. Optional: Add a reference to your sitemap file (if you have one) 5. Check for errors by validating your robots.txt file 6. Upload the robots.txt file to the root directory of your site
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | |
| Display Modes | Rate This Thread |
Threaded Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| How to add the robots txt file in weebly blog | Nexevo Technolo | Search Engine Optimization | 2 | 10-24-2017 10:04 PM |
| Check the website robots file | china-jewels | Bing | 4 | 10-11-2015 08:14 AM |
| How to Create Robots file? | hookahb2c | Search Engine Optimization | 1 | 07-09-2015 11:42 PM |
| How to create the Sitemap file manually? | sahithya | Search Engine Optimization | 4 | 11-26-2014 06:23 AM |
| How to create user friendly URL in FTP file? | sahithya | Search Engine Optimization | 2 | 08-08-2014 03:14 AM |
All times are GMT -7. The time now is 01:14 AM.