03-05-2013, 04:18 AM
|
#20 |
|
Registered User
Join Date: Mar 2012
Posts: 74
|
Robots file is a simple text file which is uploaded on root directory this is the file which is crawled by the crawler very first. So in this file we write the code which file we want to block from the crawler and other one we want to crawl.
Main use of this file is just to block the admin section which generally we wish do not crawl by the search engine. Hope you understood. |
|
|
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Threaded Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Anchor Text Vs Google Algorithm | Jimerson12 | Search Engine Optimization | 1 | 06-27-2025 05:18 AM |
| telepresence robot | hemnchy | General Discussion | 1 | 01-02-2014 04:35 AM |
| Robot text. | terrijhon | Search Engine Optimization | 18 | 06-08-2013 03:35 PM |
| Hidden text and links | farm gate | 0 | 11-11-2012 08:08 PM | |
| Text Areas | ncjbhd | HTML / DHTML | 1 | 11-15-2011 04:09 AM |
All times are GMT -7. The time now is 12:11 AM.