02-25-2013, 06:24 AM
|
#15 |
|
Registered User
Join Date: Feb 2013
Location: Ahmedabad
Posts: 39
|
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.
The structure of a robots.txt is pretty simple (and barely flexible) – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows: User-agent: Disallow: “User-agent” are search engines' crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line: # All user agents are disallowed to see the /temp directory. User-agent: * Disallow: /temp/ |
|
|
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | |
| Display Modes | Rate This Thread |
Threaded Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Anchor Text Vs Google Algorithm | Jimerson12 | Search Engine Optimization | 1 | 06-27-2025 05:18 AM |
| Benefits for Text Editor | ncjbhd | HTML / DHTML | 17 | 01-25-2017 03:36 AM |
| Hidden text and links | farm gate | 0 | 11-11-2012 08:08 PM | |
| Import links and anchor text | farm gate | 0 | 11-11-2012 07:58 PM | |
| Text Areas | ncjbhd | HTML / DHTML | 1 | 11-15-2011 04:09 AM |
All times are GMT -7. The time now is 10:13 PM.