![]() |
#3 |
Registered User
Join Date: Nov 2013
Posts: 199
|
Robots.txt is a file, which user needs to add a webpage that do not want to crawl by the spider or crawler...
eg- User Agent: * Disallow: //officeadmin Type in a notepad file and add it to website root folder ..
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
![]() |
![]() |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
RSS Feed | rajivwebads | Search Engine Optimization | 12 | 08-31-2012 07:15 AM |
Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
Don�t exceed maximum file size for robots.txt file | ClaudiaSchayffe | Search Engine Optimization | 3 | 04-02-2012 04:28 AM |