|
Solution
I think best answer for your question is Wikipedia https:// en.wikipedia.org/ wiki/ Robots_exclusion_standard , please check it and let us know, if you required any further information.
__________________
█ ktchost.com - Reliable Web Hosting Provider
█ E-commerce Ready Hosting | Pure SSD Cloud/VPS | Offshore Dedicated Server
█ 24/7 Reliable Tech Support | Monitoring Service | Free Website Builder
|