Site Owners Forums - Webmaster Forums

Site Owners Forums - Webmaster Forums (http://siteownersforums.com/index.php)
-   Search Engine Optimization (http://siteownersforums.com/forumdisplay.php?f=16)
-   -   Problem with robots.txt file (http://siteownersforums.com/showthread.php?t=180526)

basrtaj 11-18-2016 02:12 PM

Problem with robots.txt file
 
Hi friends !
Please i use wordpress for the first time, but i have a problem with robots.txt in webmaster tool, they show me that i have a problem in my robots.txt
Can you help me???

Ajaysharma 11-18-2016 11:45 PM

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth.

Pitradev 11-19-2016 12:01 AM

Simply saying, Robot.txt is the text file that can stop the crawling by the search engines.

henrybolt 11-19-2016 12:46 AM

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

sudeepkhana 11-19-2016 01:33 AM

Robots.txt is a text (not HTML) file you put on your site to tell search robots which pages you would like them not to visit. The robots.txt is a very simple text file that is placed in your root directory.

BryceBrady 11-21-2016 12:36 AM

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

basrtaj 11-22-2016 04:44 AM

Thank you all, but Should i add it to my website Host or no???

BryceBrady 11-29-2016 01:55 AM

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

Radhika Wadhwa 11-29-2016 02:11 AM

Simply saying, Robot.txt is the text file that can stop the crawling by the search engines.

basrtaj 12-31-2016 07:07 AM

Thanks a lot, it's really helpful!


All times are GMT -7. The time now is 05:08 AM.


Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.