Site Owners Forums - Webmaster Forums

Site Owners Forums - Webmaster Forums (http://siteownersforums.com/index.php)

- Search Engine Optimization (http://siteownersforums.com/forumdisplay.php?f=16)

- - robots meta tag (http://siteownersforums.com/showthread.php?t=157831)

If robots meta tag is not included in a website page the effect is

If you want to restrict search engines from crawling some important pages of your website then you need robots.txt. or otherwise you no need to add robots.txt file.

Quote:

Originally Posted by zinavoco (Post 495748)

If robots meta tag is not included in a website page the effect is

Nothing will happen, your page will be visited by robots and spiders and they will index your page as normal.

If you do ad robots meta tag with noindex then your page wont be indexed.

Quote:

Originally Posted by cpsi (Post 495761)

Robot Visit your Website And Collect your Information. If you don't include robot meta tag then google not crawl your website.

Are you sure about that? You use the robots meta tag to stop Goolge from indexing a particular page and to set nofollow link atributes to all links on that page.

You can also use the robots meta tag to only allow certain spiders like Google bot to index your page.

If you dont ad the robots metaa tag then you are basically telling Google to go ahead and index my page.

Quote:

Originally Posted by manpreetkaur (Post 500101)

The robots meta tag lets you utilize a granular, page-specific approach to controlling how an individual page should be indexed and served to users in search results. Place the robots meta tag in the section of a given page, like this:

(…)

(…)

The robots meta tag in the above example instructs all search engine not to show the page in search results. The value of the name attribute (robots) specifies that the directive applies to all crawlers. To address a specific crawler, replace the robots value of the name attribute with the name of the crawler that you are addressing. Specific crawlers are also known as user-agents (a crawler uses its user-agent to request a page.) Google's standard web crawler has the user-agent name Googlebot. To prevent only Googlebot from crawling your page, update the tag as follows:

This tag now instructs Google (but no other search engines) not to show this page in its web search results. Both the name and the content attributes are non-case sensitive.

Search engines may have different crawlers for different properties or purposes. See the complete list of Google's crawlers. For example, to show a page in Google's web search results, but not in Google News, use the following meta tag:

If you need to specify multiple crawlers individually, it's okay to use multiple robots meta tags:

If competing directives are encountered by our crawlers we will use the most restrictive directive we find.

Best answer of the whole lot.

meta robots is useful fo crawl and discrawls the site pages.

day picnic delhi ncr | picnic spots in gurgaon

The robots.txt is a text file webmasters create to instruct robots how to crawl and index pages on their website.

Thanks for sharing informative tips about meta tags, looking forward for more information in this thread

Nice Information sharing about robots. Thanks.

You can use a special HTML tag to tell robots not to index the content of a page, and/or not scan it for links to follow.