What is robots.txt? - Site Owners Forums

zitoitalahold · 12-13-2012, 10:05 PM

Robots.txt is a text (not html) files which search robots in your site which pages are not visited. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

martinsherman · 12-26-2012, 11:48 PM

The robots.txt file is a text file that tells search engine crawlers which portions of your website they should NOT index. If you don't want to restrict search engine crawlers, you should simply create an empty robots.txt file (e.g., touch robots.txt) or one that looks like this:

User-agent: *
Disallow:
Once you have created a robots.txt file, you store it in the root directory of your Web server.

Hope this helps you!!

12-13-2012, 10:05 PM	#1
zitoitalahold Registered User Join Date: Dec 2012 Posts: 36	Robots.txt is a text (not html) files which search robots in your site which pages are not visited. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. is a famous social media site and it has a dedicated customer care team to deal To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. refund issues.

12-26-2012, 11:48 PM	#2
martinsherman Registered User Join Date: Nov 2012 Location: India Posts: 33	The robots.txt file is a text file that tells search engine crawlers which portions of your website they should NOT index. If you don't want to restrict search engine crawlers, you should simply create an empty robots.txt file (e.g., touch robots.txt) or one that looks like this: User-agent: * Disallow: Once you have created a robots.txt file, you store it in the root directory of your Web server. Hope this helps you!! __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Thread Tools
Show Printable Version Email this Page
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
Robots.txt	sameerseo	Search Engine Optimization	26	09-28-2012 04:18 AM
robots.txt	wheelbarrow001	Google	2	07-27-2012 04:10 AM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM
What Is Robots.txt?	samlko	Search Engine Optimization	13	03-09-2012 02:37 PM

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)