Robots.txt File

[mixke]

Robots.txt file is used to provide indexing instructions to search engine crawlers. It is always placed in the top-level directory of the server and this file is always checked in the first place whenever a search engine crawler visits the website in order to index it. So, you can use robots.txt file to prevent search engines from indexing some sections of your website. You should also remember that anyone can see your robots.txt file and so, you should not use it to hide the content on your website.

In short, if you want to prevent search engines from indexing some sections of your website then you should use robots.txt file.

[Trusted Writer]

As you have said, there is no way to prevent robots.txt from being displayed but unless you definitely do not want to get others to know about your hidden or private content, you can always prevent that people reading your robots.txt actually access your content directly but you require a little bit of PHP knowledge to setup a "catchall" redirection file within that directory or add add the following line to your .htaccess file inside it :

Code: Select all

Order Allow,Deny
Deny from all

[garbin]

good......................

[mudit_bhargava]

Cna some one pls explain me wht is this robot.xt !!! i cannot understand what is going on !!
thnx

[sr106]

well, i dont know about it, what is it actually, can anyone tell me in details...

[sr106]

well, i dont know about it, what is it actually, can anyone tell me in details...