Testing robots.txt file...

If anyone can look at this robots.txt file for me and let me know if anything looks out of the ordinary or needs to be added... please let me know.

Many thanks!

Hello,

It looks Ok, though you never should include your admincp and modcp directories for security purposes. You should replace this line:

Code:

http://www.hometheatershack.com/forums/sitemap_index.xml.gz

with:

Code:

Sitemap: http://www.hometheatershack.com/forums/sitemap_index.xml.gz

And this one:

Code:

# All robots will spider the domain

with:

Code:

# All robots will crawl the domain

Thanks... I uploaded a new file with the admincp and modcp locations removed.

I will fix the other links as well.

Thanks!

You can use effectively for blocking bad bots visiting your site. They steal content and cot your bantwidth. See my robots.txt file here http://www.webmastersitesi.com/robots.txt there are lot of bad bots disallowed at the bottom.
See the samples.
User-agent: URL Control
Disallow: /

User-agent: Zeus Link Scout
Disallow: /

User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /

User-agent: Webster Pro
Disallow: /

User-agent: EroCrawler
Disallow: /

How did you manage to come up with all of those?

They are well known harmful bots and content collectors. I use it for many years on my robots.txt. I get them from many sources on internet. Please check basisoft.com they have more professional robots.txt service.

In that case, I suppose I would need to remove:

# All robots will crawl the domain

Originally Posted by Sonnie

In that case, I suppose I would need to remove:

# All robots will crawl the domain

That is just a comment .

Originally Posted by webmastersitesi

They are well known harmful bots and content collectors. I use it for many years on my robots.txt. I get them from many sources on internet. Please check basisoft.com they have more professional robots.txt service.

Is there that much benefit from having a super custom robots.txt?