robots.txt

Hiyas

Just looking for opinions on robots.txt. Should I disallow items like newthread.php, newreply.php, profile.php etc., or should I just have no robots.txt and let the bots go nuts, or something in between. What do you people do in this regard?

Cheers

Z

Personally I disallow them, uses up resources and bandwidth and it doesn't hurt to disallow them anyway.

I just didn't like seeing them trying to access those pages in Whos Online.

Some search engines seem to ignore nofollow as well.

bit of a follow on

Now I have VBSEO should i leave these in?

User-agent: *
Disallow: /printthread
Disallow: /send message
Disallow: /register
Disallow: /login
Disallow: /new reply
Disallow: /subscription
Disallow: /private
Disallow: /misc
Disallow: /show post
Disallow: /report
Disallow: /new thread
Disallow: /calendar
Disallow: /poll
Disallow: /links
Disallow: /arcade
Disallow: /member list
Disallow: /members
Disallow: /search
Disallow: /FAQ
Disallow: /online
Disallow: /vb shout

And is there any on the list that should not be (or maybe I have missed).

For you know I rather have only 3 good pages indexed then 100 useless pages + the 3 good ones.

I don't think you have the format of the robots.txt correct.

You have spaces in your disallow lines and your urls aren't supposed to have that.

For an example look at mine I have:
http://www.alanwake.net/robots.txt

It was done to be easier on the eyes when I pasted it from notepad. It's not like that in the the doc. But well spotted . Anyway any feedback on including them all, some or more.

You can check if you have a valid robots.txt in this link

http://tool.motoricerca.info/robots-...checkreferer=1

I see that most of you have a robots.txt file.

How important is it to have it? I have never had one. I would assume you just create it and stick it in the root.

I did quite a bit of reseach on robotx.txt yesterday, and from most of what I read, it appears important to have the file even if it is one telling robots they can follow anything. Otherwise they tend to get directed to 404 files or something like that.

I have since added on to my site that disallows nothing and it even shows up in my awstats how many times a spider successfully hit the robots.txt file.

maybe this will help my indexing. *crosses fingers*