"should" I use a robots.txt file?

i don't have a robots.txt file.....

should i have one?


i want everything indexed.....

help a newbie @ SEO out!

A robots.txt file is typically used to prevent certain pages from being spidered, or to block certain spiders.

If you want *everything* indexed, then you don't have to worry about preventing certain pages from being accessed.

Start with a basic search for "robots.txt tutorial" on google, then post any questions you may have.

thanks. did that.

its just that i want everything cataloged. it is my understanding that everything will be cataloged without limit especially without a robots.txt file... just wasn't sure if i was missing something. (i did read up on it; just say that some people use one to block certain functions... want to make sure i'm not missing anything)..

thanks for the response.

Sounds like you're good to go. Basically, most spiders will check for a robots.txt file when they spider a site. If the file tells them *not* to spider a certain page, then they won't do so.

If you want every spider to spider every page, then you probably don't need to worry about a robots.txt file.

Here is what i am using:

User-agent: *
Disallow: /printthread
Disallow: /sendmessage
Disallow: /register
Disallow: /login
Disallow: /newreply
Disallow: /subscription
Disallow: /private
Disallow: /misc
Disallow: /showpost
Disallow: /report
Disallow: /newthread
Disallow: /transfer
Disallow: /calendar.php?do=add

It is just useful for the Spiders not to go poking around places they dont have access too, and will only end up receiving permission messages if they tried to go there without the robots.txt file telling them not too.
Disallow: /poll

I just put a blank robots.txt file out there because I got tired of seeing it show up as a File Not Found in my web error logs.