Please provide a standard robots.txt

There are lots of threads here asking for robots.txt configuration and confusion still exists on which to allow/disallow.

Please provide a perfect robots.txt file in the vBSEO package.

Thank you

This is impossible to provide, simply because not everyone will be using the exact same rules.


IMO, the best robots file is one that is extremely vague.

Thus, I re-write EVERY page i want indexed, and disallow *.php in my robots .txt file.

full example is provided in my guide (link in sig)