Understanding: robots.txt

Trying to figure out why this is happening.

MSNBot Spider Unknown Location
/robots.txt
51 Minutes Ago MSNBot Spider Unknown Location
/robots.txt

2 Hours Ago Google Spider Unknown Location
/robots.txt

When I navigate to these pages I get my front page. Should there not be a vBSEO created file here? What's up with this? No biggie, Just curious.

Do you actually have a 'robots.txt' file on your server (in domain root) ?

EDIT:

Nevermind. Checked the file header of that page and it's a '404 Not Found'.
And no, vBSEO won't auto-generate the file.

Any suggestions on what I should do? A link to setting up robots.txt? Not exactly sure what to do here.

Set up redirect?

Theres no need for a robots.txt if you don't have anything to block from the search engine robots.

But still, if you want to prevent those errors from showing up, you can upload a BLANK text file with the name of 'robots.txt' to your domain root. So that it's accessible via domain.com/robots.txt

sounds like you have an outdated 404 script.

be sure

define('SKIP_SESSIONCREATE', 1);

is at the top of the 404.php script. this will stop the bad pages from showing in whos online.

Thanks for the help you guys. I have not seen the error since. Took a while for me to understand a few things but everything seems to be going well.

vBSEO Users Rock.