Is mine okay? I put it up about a week ago and googlebot hasnt visited my site since (and it used to be on there all the time)
By the way, I think I'll install vbSEO today.
vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.
This is a discussion on robots.txt within the General Discussion forums, part of the vBulletin SEO Discussion category; Is mine okay? I put it up about a week ago and googlebot hasnt visited my site since (and it ...
Is mine okay? I put it up about a week ago and googlebot hasnt visited my site since (and it used to be on there all the time)
By the way, I think I'll install vbSEO today.
Do you have a Google Webmaster account? They'll validate your robots.txt file for you.
Google Webmaster Central
If not, you can try one of the other free validators online:
Validators | Robots.txt Validator
:(:(
It says the robots.txt is blocking access to the homepage.
Here is how I have it laid out
User-Agent: *
Disallow: /
User-agent: *
Disallow: /attachment.php
Disallow: /newattachment.php
Disallow: /avatar.php
Disallow: /editpost.php
Disallow: /login.php
Disallow: /member.php
Disallow: /member2.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /private2.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendtofriend.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /sendmessage.php
Disallow: /register.php
Disallow: /subscription.php
Disallow: /admincp/
Disallow: /archive/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /subscriptions/
Because you have this code, which disallows all search engines from accessing any of your pages:
Remove those two lines.Code:User-Agent: * Disallow: /
hahaha yeah as soon as I posted that I realized what I had done. Here's the new one (I added a few more things I thought might be unncessary for search engines including the archive)
User-agent:
*Disallow: /attachment.php
Disallow: /newattachment.php
Disallow: /avatar.php
Disallow: /editpost.php
Disallow: /login.php
Disallow: /calendar.php
Disallow: /member.php
Disallow: /member2.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /private2.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendtofriend.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /sendmessage.php
Disallow: /register.php
Disallow: /subscription.php
Disallow: /admincp/
Disallow: /archive/
Disallow: /clientscript/
Disallow: /archive/index.php/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /memberslift.php
Disallow: /subscriptions/
Disallow: /faq.php