vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.

Results 1 to 5 of 5

robots.txt

This is a discussion on robots.txt within the General Discussion forums, part of the vBulletin SEO Discussion category; Is mine okay? I put it up about a week ago and googlebot hasnt visited my site since (and it ...

  1. #1
    Member
    Real Name
    Mark Prater
    Join Date
    May 2007
    Posts
    36

    robots.txt

    Is mine okay? I put it up about a week ago and googlebot hasnt visited my site since (and it used to be on there all the time)

    By the way, I think I'll install vbSEO today.

  2. #2
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,847
    Blog Entries
    9
    Do you have a Google Webmaster account? They'll validate your robots.txt file for you.
    Google Webmaster Central

    If not, you can try one of the other free validators online:
    Validators | Robots.txt Validator

  3. #3
    Member
    Real Name
    Mark Prater
    Join Date
    May 2007
    Posts
    36
    :(:(

    It says the robots.txt is blocking access to the homepage.

    Here is how I have it laid out

    User-Agent: *
    Disallow: /

    User-agent: *
    Disallow: /attachment.php
    Disallow: /newattachment.php
    Disallow: /avatar.php
    Disallow: /editpost.php
    Disallow: /login.php
    Disallow: /member.php
    Disallow: /member2.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /poll.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /private2.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /search.php
    Disallow: /sendtofriend.php
    Disallow: /showgroups.php
    Disallow: /showpost.php
    Disallow: /threadrate.php
    Disallow: /usercp.php
    Disallow: /usernote.php
    Disallow: /sendmessage.php
    Disallow: /register.php
    Disallow: /subscription.php
    Disallow: /admincp/
    Disallow: /archive/
    Disallow: /clientscript/
    Disallow: /cpstyles/
    Disallow: /customavatars/
    Disallow: /images/
    Disallow: /includes/
    Disallow: /install/
    Disallow: /modcp/
    Disallow: /subscriptions/

  4. #4
    Senior Member
    Real Name
    Keith Cohen
    Join Date
    Jul 2005
    Location
    Raleigh, NC USA
    Posts
    6,147
    Because you have this code, which disallows all search engines from accessing any of your pages:

    Code:
    User-Agent: *
    Disallow: /
    
    Remove those two lines.

  5. #5
    Member
    Real Name
    Mark Prater
    Join Date
    May 2007
    Posts
    36
    hahaha yeah as soon as I posted that I realized what I had done. Here's the new one (I added a few more things I thought might be unncessary for search engines including the archive)

    User-agent:
    *Disallow: /attachment.php
    Disallow: /newattachment.php
    Disallow: /avatar.php
    Disallow: /editpost.php
    Disallow: /login.php
    Disallow: /calendar.php
    Disallow: /member.php
    Disallow: /member2.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /poll.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /private2.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /search.php
    Disallow: /sendtofriend.php
    Disallow: /showgroups.php
    Disallow: /showpost.php
    Disallow: /threadrate.php
    Disallow: /usercp.php
    Disallow: /usernote.php
    Disallow: /sendmessage.php
    Disallow: /register.php
    Disallow: /subscription.php
    Disallow: /admincp/
    Disallow: /archive/
    Disallow: /clientscript/
    Disallow: /archive/index.php/
    Disallow: /cpstyles/
    Disallow: /customavatars/
    Disallow: /images/
    Disallow: /includes/
    Disallow: /install/
    Disallow: /modcp/
    Disallow: /memberslift.php
    Disallow: /subscriptions/
    Disallow: /faq.php

Similar Threads

  1. Robots.txt now support the sitemap file
    By Mohamed Mostafa in forum General Discussion
    Replies: 35
    Last Post: 08-24-2007, 09:33 AM
  2. add sitemap to robots.txt for autodiscovery
    By GrendelKhan{TSU} in forum General Discussion
    Replies: 2
    Last Post: 04-13-2007, 02:21 PM
  3. MSNbot - robots.txt
    By curriertech in forum General Discussion
    Replies: 1
    Last Post: 10-05-2006, 08:29 AM
  4. Temp robots.txt Brand New Forum?
    By rmjvol in forum Pre-Sales Questions
    Replies: 7
    Last Post: 08-26-2006, 01:53 AM
  5. robots.txt entries
    By shaochun in forum General Discussion
    Replies: 5
    Last Post: 12-10-2005, 07:18 PM