Results 1 to 5 of 5

robots.txt errors

This is a discussion on robots.txt errors within the Troubleshooting forums, part of the vBSEO Google/Yahoo Sitemap category; Hi - I checked google's webmasters and I found lotof urls restricted by my robots.txt i made according to a ...

  1. #1
    Senior Member AdamFL's Avatar
    Real Name
    Adam
    Join Date
    Oct 2008
    Location
    South Florida
    Posts
    167
    Liked
    0 times

    robots.txt errors

    Hi - I checked google's webmasters and I found lotof urls restricted by my robots.txt i made according to a post here somewhere :( I'm scared that i'm killing my website's SEO myself
    http://www.gomorocco.com/robots.txt

    Attached this is a screenshot of some of the errors

    Please if it's the case can you suggest any robots.txt that will optomise my website SEO?

    Thanks a million.
    Attached Thumbnails Attached Thumbnails errors.jpg  

  2. #2
    vBSEO Staff Ace Shattock's Avatar
    Real Name
    Ace Shattock
    Join Date
    Jul 2005
    Location
    Auckland, New Zealand, New Zealand
    Posts
    4,012
    Liked
    13 times
    From looking at just the screenshot of Page 5 of the errors, plus the robots.txt:

    Looks to me like the robots.txt is fine, but you are including showpost.php URLs in your Sitemap.

    Remove those, and those errors will reduce significantly.

  3. #3
    Senior Member AdamFL's Avatar
    Real Name
    Adam
    Join Date
    Oct 2008
    Location
    South Florida
    Posts
    167
    Liked
    0 times
    Thank you Ace Shattock for taking time to review my robots.txt and post.
    I set "Include Show Post Pages" to no. I'm not sure if it got a direct impact on the general performance but the less errors the better I guess


    Thank you again!

  4. #4
    Senior Member AdamFL's Avatar
    Real Name
    Adam
    Join Date
    Oct 2008
    Location
    South Florida
    Posts
    167
    Liked
    0 times
    Oh I just realised that my robots.txt might be was wrong since everything was /forum
    Expl: Disallow: /forum/ajax.php instead of Disallow: /ajax.php

    I fixed it, I hope that's ok now?
    hah I should of post this thread in the clients area to get a better feedback
    Last edited by AdamFL; 01-31-2009 at 07:01 PM.

  5. #5
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    robots.txt MUST be in root. It does NOTHING in any other directory.

    if your path is site.com/forums/ajax.php
    then you need
    Disallow: /forum/ajax.php in your robots.txt

Similar Threads

  1. Redirecting /forums/robots.txt to /robots.txt - Is it good?
    By MadK in forum Custom Rewrite Rules
    Replies: 6
    Last Post: 08-22-2008, 06:29 PM
  2. Replies: 10
    Last Post: 07-21-2008, 06:04 PM
  3. The best robots.txt
    By MaestroX in forum General Discussion
    Replies: 1
    Last Post: 01-28-2007, 04:53 PM
  4. Robots.txt Help
    By soletrader in forum General Discussion
    Replies: 7
    Last Post: 10-13-2006, 09:31 AM
  5. How is my robots.txt?
    By libertylounge in forum Troubleshooting
    Replies: 6
    Last Post: 08-26-2006, 12:17 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •