Results 1 to 6 of 6

robots.txt - Am I missing something?

This is a discussion on robots.txt - Am I missing something? within the General Discussion forums, part of the vBSEO SEO Plugin category; According to my AdSense account (Site Diagnostics page) the following URLs are being blocked in my robots.txt file. This is ...

  1. #1
    Junior Member
    Real Name
    Christina
    Join Date
    Feb 2010
    Posts
    9
    Liked
    0 times

    robots.txt - Am I missing something?

    According to my AdSense account (Site Diagnostics page) the following URLs are being blocked in my robots.txt file. This is just a small snippet of the many URLs being blocked (there are also forum posts).

    I did do a robots.txt update on February 23. Does it take a few days for Google to catch up? Or is there something that is blocking Google in my robots.txt? Thanks.

    picture-2..png

    http://www.fitnessthinktank.com/robots.txt

    ** I just noticed I posted this in the wrong forum. My apologies.

  2. #2
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,782
    Liked
    648 times
    Blog Entries
    2
    It will take a few days for a robots file to have any effect. What did you have in it before?
    Brian Cummiskey / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  3. #3
    Junior Member
    Real Name
    Christina
    Join Date
    Feb 2010
    Posts
    9
    Liked
    0 times
    Thank you, Brian. I am not exactly sure what was in it before because someone else was working on my site. I will check again in a couple of days and see if there is a change. I assume the robots file looks ok? Do I need to place a link to my sitemap? Anything else?

  4. #4
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,782
    Liked
    648 times
    Blog Entries
    2
    The robots file looks ok. As it sits, it shouldn't be blocking that page.

    You can add the sitemap directive at the bottom of the file like:

    Sitemap: http://site.com/sitemap_index.xml.gz
    Brian Cummiskey / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  5. #5
    Junior Member
    Real Name
    Christina
    Join Date
    Feb 2010
    Posts
    9
    Liked
    0 times
    Thanks again for your help, Brian.

  6. #6
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,782
    Liked
    648 times
    Blog Entries
    2
    np
    Brian Cummiskey / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


Similar Threads

  1. Replies: 6
    Last Post: 10-19-2009, 11:55 AM
  2. Redirecting /forums/robots.txt to /robots.txt - Is it good?
    By MadK in forum Custom Rewrite Rules
    Replies: 6
    Last Post: 08-22-2008, 07:29 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •