Results 1 to 3 of 3

robots.txt

This is a discussion on robots.txt within the General Discussion forums, part of the vBulletin SEO Discussion category; Can someone more knowledgeable than I take a look at my robots.txt and let me know if it's ok as ...

  1. #1
    Junior Member
    Real Name
    frank
    Join Date
    Feb 2008
    Posts
    17
    Liked
    0 times

    robots.txt

    Can someone more knowledgeable than I take a look at my robots.txt and let me know if it's ok as is or if there's something that should be changed....

    I am running vb, vbportal, photopost gallery, photopost reviews, vb blog. The robots.txt was part of the vbportal install i believe that I updated for showpost and sitemap as read on this forum.

    What I dont understand is why the gallery and reviews sections are being disallowed?

    Thank you in advance for your help!

    # Allow Archiver
    User-agent: ia_archiver
    Allow: /

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /forums/admincp/
    Disallow: /forums/clientscript/
    Disallow: /forums/cpstyles/
    Disallow: /forums/customavatars/
    Disallow: /forums/dlm/
    Disallow: /forums/images/
    Disallow: /forums/includes/
    Disallow: /forums/install#/
    Disallow: /forums/modcp/
    Disallow: /forums/showpost.php
    Disallow: /forums/subscriptions/
    Disallow: /forums/themes/
    Disallow: /To be removed/
    Disallow: /XXpafaq/
    Disallow: /amember/
    Disallow: /banners/
    Disallow: /cpdemo/
    Disallow: /downloads/
    Disallow: /gallery/
    Disallow: /images/
    Disallow: /includes/
    Disallow: /language/
    Disallow: /modules/
    Disallow: /portalcp/
    Disallow: /ppclassifieds/
    Disallow: /reviews/
    Disallow: /store/
    Disallow: /vbpinstall/
    Disallow: /webalizer/
    Disallow: banners.php
    Disallow: footer.php
    Disallow: global.php
    Disallow: header.php
    Disallow: mainfile.php
    Disallow: user.php
    Disallow: search.php
    Disallow: modules.php
    Disallow: articles.php
    Disallow: external.php
    Sitemap: http://www.mysite.com/forums/sitemap_index.xml.gz

  2. #2
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    you can remove them if you want.

    most gallery systems are LOADED with duplicate content urls, and may be why that person decided to block it entirely.

  3. #3
    Junior Member
    Real Name
    frank
    Join Date
    Feb 2008
    Posts
    17
    Liked
    0 times
    Hi thank you for your reply.

    Do you see anything that should be added or removed besides the gallery and reviews?

    Thank you

Similar Threads

  1. robots.txt
    By Zenith in forum General Discussion
    Replies: 64
    Last Post: 12-01-2010, 08:52 PM
  2. About robots.txt
    By Diver in forum General Discussion
    Replies: 1
    Last Post: 02-18-2008, 08:18 PM
  3. No Robots.txt
    By friendly in forum Troubleshooting
    Replies: 1
    Last Post: 10-26-2006, 11:30 AM
  4. How is my robots.txt?
    By libertylounge in forum Troubleshooting
    Replies: 6
    Last Post: 08-26-2006, 01:17 PM
  5. robots.txt
    By croportal in forum General Discussion
    Replies: 1
    Last Post: 02-05-2006, 01:31 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •