Results 1 to 4 of 4

Robot.txt file

This is a discussion on Robot.txt file within the General Discussion forums, part of the vBulletin SEO Discussion category; Can someone tell me how this looks: User-agent: * Disallow: /admincp/ Disallow: /cgi-bin/ Disallow: /clientscript/ Disallow: /includes/ Disallow: /install/ Disallow: ...

  1. #1
    Senior Member BamaStangGuy's Avatar
    Real Name
    Brent Wilson
    Join Date
    Aug 2005
    Location
    Huntsville, Alabama
    Posts
    2,484
    Liked
    1 times

    Robot.txt file

    Can someone tell me how this looks:

    User-agent: *
    Disallow: /admincp/
    Disallow: /cgi-bin/
    Disallow: /clientscript/
    Disallow: /includes/
    Disallow: /install/
    Disallow: /gallery/showmembers.php
    Disallow: /gallery/misc.php
    Disallow: /gallery/comments.php?
    Disallow: /modcp/
    Disallow: /subscriptions/
    Disallow: /calendar.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /faq.php
    Disallow: /global.php
    Disallow: /image.php
    Disallow: /joinrequests.php
    Disallow: /login.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newattachment.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /register.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /sendmessage.php
    Disallow: /showgroups.php
    Disallow: /showpost.php
    Disallow: /subscription.php
    Disallow: /subscriptions.php
    Disallow: /threadrate.php
    Disallow: /usercp.php

  2. #2
    Member
    Real Name
    Mike Fara
    Join Date
    Feb 2006
    Posts
    56
    Liked
    7 times
    I have noticed that vbseo.com doesn't use a robots.txt. I have wondered if a gigantic robots.txt file is more benefit than it is worth. Is there any evidence the spiders may penalize for this? I am considering just eliminating the whole robots.txt...

  3. #3
    Senior Member
    Real Name
    Jon
    Join Date
    Mar 2006
    Posts
    158
    Liked
    0 times
    Mine is HUGE... I hope they dont penalize for it lol...

    I just did it yesterday.

    User-agent: *
    Disallow: /attachment.php
    Disallow: /avatar.php
    Disallow: /calendar.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /faq.php
    Disallow: /global.php
    Disallow: /joinrequests.php
    Disallow: /image.php
    Disallow: /login.php
    Disallow: /member.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newreply.php
    Disallow: /newattachment.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /poll.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /profile.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /register.php
    Disallow: /search.php
    Disallow: /showgroups.php
    Disallow: /showpost.php
    Disallow: /sendmessage.php
    Disallow: /subscription.php
    Disallow: /sendtofriend.php
    Disallow: /threadrate.php
    Disallow: /usercp.php
    Disallow: /memberlist.php
    Disallow: /administrators/
    Disallow: /moderators/
    Disallow: /images/
    Disallow: /admincp/
    Disallow: /cgi-bin/
    Disallow: /clientscript/
    Disallow: /includes/
    Disallow: /install/
    Disallow: /modcp/

  4. #4
    Senior Member
    Real Name
    FAA Zooman
    Join Date
    Dec 2005
    Location
    Cumbria, UK
    Posts
    208
    Liked
    0 times
    I just came along to cheat on a robot.txt for vb I have to say that I think that they both are very good.

Similar Threads

  1. Problem with the sitemap file
    By Toocool in forum Troubleshooting
    Replies: 3
    Last Post: 01-29-2006, 04:33 PM
  2. Which file do we manually submit to google?
    By Mike in forum General Discussion
    Replies: 5
    Last Post: 11-27-2005, 06:24 PM
  3. Replies: 2
    Last Post: 11-04-2005, 12:52 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •