Results 1 to 10 of 10

robots.txt question

This is a discussion on robots.txt question within the General Discussion forums, part of the vBSEO SEO Plugin category; Can anyone help me setup my robots.txt file? Here is what I have so far, but I want to make ...

  1. #1
    Senior Member Citricguy's Avatar
    Real Name
    Josh
    Join Date
    Nov 2006
    Location
    Maui, Hawaii
    Posts
    143
    Liked
    0 times
    Blog Entries
    7

    robots.txt question

    Can anyone help me setup my robots.txt file? Here is what I have so far, but I want to make sure I’m not hurting myself with the settings I have now, and if there is anything I could add.

    Code:
    
    User-agent: *
    Disallow: /ajax.php
    Disallow: /attachment.php
    Disallow: /calendar.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /global.php
    Disallow: /image.php
    Disallow: /inlinemod.php
    Disallow: /joinrequests.php
    Disallow: /login.php
    Disallow: /member.php
    Disallow: /memberlist.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newattachment.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /poll.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /register.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /search.php
    Disallow: /sendmessage.php
    Disallow: /showgroups.php
    Disallow: /subscription.php
    Disallow: /threadrate.php
    Disallow: /usercp.php
    Disallow: /usernote.php
    

  2. #2
    Senior Member BamaStangGuy's Avatar
    Real Name
    Brent Wilson
    Join Date
    Aug 2005
    Location
    Huntsville, Alabama
    Posts
    2,484
    Liked
    1 times
    I use this one at vBulletin Zone

    User-agent: *
    Disallow: /admincp/
    Disallow: /cgi-bin/
    Disallow: /clientscript/
    Disallow: /includes/
    Disallow: /install/
    Disallow: /modcp/
    Disallow: /printthread.php
    Disallow: /subscription.php
    Disallow: /payments.php
    Disallow: /profile.php
    Disallow: /faq.php
    Disallow: /calendar.php
    Disallow: /private.php
    Disallow: /sendmessage.php
    Disallow: /sendmessage.php?do=
    Disallow: /showgroups.php
    Disallow: /reputation.php
    Disallow: /report.php
    Disallow: /threadrate.php
    Disallow: /postings.php
    Disallow: /newthread.php
    Disallow: /newreply.php
    Disallow: /register.php
    Disallow: /login.php
    Disallow: /faq.php
    Disallow: /image.php
    Disallow: /cron.php
    Disallow: /joinrequests.php
    Disallow: /usercp.php

  3. #3
    Senior Member Citricguy's Avatar
    Real Name
    Josh
    Join Date
    Nov 2006
    Location
    Maui, Hawaii
    Posts
    143
    Liked
    0 times
    Blog Entries
    7
    hey thanks! I like that one alot better.

  4. #4
    Member Crow's Avatar
    Real Name
    Crow
    Join Date
    Jan 2006
    Location
    Michigan
    Posts
    82
    Liked
    0 times
    Blog Entries
    1
    Bama, Mind if I use yours also?

  5. #5
    Senior Member BamaStangGuy's Avatar
    Real Name
    Brent Wilson
    Join Date
    Aug 2005
    Location
    Huntsville, Alabama
    Posts
    2,484
    Liked
    1 times
    Go for it

  6. #6
    Senior Member Brandon Sheley's Avatar
    Real Name
    Brandon Sheley
    Join Date
    Oct 2005
    Location
    Kansas
    Posts
    2,348
    Liked
    20 times
    Blog Entries
    1
    I have a question..

    a client wants to keep the google image bot away from everything.

    will this work if I add this to that ?

    Code:
    User-agent: Googlebot-Image
    Disallow: /
    User-agent: *
    Disallow: /ajax_cron.php
    Disallow: /ajax.php
    Disallow: /attachment.php
    Disallow: /calendar.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /global.php
    Disallow: /image.php
    Disallow: /inlinemod.php
    Disallow: /joinrequests.php
    Disallow: /login.php
    Disallow: /member.php
    Disallow: /memberlist.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newattachment.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /poll.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /profile.php
    Disallow: /register.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /search.php
    Disallow: /sendmessage.php
    Disallow: /showgroups.php
    Disallow: /subscription.php
    Disallow: /threadrate.php
    Disallow: /usercp.php
    Disallow: /usernote.php
    My forums: General Forums | Admin Talk (running xenforo)

  7. #7
    Senior Member ruhrpottforum's Avatar
    Real Name
    Chris
    Join Date
    Sep 2006
    Location
    Essen, Nordrhein-Westfalen, Germany
    Posts
    160
    Liked
    1 times

  8. #8
    Senior Member Snake's Avatar
    Real Name
    Josh
    Join Date
    Oct 2005
    Location
    Cleveland, OH
    Posts
    413
    Liked
    1 times
    Why are you doing this?
    Metal Gear Forums - Discussion on the popular series of console stealth-based games.

    My Mods: Coming Soon | My Tutorials: Coming Soon

  9. #9
    Junior Member
    Real Name
    Colby
    Join Date
    Aug 2006
    Posts
    28
    Liked
    0 times
    User-agent: *
    Disallow: /forum/ajax.php
    Disallow: /forum/attachment.php
    Disallow: /forum/calendar.php
    Disallow: /forum/cron.php
    Disallow: /forum/editpost.php
    Disallow: /forum/global.php
    Disallow: /forum/image.php
    Disallow: /forum/inlinemod.php
    Disallow: /forum/joinrequests.php
    Disallow: /forum/login.php
    Disallow: /forum/member.php
    Disallow: /forum/memberlist.php
    Disallow: /forum/misc.php
    Disallow: /forum/moderator.php
    Disallow: /forum/newattachment.php
    Disallow: /forum/newreply.php
    Disallow: /forum/newthread.php
    Disallow: /forum/online.php
    Disallow: /forum/poll.php
    Disallow: /forum/postings.php
    Disallow: /forum/printthread.php
    Disallow: /forum/private.php
    Disallow: /forum/profile.php
    Disallow: /forum/register.php
    Disallow: /forum/report.php
    Disallow: /forum/reputation.php
    Disallow: /forum/search.php
    Disallow: /forum/sendmessage.php
    Disallow: /forum/showgroups.php
    Disallow: /forum/subscription.php
    Disallow: /forum/threadrate.php
    Disallow: /forum/usercp.php
    Disallow: /forum/usernote.php

    is this correct i did i put it in my forum folder? is there anyway to check?

  10. #10
    Senior Member BamaStangGuy's Avatar
    Real Name
    Brent Wilson
    Join Date
    Aug 2005
    Location
    Huntsville, Alabama
    Posts
    2,484
    Liked
    1 times
    You don't put it in your forum folder you put it in your root of your domain. Should be able to get to it by yourdomain.com/robots.txt

Similar Threads

  1. G. Sitemaps (2200 not found/unreachable urls + robots.txt question)
    By JustinBrand in forum General Discussion
    Replies: 1
    Last Post: 10-15-2006, 05:34 PM
  2. MSNbot - robots.txt
    By curriertech in forum General Discussion
    Replies: 1
    Last Post: 10-05-2006, 08:29 AM
  3. Question about location of robots.txt file
    By Mike in forum General Discussion
    Replies: 2
    Last Post: 05-22-2006, 01:05 PM
  4. robots.txt entries
    By shaochun in forum General Discussion
    Replies: 5
    Last Post: 12-10-2005, 07:18 PM
  5. Possible to use Disallow: /*?pp=10 in robots.txt?
    By PageUp in forum General Discussion
    Replies: 3
    Last Post: 11-03-2005, 09:01 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •