Results 1 to 9 of 9

My SE Bots Activity Details

This is a discussion on My SE Bots Activity Details within the General Discussion forums, part of the vBSEO Google/Yahoo Sitemap category; What would you say about this activity? Something more to disable in robots.txt? What? Thanks a lot! Marcel...

  1. #1
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    247
    Liked
    0 times

    My SE Bots Activity Details

    What would you say about this activity?
    Something more to disable in robots.txt?
    What?

    Thanks a lot!
    Marcel
    test.jpg

  2. #2
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    247
    Liked
    0 times
    The day before could be more interesting:

    test2.jpg

  3. #3
    Member Mike54's Avatar
    Real Name
    Mike
    Join Date
    Oct 2006
    Posts
    72
    Liked
    0 times
    I block most of that via robots.txt.

    There's nothing to be gained by having the search engines crawling register.php, for instance. You want to keep them directed to your forum content.

  4. #4
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    247
    Liked
    0 times
    And which of them do you leave unblocked exactly?

  5. #5
    Member Mike54's Avatar
    Real Name
    Mike
    Join Date
    Oct 2006
    Posts
    72
    Liked
    0 times
    It's a bit of different strokes for different folks, but from your list I would block -

    printthread.php
    search.php
    newreply.php
    private.php
    attachment.php
    album.php
    register.php
    poll.php
    newthread.php
    editpost.php
    inlinemod.php
    arcade.php
    clientscript/
    report.php
    subscription.php
    customavatars/
    profile.php
    sendmesssage.php

    Briansol suggests rewriting everything you want indexed with no extensions and then blocking all .php scripts. This is on my list of things to do, as I think it makes excellent sense.

    It gets recommended a lot, but his Ultimate Guide is loaded with great advice - Briansol's Ultimate Guide to vBSEO

  6. #6
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    247
    Liked
    0 times
    What I don't understand:
    This is my robots.txt:

    # robots.txt for Symptome, Ursachen von Krankheiten - Forum, Hilfe, Tipps zu Gesundheit
    # Zugriff auf alle Dateien erlauben
    User-agent: *
    Disallow: /images/
    Disallow: /faq.php
    Disallow: /attachment.php
    Disallow: /avatar.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /calendar.php
    Disallow: /member.php
    Disallow: /memberlist.php
    Disallow: /printthread.php
    Disallow: /sendmessage.php
    Disallow: /register.php
    Disallow: /sendtofriend.php
    Disallow: /login.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /search.php
    Disallow: /subscription.php
    Disallow: /threadrate.php
    Disallow: /private.php
    Disallow: /misc.php
    Disallow: /poll.php
    Disallow: /showpost.php
    Disallow: /showtext.php
    Disallow: /profile.php
    Disallow: /usercp.php
    Disallow: /showgroups.php
    Disallow: /cmps_index.php
    Disallow: ../wikired/
    Disallow: /sitemap/
    Disallow: /archive/

    sitemap: http://www.symptome.ch/vbboard/sitemap_index.xml.gz
    Why is there still bot activity on scripts like printthread and others, if I have disallowed them a long time ago?

    If I enter my site, I'm directed to Symptome, Ursachen von Krankheiten - Forum, Hilfe, Tipps zu Gesundheit automatically.
    But the above robots.txt is places within Symptome, Ursachen von Krankheiten - Forum, Hilfe, Tipps zu Gesundheit
    Could it be, that the paths are wrong?

  7. #7
    Member Mike54's Avatar
    Real Name
    Mike
    Join Date
    Oct 2006
    Posts
    72
    Liked
    0 times
    Yes, you need specify the vbboard directory.

    Disallow: /vbboard/images/
    Disallow: /vbboard/faq.php

  8. #8
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    247
    Liked
    0 times
    Thanks a lot!
    So my robots.txt did quite nothing. Good to know.

  9. #9
    Senior Member
    Real Name
    Admin
    Join Date
    Nov 2006
    Posts
    247
    Liked
    0 times
    Some days later, the SE bot activity still shows the same.
    Why? Does it take so long to see my changed robots.txt?

Similar Threads

  1. SE Bots Activity Details,crawling members!
    By Seleno in forum Troubleshooting
    Replies: 4
    Last Post: 04-11-2008, 11:10 PM
  2. SE Bots Activity
    By 7ayr.net in forum General Discussion
    Replies: 11
    Last Post: 12-04-2007, 09:49 AM
  3. SE Bots Activity Log
    By Itworx4me in forum Troubleshooting
    Replies: 12
    Last Post: 11-26-2006, 01:29 PM
  4. SE Bots Activity Log
    By Sonnie in forum Troubleshooting
    Replies: 8
    Last Post: 09-30-2006, 03:48 PM
  5. SE Bots Activity Details
    By fsb-holger in forum Bug Reporting
    Replies: 3
    Last Post: 11-22-2005, 09:35 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •