Results 1 to 5 of 5

Hows this robots.txt file?

This is a discussion on Hows this robots.txt file? within the General Discussion forums, part of the vBSEO SEO Plugin category; Hello, We: - allow attachments. In fact, that's the only way to show pictures on the forum - run google ...

  1. #1
    Junior Member Array
    Join Date
    Nov 2005
    Posts
    21
    Liked
    0 times

    Hows this robots.txt file?

    Hello,

    We:

    - allow attachments. In fact, that's the only way to show pictures on the forum
    - run google adsense ads

    Our robots.txt is as follows:

    User-agent: Mediapartners-Google*
    Disallow:
    User-agent: *
    Disallow: /forum/referrers.php
    Disallow: /forum/ajax_cron.php
    Disallow: /forum/ajax.php
    Disallow: /forum/calendar.php
    Disallow: /forum/cron.php
    Disallow: /forum/editpost.php
    Disallow: /forum/global.php
    Disallow: /forum/image.php
    Disallow: /forum/inlinemod.php
    Disallow: /forum/joinrequests.php
    Disallow: /forum/login.php
    Disallow: /forum/member.php
    Disallow: /forum/memberlist.php
    Disallow: /forum/misc.php
    Disallow: /forum/moderator.php
    Disallow: /forum/newattachment.php
    Disallow: /forum/newreply.php
    Disallow: /forum/newthread.php
    Disallow: /forum/online.php
    Disallow: /forum/poll.php
    Disallow: /forum/postings.php
    Disallow: /forum/printthread.php
    Disallow: /forum/private.php
    Disallow: /forum/profile.php
    Disallow: /forum/register.php
    Disallow: /forum/report.php
    Disallow: /forum/reputation.php
    Disallow: /forum/search.php
    Disallow: /forum/sendmessage.php
    Disallow: /forum/showgroups.php
    Disallow: /forum/subscription.php
    Disallow: /forum/threadrate.php
    Disallow: /forum/usercp.php
    Disallow: /forum/usernote.php
    Disallow: /forum/users/

    Sitemap:http://www.team-bhp.com/forum/sitemap_index.xml.gz
    Thanks!

  2. #2
    vBSEO Staff Array Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    675 times
    Blog Entries
    2
    you should remove

    User-agent: Mediapartners-Google*
    Disallow:


    Allow is default.

    see also:
    Tutorial how to make a good robots.txt file - vBulletin SEO Forums
    Brian Cummiskey / Crawlability Inc.
    Security bulletin - Patch Level for all supported versions released

    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  3. #3
    Junior Member Array
    Join Date
    Nov 2005
    Posts
    21
    Liked
    0 times
    Thank you Brian, your recommendation has been implemented.

  4. #4
    vBSEO Staff Array Andrés Durán Hewitt's Avatar
    Real Name
    Andrés Durán
    Join Date
    Jul 2009
    Location
    Costa Rica
    Posts
    3,858
    Liked
    565 times
    Blog Entries
    2
    Hello

    I also advise you to implement a regex to avoid Google and other SE's indexing announcement URLs. Most of them are indexed and in the most ocassions they don't have a great SEO value.
    Andrés Durán / Crawlability Inc.
    ˇvBSEO 3.6.0 GOLD Liberado!
    Inaugurando el NUEVO vBSEO Sitemap Generator 3.0. - ˇAHORA disponible para Clientes de vBSEO!

    Síguenos en: Facebook | Síguenos en: Twitter


  5. #5
    Junior Member Array
    Join Date
    Nov 2005
    Posts
    21
    Liked
    0 times
    Thank you for the advice, Andrés. I will have it done.

Similar Threads

  1. Is my robots.tx file ok?
    By swiftor in forum General Discussion
    Replies: 4
    Last Post: 10-11-2010, 05:15 AM
  2. how to configure the robots.txt file
    By universo in forum General Discussion
    Replies: 2
    Last Post: 12-23-2009, 05:30 PM
  3. Robots.txt file changes
    By Blind Dragon in forum General Discussion
    Replies: 3
    Last Post: 09-03-2009, 01:00 PM
  4. Please review my robots.txt file...
    By Sonnie in forum General Discussion
    Replies: 11
    Last Post: 09-08-2006, 03:30 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •