vBulletin 4, the most powerful community software + vBSEO 3.5, the ultimate SEO solution = Your ultimate platform for 2010 and beyond. Click below to learn more.

Results 1 to 7 of 7

Mydomain.com/robots.txt | Mydomain.com/forum/robots.txt : How to config ?

This is a discussion on Mydomain.com/robots.txt | Mydomain.com/forum/robots.txt : How to config ? within the General Discussion forums, part of the vBulletin SEO Discussion category; Hello, I know, we need only 1 robots.txt on a website. But I've got 2 robots. 1 Root ( http://www.mysite.com/robots.txt ...

  1. #1
    Junior Member
    Real Name
    patrick
    Join Date
    Sep 2007
    Posts
    29

    Post Mydomain.com/robots.txt | Mydomain.com/forum/robots.txt : How to config ?

    Hello,

    I know, we need only 1 robots.txt on a website.

    But I've got 2 robots.
    1 Root (http://www.mysite.com/robots.txt)
    1 Forum folder (http://www.mysite.com/forum/robots.txt)

    Question :
    How can I move my robots.txt Forum folder TO Mydomain.com/robots.txt :

    My http://www.mydomain.com/forum/robotx.tx looks like that :
    # Comment the following line (add '#' at the beginning)
    # to disable mod_rewrite functions.
    # Please note: you still need to disable the hack in
    # the vBSEO control panel to stop url rewrites.
    RewriteEngine On

    # Some servers require the Rewritebase directive to be
    # enabled (remove '#' at the beginning to activate)
    # Please note: when enabled, you must include the path
    # to your root vB folder (i.e. RewriteBase /forum/)
    #RewriteBase /

    RewriteCond %{HTTP_HOST} !^www\.mydomain\.com
    RewriteRule (.*) http://www.mydomain.com/forum/$1 [R=301,L]

    RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

    RewriteCond %{REQUEST_URI} !(admincp/|modcp/|cron)
    RewriteRule ^((archive/)?(.*\.php(/.*)?)?)$ vbseo.php [L,QSA]

    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
    RewriteRule ^(.+)$ vbseo.php [L,QSA]


    What should it looks like If I move it to my http://www.mydomain.com/robots.txt ?


    ----------- Like that ? It is correct please ? ------------------
    # Comment the following line (add '#' at the beginning)
    # to disable mod_rewrite functions.
    # Please note: you still need to disable the hack in
    # the vBSEO control panel to stop url rewrites.
    RewriteEngine On

    # Some servers require the Rewritebase directive to be
    # enabled (remove '#' at the beginning to activate)
    # Please note: when enabled, you must include the path
    # to your root vB folder (i.e. RewriteBase /forum/)
    #RewriteBase /

    RewriteCond %{HTTP_HOST} !^www\.mydomain\.com
    RewriteRule (.*) http://www.mydomain.com/forum/$1 [R=301,L]

    RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ forum/vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

    RewriteCond %{REQUEST_URI} !(admincp/|modcp/|cron)
    RewriteRule ^((archive/)?(.*\.php(/.*)?)?)$ vbseo.php [L,QSA]

    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
    RewriteRule ^(.+)$ forum/vbseo.php [L,QSA]


    Thank you.

  2. #2
    vBSEO.com Webmaster Mert Gökçeimam's Avatar
    Real Name
    Lizard King
    Join Date
    Oct 2005
    Location
    Istanbul
    Posts
    16,944
    Blog Entries
    4
    Hello ,

    Robots.txt file should only reside on your root folder.
    Mert Gökçeimam / Crawlability Inc.

    Yeni vBSEO Kontrol Paneli ve vBSEO 3.5 Yayın Tarihleri

    vBSEO 3.5 Beta 2 Yayınlandı - vBulletin 4 Uyumlu!



    Kişisel Sitem : Felsefe - Kültür merkeziniz

  3. #3
    Junior Member
    Real Name
    patrick
    Join Date
    Sep 2007
    Posts
    29
    Yes,
    My root folder is www.mydomain.com
    But my Forum is www.mydomain.com/forum/

    My robots.txt should be in www.mydomain.com/robots.txt

    The question is :
    This robots.txt it is correct ? (the paths in purple color) ?

    Thx !

    # Comment the following line (add '#' at the beginning)
    # to disable mod_rewrite functions.
    # Please note: you still need to disable the hack in
    # the vBSEO control panel to stop url rewrites.
    RewriteEngine On

    # Some servers require the Rewritebase directive to be
    # enabled (remove '#' at the beginning to activate)
    # Please note: when enabled, you must include the path
    # to your root vB folder (i.e. RewriteBase /forum/)
    #RewriteBase /

    RewriteCond %{HTTP_HOST} !^www\.mydomain\.com
    RewriteRule (.*) http://www.mydomain.com/forum/$1 [R=301,L]

    RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ forum/vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

    RewriteCond %{REQUEST_URI} !(admincp/|modcp/|cron)
    RewriteRule ^((archive/)?(.*\.php(/.*)?)?)$ vbseo.php [L,QSA]

    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
    RewriteRule ^(.+)$ forum/vbseo.php [L,QSA]

  4. #4
    vBSEO.com Webmaster Mert Gökçeimam's Avatar
    Real Name
    Lizard King
    Join Date
    Oct 2005
    Location
    Istanbul
    Posts
    16,944
    Blog Entries
    4
    You should use your sitemap url in your robots.txt file

    Code:
    User-agent: *
    Disallow: 
    Sitemap: http://www.mydomain.com/forum/sitemap_index.xml.gz
    
    You need to upload default vBSEO .htaccess file to your forum folder and should not add the purple parts in .htaccess file.
    Mert Gökçeimam / Crawlability Inc.

    Yeni vBSEO Kontrol Paneli ve vBSEO 3.5 Yayın Tarihleri

    vBSEO 3.5 Beta 2 Yayınlandı - vBulletin 4 Uyumlu!



    Kişisel Sitem : Felsefe - Kültür merkeziniz

  5. #5
    Junior Member
    Real Name
    patrick
    Join Date
    Sep 2007
    Posts
    29
    I confused between robots and htaccess.. I'm tired sorry.

  6. #6
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    seoville
    Posts
    3,883
    htaccess is a file your SERVER uses as instructions for apache/etc.

    robots.txt is a file (that only works in root) that provides instructions for OUTSIDE services/bots.
    Brian Cummiskey / Crawlability Inc.
    Sneek Preview Video of the new Control Panel

    vBSEO 3.5 RC2 (Final)- Released for your Evaluation


    Check us out on FaceBook!

  7. #7
    Junior Member
    Real Name
    patrick
    Join Date
    Sep 2007
    Posts
    29
    Thanks you all.
    It's time for me to go bed ^^ lol

    Thank you team

Similar Threads

  1. linkbacks and mydomain
    By kavehch in forum LinkBacks
    Replies: 0
    Last Post: 05-09-2009, 01:10 AM
  2. all forum has restricted by robots.txt !
    By mountainboarder21 in forum General Discussion
    Replies: 4
    Last Post: 10-19-2008, 11:51 PM
  3. Redirecting /forums/robots.txt to /robots.txt - Is it good?
    By MadK in forum Custom Rewrite Rules
    Replies: 6
    Last Post: 08-22-2008, 06:29 PM
  4. Little .htaccess username.mydomain.com
    By Snake in forum Custom Rewrite Rules
    Replies: 0
    Last Post: 05-04-2007, 09:11 AM
  5. Incorrectly redirecting from www.mydomain to mydomain.com
    By fischermx in forum Troubleshooting
    Replies: 10
    Last Post: 09-05-2006, 07:13 PM

Tags for this Thread