Results 1 to 2 of 2

Robots.txt, is it needed?

This is a discussion on Robots.txt, is it needed? within the General Discussion forums, part of the vBulletin SEO Discussion category; Just wondering, with vbseo and the sitemap installed, is there a reason to have a robots.txt? Like, are they in ...

  1. #1
    Junior Member
    Real Name
    Carlos
    Join Date
    Mar 2007
    Posts
    21
    Liked
    0 times

    Robots.txt, is it needed?

    Just wondering, with vbseo and the sitemap installed, is there a reason to have a robots.txt? Like, are they in any way helpful?

  2. #2
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,789
    Liked
    657 times
    Blog Entries
    2
    It's a good idea if you need or want to block stuff from bots without making the area private.

    ie, search.php, usercp.php... bots will never index or use these pages, but there's no point to making them private, so to save the bot some time and to let it focus on your important areas, its nice to block those areas.

Similar Threads

  1. Replies: 6
    Last Post: 10-19-2009, 10:55 AM
  2. robots.txt
    By zems in forum General Discussion
    Replies: 0
    Last Post: 05-03-2009, 03:15 PM
  3. Redirecting /forums/robots.txt to /robots.txt - Is it good?
    By MadK in forum Custom Rewrite Rules
    Replies: 6
    Last Post: 08-22-2008, 06:29 PM
  4. About robots.txt
    By Diver in forum General Discussion
    Replies: 1
    Last Post: 02-18-2008, 07:18 PM
  5. Robots.txt Help
    By soletrader in forum General Discussion
    Replies: 7
    Last Post: 10-13-2006, 09:31 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •