Just wondering, with vbseo and the sitemap installed, is there a reason to have a robots.txt? Like, are they in any way helpful?
This is a discussion on Robots.txt, is it needed? within the General Discussion forums, part of the vBulletin SEO Discussion category; Just wondering, with vbseo and the sitemap installed, is there a reason to have a robots.txt? Like, are they in ...
Just wondering, with vbseo and the sitemap installed, is there a reason to have a robots.txt? Like, are they in any way helpful?
It's a good idea if you need or want to block stuff from bots without making the area private.
ie, search.php, usercp.php... bots will never index or use these pages, but there's no point to making them private, so to save the bot some time and to let it focus on your important areas, its nice to block those areas.
Brian Cummiskey / Crawlability Inc.
Security vbulletin - Patch Level for all supported versions released!
Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!