Hello,
can someone give me a good robot.txt file for my site setup?
thanks
This is a discussion on what is a good robot.txt for my site structure? within the General Discussion forums, part of the vBSEO Google/Yahoo Sitemap category; Hello, can someone give me a good robot.txt file for my site setup? thanks...
Hello,
can someone give me a good robot.txt file for my site setup?
thanks
Am I optimized with this below?
PHP Code:User-agent: *
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/includes/
Disallow: /forums/modules/
Disallow: /forums/vbseo/
Disallow: /forums/vbseo_sitemap/
Disallow: /forums/ajax.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/faq.php
Disallow: /forums/global.php
Disallow: /forums/infraction.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderation.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Here's a good tutorial for robots.txt file : Tutorial how to make a good robots.txt file
Marco Mamdouh Fahem / Crawlability Inc.
Security vbulletin - Patch Level for all supported versions released!
Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!
never ever include your admin directories.
Disallow: /forums/admincp/
Disallow: /forums/modcp/
All you are doing is telling would-be hackers where your private areas are located.
Of course, we advise all vbseo customers to re-name their admin directories and lock them down for security purposes.
Brian Cummiskey / Crawlability Inc.
Security vbulletin - Patch Level for all supported versions released!
Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!
Disallow: /vbseo/
Disallow: /VBSEO_DAT_FOLDER_BOT/
Disallow: /vbseo_sitemap
Disallow: /vbseo.php
Will this have an ill effect if i exclude the above, ie preventing bot from seeing sitemap?
thanks
Why you will disallow sitemap generator folders ? I've never see it in Google results and no one has disallow it, You can disallow vbseo.php and vbseocp.php and /vbseo/ .
Marco Mamdouh Fahem / Crawlability Inc.
Security vbulletin - Patch Level for all supported versions released!
Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!