vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
New vBSEO Discount Level for Network Builders Meet vBSEO Team in New York (Nov. 3rd & 4th) vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

robots.txt

This is a discussion on robots.txt within the General Discussion forums, part of the vBSEO Google/Yahoo Sitemap category; Does anyone have any recommendations on robots.txt file? I would rather be safe than sorry Thanks for your thoughts...

Go Back   vBulletin SEO Forums > vBSEO Google/Yahoo Sitemap > General Discussion

robots.txt robots.txt

Enhancing 80 million pages.

 Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Reply

 

LinkBack Thread Tools
  #1  
Old 11-21-2007, 02:12 PM
Junior Member
  
Real Name: wzrdsucks
Join Date: Oct 2007
Posts: 13
robots.txt
Does anyone have any recommendations on robots.txt file?

I would rather be safe than sorry

Thanks for your thoughts
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2  
Old 11-21-2007, 03:50 PM
briansol's Avatar
Senior Member
vBSEO Pre-Release TeamDesign for SEOBig Board Administrator
  
Real Name: Brian
Join Date: Apr 2006
Location: Central CT, USA
Posts: 5,459
assuming your forums are in / (root)

add /forums/ to the beginning if you are in /forums

also note, i block showpost, the archive, and print versions.


Code:
User-agent: *

Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /faq.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /payments.php
Disallow: /showpost.php
Disallow: /showgroups.php
Disallow: /spiders.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /admincp/
Disallow: /cgi-bin/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /customavatars/
Disallow: /xmlrpc.php
Disallow: /archive/
Disallow: /sitemap/


#use the following if you have photopost:

Disallow: /gallery/member.php
Disallow: /gallery/showgallery.php?cat=
Disallow: /gallery/showgallery.php?limit=
Disallow: /gallery/comshow.php
Disallow: /gallery/search.php
Disallow: /gallery/*search.php
Disallow: /gallery/showframe.php
Disallow: /gallery/slideshow.php
Disallow: /gallery/*limit
Disallow: /gallery/*ppuser
Disallow: /gallery/*views
Disallow: /gallery/*date
Disallow: /gallery/*recent
Disallow: /gallery/*in


Sitemap: http://typeyourdomainhere.com/sitemap_index.xml.gz
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3  
Old 11-21-2007, 03:53 PM
Junior Member
  
Real Name: wzrdsucks
Join Date: Oct 2007
Posts: 13
Thanks alot, this will be a good start.

should I add this line as well?

Sitemap: http://typeyourdomainhere.com/sitemap_index.xml.gz
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4  
Old 11-21-2007, 04:30 PM
Sysop's Avatar
Senior Member
  
Real Name: Toni
Join Date: Oct 2007
Location: Italy
Posts: 119
that's be good in the robot.txt file?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5  
Old 11-21-2007, 04:42 PM
Keith Cohen's Avatar
vBSEO Staff
vBSEO Total Customer SupportBig Board Administrator
  
Real Name: Keith Cohen
Join Date: Jul 2005
Location: Raleigh, NC USA
Posts: 6,264
Blog Entries: 1
Quote:
Originally Posted by Sysop View Post
that's be good in the robot.txt file?
Yes: Robots.txt now support the sitemap file
__________________
Keith Cohen / Crawlability Inc.
vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

vBSEO Google Sitemap Generator - Version 2.2 Released Mandatory Upgrade for vBSEO 3.2.0 GOLD

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.


My Personal Sites: My Blog | GPS Discussion Forum
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply

« Ideal Settins for sitemap spider crawling? | Seo results 1 week »
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads

Thread Thread Starter Forum Replies Last Post
Google ignores my robots.txt shokmuzik.com General Discussion 2 10-25-2007 08:09 PM
Robots.txt now support the sitemap file Mohamed Mostafa General Discussion 35 08-24-2007 10:33 AM
New SEO Tools: Robots.txt Builder, Term Targeting & Crawl Test Eros SEO Buzz 0 04-21-2007 06:30 AM
add sitemap to robots.txt for autodiscovery GrendelKhan{TSU} SEO Buzz 2 04-13-2007 03:21 PM
Google and robots.txt jdougher General Discussion 2 02-22-2007 01:36 PM


All times are GMT -4. The time now is 06:42 PM.

Contact Us - vBSEO Home - Site Map - Top

Powered by vBulletin Version 3.8.0 Beta 3
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.