vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
vBSEO Total Support Team Launches DeskPro New vBSEO Discount Level for Network Builders vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

Do i need a robots.txt?

This is a discussion on Do i need a robots.txt? within the Google Adsense, YPN, & Other Ad Networks forums, part of the Monetizing category; I'm just wondering, will it help or hurt my site? Do i really need it?...

Go Back   vBulletin SEO Forums > Monetizing > Google Adsense, YPN, & Other Ad Networks

Do i need a robots.txt? Do i need a robots.txt?

Enhancing 80 million pages.

 Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Reply
Page 1 of 2 1 2

 

LinkBack Thread Tools
  #1  
Old 01-17-2008, 09:06 PM
Member
  
Real Name: Sean
Join Date: Jan 2008
Posts: 31
Do i need a robots.txt?
I'm just wondering, will it help or hurt my site? Do i really need it?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #2  
Old 01-17-2008, 09:18 PM
Senior Member
  
Real Name: dave
Join Date: Jun 2006
Posts: 346
Blog Entries: 1
It'll help reduce duplicate content from my understanding.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #3  
Old 01-17-2008, 09:26 PM
Member
  
Real Name: Sean
Join Date: Jan 2008
Posts: 31
is duplicate content a bad thing?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #4  
Old 01-17-2008, 09:36 PM
Michael Biddle's Avatar
vBSEO Staff
vBSEO Total Customer Support
  
Real Name: Michael Biddle
Join Date: Jan 2007
Location: Southern California
Posts: 3,003
Blog Entries: 5
Yes, very much so.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #5  
Old 01-17-2008, 09:47 PM
Member
  
Real Name: Sean
Join Date: Jan 2008
Posts: 31
ok so when i go to create one, what is the very best thing to put in the txt file itself so that i can just c/p what the masters have put in theirs? :P
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #6  
Old 01-17-2008, 10:30 PM
Member
  
Real Name: Sean
Join Date: Jan 2008
Posts: 31
i found a post from the dude that made like 1400 in a day or whatever and put this in my txt:

Quote:
User-agent: *
Disallow: /admincp/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /subscription.php
Disallow: /payments.php
Disallow: /profile.php
Disallow: /faq.php
Disallow: /calendar.php
Disallow: /search.php
Disallow: /private.php
Disallow: /online.php
Disallow: /sendmessage.php
Disallow: /sendmessage.php?do=
Disallow: /showgroups.php
Disallow: /reputation.php
Disallow: /report.php
Disallow: /threadrate.php
Disallow: /postings.php
Disallow: /newthread.php
Disallow: /newreply.php
Disallow: /register.php
Disallow: /login.php
Disallow: /faq.php
Disallow: /image.php
Disallow: /cron.php
Disallow: /joinrequests.php
Disallow: /printthread.php
Disallow: /showpost.php
Disallow: /archive/
now just waiting for google to update it's 404 message.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #7  
Old 01-18-2008, 11:53 AM
curriertech's Avatar
Senior Member
  
Real Name: Josh
Join Date: Feb 2006
Location: NH
Posts: 101
IMO the main purpose of robots.txt is to keep spiders from indexing content that doesn't matter, leaving them with more time to index the content that does matter. It's about indexing efficiency more than anything, but restricting them from showpost.php and /archive/ does reduce duplicate content and strengthens the validity of your indexed pages by essentially only indexing whole threads in your forum content.

I don't think any of this would have much impact on the AdSense spider though. It's going to hit new pages and index them regardless of what other pages it's trying to look at, because it's called by the ad script as a user loads that page, I think.

Your mileage may vary.
__________________
-Josh
www.nestreetriders.com
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #8  
Old 02-02-2008, 08:50 AM
Member
  
Real Name: Chris
Join Date: Dec 2006
Posts: 42
I'm thinking of copying seangworld's list in my own robots.txt file. Anyone here not think that's a good idea? I don't know much about this kind of thing so I'm looking for advice.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #9  
Old 02-02-2008, 01:08 PM
REVHEAD's Avatar
Member
  
Real Name: David
Join Date: Jan 2008
Posts: 72
The one I found is differant -
Code:
[User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/customprofilepics/
Disallow: /forum/images/
Disallow: /forum/modcp/
Disallow: /forum/ajax.php
Disallow: /forum/attachment.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.php
Disallow: /forum/image.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php
Can we have a definitive robots text from some one high up here please, they all seem to be differant.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #10  
Old 02-04-2008, 05:50 PM
briansol's Avatar
Senior Member
vBSEO Pre-Release TeamDesign for SEOBig Board Administrator
  
Real Name: Brian
Join Date: Apr 2006
Location: Central CT, USA
Posts: 5,534
robots.tx ONLY works in root.

if your site is in /forums, you should use the 2nd version with the /forums/ directive
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #11  
Old 02-04-2008, 06:00 PM
briansol's Avatar
Senior Member
vBSEO Pre-Release TeamDesign for SEOBig Board Administrator
  
Real Name: Brian
Join Date: Apr 2006
Location: Central CT, USA
Posts: 5,534
Quote:
Originally Posted by curriertech View Post
I don't think any of this would have much impact on the AdSense spider though. It's going to hit new pages and index them regardless of what other pages it's trying to look at, because it's called by the ad script as a user loads that page, I think.
The adsense spider is a totally different bot/service from the indexer.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #12  
Old 02-05-2008, 12:27 AM
Member
  
Real Name: Sean
Join Date: Jan 2008
Posts: 31
grr, that makes sense. i believe i used the first one.
correcting this now...

i took out 2 things from it tho: the poll and profile.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #13  
Old 02-05-2008, 12:59 AM
REVHEAD's Avatar
Member
  
Real Name: David
Join Date: Jan 2008
Posts: 72
thanks guys
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #14  
Old 02-06-2008, 05:12 PM
curriertech's Avatar
Senior Member
  
Real Name: Josh
Join Date: Feb 2006
Location: NH
Posts: 101
Quote:
Originally Posted by briansol View Post
The adsense spider is a totally different bot/service from the indexer.
Yep, I was just saying that with that one there's no delay in 'crawling' since it's called by the ad script. So is the adsense spider not governed by robots.txt at all?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #15  
Old 02-06-2008, 06:59 PM
Member
  
Real Name: Hendrik Kleinwaechter
Join Date: Jun 2007
Posts: 98
Blog Entries: 4
Quote:
Originally Posted by seangworld View Post
i found a post from the dude that made like 1400 in a day or whatever and put this in my txt:



now just waiting for google to update it's 404 message.
That guy is forbidding the archive? That sounds very stupid to me...
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
Reply
Page 1 of 2 1 2

« vBulletin Google Adsense Deep Integration Tutorial | Your highest CTR locations + styles? »
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads

Thread Thread Starter Forum Replies Last Post
robots.txt vBSEO Blogger Member Blogs 0 01-06-2008 12:40 PM
The best robots.txt MaestroX General Discussion 1 01-28-2007 05:53 PM
No Robots.txt friendly Troubleshooting 1 10-26-2006 11:30 AM
Robots.txt Help soletrader General Discussion 7 10-13-2006 10:31 AM
How is my robots.txt? libertylounge Troubleshooting 6 08-26-2006 01:17 PM


All times are GMT -4. The time now is 03:18 PM.

Contact Us - vBSEO Home - Site Map - Top

Powered by vBulletin Version 3.8.0 Beta 4
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.