vBulletin SEO Forums

SEO

vBulletin Search Engine Optimization

Buy vBSEO Now! HACKER SAFE certified sites prevent over 99.9% of hacker crime.
ne nw
vBSEO Total Support Team Launches DeskPro New vBSEO Discount Level for Network Builders vBSEO 3.2.0 GOLD Has Landed Success with vBSEO = 600ore Web Visitors + $1400 in a Day! Crawlability Inc. Files for SEO Technology Patent
se sw

bot dont read my robots.txt

This is a discussion on bot dont read my robots.txt within the General Discussion forums, part of the vBulletin SEO Discussion category; I insert robot.text in root but bots still go to places i dont want them . like member list ,profilesand ...

Go Back   vBulletin SEO Forums > vBulletin SEO Discussion > General Discussion

bot dont read my robots.txt bot dont read my robots.txt

Enhancing 80 million pages.

 Register FAQ Members List Social Groups Calendar Search Today's Posts Mark Forums Read

Reply

 

LinkBack Thread Tools
  #1  
Old 07-29-2008, 11:20 AM
Member
  
Real Name: RAFAEL
Join Date: Sep 2005
Posts: 33
bot dont read my robots.txt
I insert robot.text in root but bots still go to places i dont want them .
like member list ,profilesand register

my robot.txt

Code:
 
User-agent: *
Disallow: forums/referrers.php
Disallow: forums/ajax_cron.php
Disallow: forums/ajax.php
Disallow: forums/attachment.php
Disallow: forums/calendar.php
Disallow: forums/cron.php
Disallow: forums/editpost.php
Disallow: forums/global.php
Disallow: forums/image.php
Disallow: forums/inlinemod.php
Disallow: forums/joinrequests.php
Disallow: forums/login.php
Disallow: forums/member.php
Disallow: forums/memberlist.php
Disallow: forums/misc.php
Disallow: forums/moderator.php
Disallow: forums/newattachment.php
Disallow: forums/newreply.php
Disallow: forums/newthread.php
Disallow: forums/online.php
Disallow: forums/poll.php
Disallow: forums/postings.php
Disallow: forums/printthread.php
Disallow: forums/private.php
Disallow: forums/profile.php
Disallow: forums/register.php
Disallow: forums/report.php
Disallow: forums/reputation.php
Disallow: forums/search.php
Disallow: forums/sendmessage.php
Disallow: forums/showgroups.php
Disallow: forums/subscription.php
Disallow: forums/threadrate.php
Disallow: forums/usercp.php
Disallow: forums/usernote.php
Disallow: forums/memberforums/
Disallow: forums/membersforums/
Disallow: forums/tagsforums/
Sitemap:http://www.fta-gods.com/forums/sitemap_index.xml.gz
Attached Thumbnails
spider.jpg  
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #2  
Old 07-29-2008, 11:33 AM
Oleg Ignatiuk's Avatar
vBSEO Staff
vBSEO Total Customer SupportvBSEO Documenter
  
Real Name: Oleg Ignatiuk
Join Date: Jun 2005
Location: Belarus
Posts: 21,581
Hello,

you should have a leading slash character in disallow lines, like:
Code:
Disallow: /forums/referrers.php
Also, you have your member list URLs rewritten, so you should use this:
Code:
Disallow: /forums/members/list/
__________________
Oleg Ignatiuk / Crawlability Inc.
Support Team Launches New DeskPro Powered Tool Enhanced Support at Your Service

vBSEO 3.2.0 Launched - Maximum Overdrive for Your Web Traffic! Over 100 Instant SEO Optimizations

6X Traffic - $1400 in One Day with vBSEO! Imagine What the vBSEO Patent Pending Technology Can Do For You.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
  #3  
Old 07-29-2008, 11:41 AM
Member
  
Real Name: RAFAEL
Join Date: Sep 2005
Posts: 33
thanks for reply
now i reupload bot.txt to root . i forgot to metion my forum is on folder forums .

now my robot.txt is :

Code:
 
 
User-agent: *
Disallow: /forums/referrers.php
Disallow: /forums/ajax_cron.php
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/member/forums/
Disallow: /forums/members/forums/
Disallow: /forums/tags/forums/
Disallow: /forums/members/list/
Sitemap:http://www.fta-gods.com/forums/sitemap_index.xml.gz
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!Share on Facebook!
Reply With Quote
Reply

« Installing vBseo in a subfolder doesn't work | Getting rid of profile pages in SE Index »
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads

Thread Thread Starter Forum Replies Last Post
I dont think Google can read my sitemap... beansbaxter General Discussion 9 05-16-2008 01:56 PM
Ive done something wrong dont know what tho Benj Troubleshooting 1 11-22-2007 12:06 PM
Can robots read if forum is blocked? majordude General Discussion 4 08-27-2006 05:59 AM
i dont understand this PR ? H2k Off-Topic & Chit Chat 2 12-19-2005 07:55 AM
How come you dont msimonds Off-Topic & Chit Chat 15 11-30-2005 04:43 PM


All times are GMT -4. The time now is 11:15 AM.

Contact Us - vBSEO Home - Site Map - Top

Powered by vBulletin Version 3.8.0 Beta 4
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.5 ©2008, Crawlability, Inc.