robots.txt

thanks, how does this look?

http://www.yoliverpool.com/robots.txt

For those of us who get excited about such things, it looks great.

Hello i would like to know if this robots.txt file it's good:

# Allow Archiver
User-agent: ia_archiver
Allow: /


User-agent: Slurp
Crawl-delay: 60



User-agent: Googlebot-Image
Allow: /*


User-agent: *

Allow: /gallery/*.gif$
Allow: /gallery/*.png$
Allow: /gallery/*.jpeg$
Allow: /gallery/*.jpg$
Allow: /gallery/*.jpg$
Allow: /gallery/images/



User-agent: Mediapartners-Google*



User-agent: *

Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /forum/*.php
Disallow: /forum/*.js
Disallow: /forum/*.jsp
Disallow: /forum/*.cfm
Disallow: /forum/*.asp
Disallow: /forum/*.html
Disallow: /forum/*.htm
Disallow: /forum/*.aspx
Disallow: /forum/*.cgi
Disallow: /forum/includes/
Disallow: /forum/install/
Disallow: /forum/customavatars/
Disallow: /forum/archive/
Disallow: /forum/sitemap/

because the validator says that i have alot errors

Thanks

Originally Posted by webwizzy

okk.. I need to know something related to this, so I'm posting it here.

I wanted to know:-

1. Disallowing means that the file/folder WILL NOT be indexed in the search engines, right ?

Thank you

I don't think this question got answered, and it's fairly important.

Not exactly. It means that your robots.txt is telling the crawlers where to go or not to go. Whether they follow that ruling is up to their coder. Certain bots will read your robots.txt and do exactly the opposite, because they want to go where you don't want them to. The only way to ENSURE that a certain page cannot be viewed by a bot is to .htaccess protect it.

This thread is 4 years old. Can we have an updated thread and maybe sticky with new robots.txt disallows which would include the vbseo's files?

Robots will be different for each website, what did you need help with?

Originally Posted by Michael Biddle

Robots will be different for each website, what did you need help with?

This is my robots.txt file and it doesn't prevent Google from spidering users' profiles. What am I missing? I have vbseo installed. Google keeps spidering mysite.com/members/myusername.html type urls.

Code:

User-agent: *
Disallow: /ajax.php
Disallow: /album.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /clientscript/
Disallow: /cron.php
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /includes/
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /posthistory.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /vbseo.php
Disallow: /vbseocp.php
Disallow: /vbseocpform.php

If you don't want bots to index your profiles make sure you don't include them to your sitemaps and disallow them within robotx.txt with the followind

Disallow: /members/

Originally Posted by Mert Gökçeimam

If you don't want bots to index your profiles make sure you don't include them to your sitemaps and disallow them within robotx.txt with the followind

Disallow: /members/

Great. Thanks for that. Do you know of any other directories I should disallow in robots, now with vbseo installed? Basically I'm including in the sitemaps only showthreads and forumdisplay as my forum is a bit large and I don't want to have Google crawling useless pages. What else should I disallow in the robots.txt file?

Groups : Disable it from sitemap generator settings and add this to robots.txt

Code:

Disallow: /groups/

What ever URLs you want to rewrite, just look into the URL of that page and it should be the part after your .com.

Example: yoursite.com/members/1-username.html

Done. Thanks guys.

Originally Posted by Michael Biddle

What ever URLs you want to rewrite, just look into the URL of that page and it should be the part after your .com.

Example: yoursite.com/members/1-username.html

Good point Mike that's will help the customers more

Originally Posted by wac

Done. Thanks guys.

You are welcome and check Michael's post it will help you

I was wondering if my robots.txt was optimized. - FYI I'm running vbadvanced 3.20 and vbulletin 3.8.4.

Code:

User-agent: *

Disallow: /vbseocp.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /newthread.php
Disallow: /newreply.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /search.php
Disallow: /sendmessage.php

Thanks

Originally Posted by AdamFL

I was wondering if my robots.txt was optimized. - FYI I'm running vbadvanced 3.20 and vbulletin 3.8.4.

Code:

User-agent: *

Disallow: /vbseocp.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /newthread.php
Disallow: /newreply.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /search.php
Disallow: /sendmessage.php

Thanks

Hello Adam,

What is the link to your board?