robots.txt

User-agent: *
Disallow:
I allow everything.

same here, put I am considering a new robot.txt see in this theard but for now everything.

I might add some disallows in once I get indexed really well in google, but for now I am letting the bots go everywhere.

Well the bots will be wasting their time and get a nice chunk of permission denied errors if you do not disallow newreply.php private.php etc etc

You can watch them via who is online and see the red stop sign next to the area they are trying to index

If they adhere to nofollow tags, they won't hit those anyway...

Originally Posted by ConqSoft

If they adhere to nofollow tags, they won't hit those anyway...

Yahoo apparently doesn't even though they say the do because I allowed the bots to go everywhere for one week banking only on the relnofollows and yahoo was still trying to index pms and reply to messages

Yep, seems that a lot of them ignore them.

Hi,

Can somebody please verify this is ok:

http://www.diskusije.net/robots.txt

Thank you!

Looks alright to me.

I thought there was so many php files in it.
Is google etc. going to be able to fully crawl stuff from my site?

Damir

That is preventing them from crawling irrelevant locations to your website. Ones in which they do not have access to see.

Hi guys ,I am running the latest Versions of VBulletin and VB Advanced , Downloads 2 mod and TV and radio mod, and of coarse VBseo, my robots text looks like this so far

HTML Code:

User-agent: *
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/faq.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/payments.php
Disallow: /forums/showpost.php
Disallow: /forums/showgroups.php
Disallow: /forums/spiders.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/admincp/
Disallow: /forums/includes/
Disallow: /forums/install/
Disallow: /forums/modcp/
Disallow: /forums/customavatars/
Disallow: /forums/xmlrpc.php
Disallow: /forums/archive/
Disallow: /forums/sitemap/
Disallow: /forums/donate.php /
Disallow: /gallery/member.php
Disallow: /gallery/showgallery.php?cat=
Disallow: /gallery/showgallery.php?limit=
Disallow: /gallery/comshow.php
Disallow: /gallery/search.php
Disallow: /gallery/*search.php
Disallow: /gallery/showframe.php
Disallow: /gallery/slideshow.php
Disallow: /gallery/*limit
Disallow: /gallery/*ppuser
Disallow: /gallery/*views
Disallow: /gallery/*date
Disallow: /gallery/*recent
Disallow: /gallery/*in
Disallow: /400.shtml
links/addlink.php
links/ajax.php
links/managecats.php
links/misc.php
links/moderate.php
links/newreply.php
links/search.php
Disallow: /403.shtml
Disallow: /404.shtml
Disallow: /500.shtml
Disallow: /favicon.ico

Should I be adding or removing anything to suit my new version Of Vbulletins?

Originally Posted by BamaStangGuy

Well the bots will be wasting their time and get a nice chunk of permission denied errors if you do not disallow newreply.php private.php etc etc

You can watch them via who is online and see the red stop sign next to the area they are trying to index

]


wow @ yours

nice work

That's the OCD in me lol

okk.. I need to know something related to this, so I'm posting it here.

I wanted to know:-

1. Disallowing means that the file/folder WILL NOT be indexed in the search engines, right ?

2. If I have groups and albums enabled for guests, so should I still disallow group.php and album.php ?

3. Also, if its okay to disallow announcement.php and poll.php ! Will it not block search engines to read polls and announcements ?

4. Lastly, is it okay to disallow the complete vbseo_sitemap folder and vbseo.php, vbseocp.php, vbseocpform.php files ?

Thank you