robots.txt

**firassalim** · 07-11-2009 02:36 AM

vbSEO installed just installed for me the package
am excited about it
however;
two puzzling things
there's no robots.txt file in my root folder?
also
I cannot see the sitemap file in my root directory using ftp access

Any ideas?

**Shadab** · 07-11-2009 03:40 AM

there's no robots.txt file in my root folder?

Not supplied with the vbseo package. It's optional.
You can create it if you need to block certain parts of the Forum from being indexed.

I cannot see the sitemap file in my root directory using ftp access

The sitemap files actually reside in the /vbseo_sitemap/data/ directory.

**Michael Biddle** · 07-11-2009 03:50 AM

Thanks Shadab!

I also answered your support ticket with about the same answers.

**Shadab** · 07-11-2009 04:41 AM

Originally Posted by Michael Biddle

Thanks Shadab!

You're welcome!

**firassalim** · 07-20-2009 05:37 PM

OK I've got Michael Biddle telling me to disallow *.php as vbSEO will rewrite all URLs in html, I did that
Now after few days what happens is that Google is indexing everything but not the forums index pages!
Also in my analytics I can see that the forum indexes are still index.php
Also when I go to mysite.com/forumid/index.html or /index.php both pages are not there
Please help me get Google bot index my forum indexes
Should I remove the robots.txt or should remove the line disallow *.php in it?

**firassalim** · 07-20-2009 05:44 PM

Please see my post below

**Shadab** · 07-20-2009 10:22 PM

@firassalim:

Where is your vBulletin installed ?
Is it directly installed in root, or in the /forums/ directory ?

**firassalim** · 07-20-2009 11:33 PM

Originally Posted by Shadab

@firassalim:

Where is your vBulletin installed ?
Is it directly installed in root, or in the /forums/ directory ?

in the root directory!
So silly to put /forums/ in the disallow!

**firassalim** · 07-20-2009 11:59 PM

Sorry there was a mistake in the robots.txt code that uploaded
here's the correct one

Code:

User-agent: *
Disallow: /admincp/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: calendar.php
Disallow: editpost.php
Disallow: faq.php
Disallow: global.php
Disallow: inlinemod.php
Disallow: joinrequests.php
Disallow: login.php
Disallow: member.php
Disallow: memberlist.php
Disallow: misc.php
Disallow: moderator.php
Disallow: newattachment.php
Disallow: newreply.php
Disallow: newthread.php
Disallow: online.php
Disallow: payments.php
Disallow: poll.php
Disallow: postings.php
Disallow: printthread.php
Disallow: private.php
Disallow: profile.php
Disallow: register.php
Disallow: report.php
Disallow: reputation.php
Disallow: search.php
Disallow: sendmessage.php
Disallow: sendmessage.php?do=
Disallow: showgroups.php
Disallow: subscription.php
Disallow: threadrate.php
Disallow: usercp.php
Disallow: usernote.php
Disallow: vbseocp.php
Sitemap: http://www.mysite.com/sitemap_index.xml.gz

Q1) Please advise whether this is a correct robots.txt to go with or not

Q2) If you notice that I disallowed member and memberlist because I don't to index members profiles
Now how should I tweak vbSEO not to include member profiles in the sitemap?

**Shadab** · 07-21-2009 12:27 AM

1. There's no point in listing vBulletin's core directories or includes files. This should be more than enough:

Code:

User-agent: *
Disallow: /calendar.php
Disallow: /misc.php
Disallow: /register.php
Disallow: /faq.php
Disallow: /login.php
Disallow: /search.php
Disallow: /sendmessage.php

Sitemap: http://www.mysite.com/sitemap_index.xml.gz

2a. If you want to disallow bots from indexing user profile pages,
disallow the 'rewritten' profile URLs; example: Disallow: /members/

2b. AdminCP » Sitemap Options ; or AdminCP » vBulletin Options » vBSEO Google/Yahoo Sitemap Generator
Then, set "Include Member Profile Page" to "No" and [Save]

**firassalim** · 07-21-2009 12:34 AM

Thanks Shadab

robots.txt

robots.txt

Similar Threads

Redirecting /forums/robots.txt to /robots.txt - Is it good?

My robots.txt - what is in there / what isn't there?

robots.txt

Robots.txt help

How is my robots.txt?