Understanding Spiders, vBSEO, and Sitemap

I guess I'm confused about how all this works. I've been trying to read up but still can't get a grasp on it and have a few questions.

I initially installed vBSEO Google/Yahoo Sitemap and then later installed vBSEO. I've had both installed for about a week. I have a vB Cron job setup to generate the sitemaps daily. Of course I understand the Google sitemap is automatically sent to them. I sent my first sitemap.txt.gz file to Yahoo today and immediately the Yahoo spiders started slaugtering our forum, which I suppose is a good thing. I see in the Who's Online we have about 4-5 Yahoo spiders but I have never seen more than one Google bot. I did notice that in my vBSEO Sitemap generator settings I didn't have "vBSEO installed" checked so I checked it.

Why is it that Google only sends one spider and Yahoo sends 4-5?

How can I setup a log to see what spiders are doing on our forum?

Is there anything else I need to do?

Many thanks!

Edit: How often should I upload the sitemap.txt.gz to Yahoo?

Okay... I figured out how to view my spider activity...

Is this good, bad, okay?

http://www.snapbug.ws/images/bots.gif

Any activity is good activity, however, this is a little slow. I wouldn't worry. It's just a waiting game. Expect solid 1-3 days where they start pulling down thousands of pages.

Originally Posted by Joe Ward

Any activity is good activity, however, this is a little slow. I wouldn't worry. It's just a waiting game. Expect solid 1-3 days where they start pulling down thousands of pages.

It depends how many posts, threads and members the board has. Let say it's a new board.... this would be excellent!

Sonnie, could you give us an idea of this info?

Home Theater Shack

Members: 636
Threads: 342
Posts: 2,332

We officially opened the doors on April 19th... so tomorrow will be 30 days.

Here's where we are at:

Threads: 342, Posts: 2,333, Members: 636 | Registered/Last 24hrs: 26

Activity continues to improve daily.

I have noticed that we went from about 50 pages on Google (site:Home Theater Shack ) to about 200 pages in the last few days.

But I just haven't noticed more than one Google Spider visiting the site at one time. He (or she) stays there pretty much 24/7 though. And as stated above... within minutes of uploading my sitemap to Yahoo... they busted loose on me.

Google would eat up all the content in next to no time.... once it has I would expect to see the levels just a little bit higher than what you are showing now.

One thing that is puzzling is that your robots.txt on the face of it exists, but spiders cannot retrieve it.

I'm really not sure how spiders would react to a permission denied message, or being told they must be logged in to see robots.txt

How can I fix that? I thought it was all setup right.

change permissions on the file I would guess.

You mean chmod it from 644 to 777 ?


EDIT: Would this have anything to do with it:

Index Manager

The index manager will allow you to turn indexes on or off for a directory. If indexes are off, people will not be able to see what files are in a directory that lacks an index file.

Please select the folder you wish to protect by clicking on its name. You can navigate by clicking on icons. (All are protected)

/ (Current Folder)
bfdguide
cgi-bin
forums
images
manufacturers
roomeq
winadvd


Sorry for all the questions guys... I'm trying to learn.

I was just looking at my server load and it has red flashing dot on it (all others are green):

Server Load 1.09 (2 cpus)

Is this bad? What does it mean?


EDIT: Hmmmm ... just like that it's at: Server Load 0.36 (2 cpus) with a green dot.

Hello Sonnie,

re: robots.txt

it shouldn't be related to the "Index Manager", but it seems like you have some sort of hotlinking protection used that may be related to this issue.

Enable Hotlink Protection:
Urls to Allow Access:
http:// www. hometheatershack.com
http:// hometheatershack.com
http:// www. snapbug.ws


Extensions to allow (seperate by commas): jpg,jpeg,gif,png,bmp,xls,cal,txt,mpeg,mp3

Url to Redirect to: http:// hometheatershack.com/display.html

Allow direct requests (ie. entering the url to an image in your browser): NO

Hotlink Protection: ENABLED

Having ".txt" files protected I suppose is redirecting Googlebot to the "/display.html" page. I suppose I need to take ".txt" out of the extensions to allow to protect.

Yes, you are right.