Google Spidering: Too much of a coincidence to not be true...

I've been wondering what in the HECK has caused Google to stop spidering my site. It's been a LONG time since it has come around. (Before I installed vbSEO.) I've getting eaten up by all the others, and my Google Sitemaps are even being downloaded regularly. Still nothing.

So, today, out of desperation I decided to remove my EMPTY robots.txt file from my site. (I added an empty/dummy one a while back because I was tired of seeing the File Not Found errors in my error log.)

Now, I'm starting to see Google show up in my vbSEO bot stats again.

I can't believe than an empty robots.txt would cause that, but it sure looks that way. I could be wrong, but I'll keep an eye on it now to see.

(By the way, looking at the Errors tab in my Google Webmaster account, it shows nothing in the list for "URLs Skipped Due To robots.txt.)

I may have spoke too soon. My new vbSEO Bot Stats only show 35 "hits" from Google, then it went away again. We shall see....

It's not logical that G would stop spidering your site if you have a blank robots.txt. Although you *should* add at least

User-agent: *

into it.

Do we need to add robots.txt in order for google spider our site ? because i dont have it.. i give google a free time to spider any page in our site

Originally Posted by H2k

Do we need to add robots.txt in order for google spider our site ? because i dont have it.. i give google a free time to spider any page in our site

If you do not have one they will still spider your site. Mine consists of the following:

User-agent: *
Disallow: /forum/editpost.php
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/report.php
Disallow: /forum/sendtofriend.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/a69er/
Disallow: /forum/modcp/
Disallow: /forum/sendmessage.php
Disallow: /forum/subscription.php

I dont have any robots.txt, or meta tags relating to spiders.

google bot is not scouring anywhere from 500 to 5000 pages of my site every day, I always have an msn bot online, and ususally 20 or more yahooslurps.

The vbSEO Sitemap really made a difference for me. It brought Google back.

Report attached.

Good news ConqSoft I've been wondering about this situation. Probably just an anomaly.

Keep us posted.