Small number of indexed URLs

**infobeto** · 02-03-2010 03:04 AM

Hi everyone.

New to vbseo, I just installed yesterday.

The first time I generated the google sitemap and submitted it to webmaster tools I got around 40,000 URLs and about 7,000 indexed. That was yesterday...

Today, I saw in the webmaster tools about the same amount of URLs processed but only 1!!!! indexed.

What may have gone wrong?

Thank you.

**infobeto** · 02-03-2010 03:36 AM

Another question I have...

I'm using two sitemaps. One for joomla which is in root and one for vbulletin which is in /forum/ directory.
I have created a robots.txt file for the root to disallow folders from joomla I don't want to be crawled. Do I need to create another robots.txt file inside /forum/ directory to disallow folders from vbulletin? Or just update the one I have in root?

Another thing (last I promise

) is that I have a lot of folders in root which are old and I don't longer use. Are bots going to try to index those as well? Should I remove them?

Thank you.

**Ceri May** · 02-03-2010 06:54 AM

Originally Posted by infobeto

Hi everyone.

New to vbseo, I just installed yesterday.

The first time I generated the google sitemap and submitted it to webmaster tools I got around 40,000 URLs and about 7,000 indexed. That was yesterday...

Today, I saw in the webmaster tools about the same amount of URLs processed but only 1!!!! indexed.

What may have gone wrong?

Thank you.

Have you made any changes to your URL structure since yesterday? Could you post a link to your sitemap so I can check for errors.

Originally Posted by infobeto

Another question I have...

I'm using two sitemaps. One for joomla which is in root and one for vbulletin which is in /forum/ directory.
I have created a robots.txt file for the root to disallow folders from joomla I don't want to be crawled. Do I need to create another robots.txt file inside /forum/ directory to disallow folders from vbulletin? Or just update the one I have in root?

Another thing (last I promise

) is that I have a lot of folders in root which are old and I don't longer use. Are bots going to try to index those as well? Should I remove them?

Thank you.

Robots.txt is only valid in root so you need to update the one in root and place full relative URL's so in otherwords include the /forum/ folder in all links.

Bots are known to search for random dictionary words from time to time to see if they find new pages so I would suggest removing unused folders, plus it makes server management much easier when everything is neat and tidy too...

**infobeto** · 02-03-2010 07:01 AM

Here is my sitemap

Please delete the link when you get it.

As far I remember I did not change anything in the URL structure after I submitted the sitemap to google. I did that before and the forum was closed to users.

Any suggestions on what to exclude from the forum in robots.txt?

Thank you.

**Ceri May** · 02-03-2010 07:15 AM

Link Deleted - Thank you but I didn't find any problems.

You may have submitted after you had made the URL change but had the sitemap regenerated yet? The sitemap is generated once a day so it is possible that when you submitted the sitemap it was all of the Old format then the sitemap was re-generated and so next time google looks all the URL's have changed. < This is speculation Just watch the Indexed count over next few days and it should grow also look out for errors in Webmaster tools.

What you disallow from your Robots.txt is up to you I personally use

Disallow: /forums/*-print/
Disallow: /forums/vbseocp.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/newthread.php
Disallow: /forums/newreply.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php

Ceri

**infobeto** · 02-03-2010 07:30 AM

I'm pretty sure I generated the sitemap manually as I had 7000 pages indexed the first time I used vbseo. Then I left it like that and the next day it was just 1 page.

Strange....

In Webmaster Tools I don't really see any errors.

Thank you for your help.

**infobeto** · 02-03-2010 08:20 AM

I think I forgot to mention something...

I used to generate google sitemap prior to vbseo installation. So google has indexed my pages in the old format (without seo). Does this have to do with my problem?

**Mert Gökçeimam** · 02-03-2010 09:10 AM

You may want to request a site critique. I checked your site and noticed couple flaws like

Too many affliate banners before page content
Relevant Replacements missing
Your pages are not validated
Some of your forums are not open to public , as you know non public forums can't be indexed by SE's

**infobeto** · 02-03-2010 09:18 AM

Oh my god. Too many W3C errors. Does this affect indexing?

I don't know if I am going to gain anything from relevant replacements since the forum language is in greek and so urls are not in native english (they don't make much sense...)

Banners are also important to us. We can't remove them...

**Mert Gökçeimam** · 02-03-2010 10:01 AM

Relevant Replacements surely will make difference. My site is in Turkish but i am still using and benefiting from Relevant Replacements.

Yes validation is important as if page is not correctly coded , bots may have problem when visiting your pages.

**infobeto** · 02-04-2010 11:57 AM

I used a tool in google webmasters Fetch as Googlebot

and I was surprised to see that my forum is not being read by Googlebot!

Any ideas?

Here is what it is being read.

PHP Code:

  Date: Thu Feb 04 08:51:30 PST 2010
 HTTP/1.1 301 Moved Permanently
Date: Thu, 04 Feb 2010 16:42:09 GMT
Server: Apache/2.0.52 (Red Hat)
Location: http://www.infobeto.com/forum/
Content-Length: 321
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://www.infobeto.com/forum/">here</a>.</p>
<hr>
<address>Apache/2.0.52 (Red Hat) Server at www.infobeto.com Port 80</address>
</body></html>

**infobeto** · 02-04-2010 12:15 PM

The error above is when I use

PHP Code:

 http://www.infobeto.com/forum

When I use

PHP Code:

 http://www.infobeto.com/forum/

the page is being read.

Do you think that this has to do with the fact that I still have only 1 indexed page out of 39,000?

**Brian Cummiskey** · 02-04-2010 12:44 PM

Google doesn't like 301's. It wants you to send it to the Final 200 destination url.

Have you moved install folders? changed vbseo options to new url settings, etc? If you're submitting new things, the only file that will match the google index would be root. The rest would all be new different URL's.

**infobeto** · 02-04-2010 01:24 PM

Originally Posted by Brian Cummiskey

Google doesn't like 301's. It wants you to send it to the Final 200 destination url.

Have you moved install folders? changed vbseo options to new url settings, etc? If you're submitting new things, the only file that will match the google index would be root. The rest would all be new different URL's.

No, I haven't moved anything. But I used to sumbit a forum sitemap prior to VBSEO installation

**Brian Cummiskey** · 02-04-2010 01:28 PM

That makes sense then... it has 'showthread.php?t=123' in it's index, and you're sending them 'thread-123/' now.

Thus, only the homepage matches. It will take a bit for the old urls to get updated. Keep an eye on it daily, but don't expect to see a lot of movement in the first week or so.

Small number of indexed URLs

Small number of indexed URLs

Similar Threads

Number of indexed pages in Google

Number of indexed urls - is it low?

vBSEO tripled my small earnings and indexed my new URLs in just a week!

Number of indexed pages from 10k to 60k within 1 month