Google Crawling and Indexing

Hello,

I have been running a forum for about 6 years.
Some months ago i upgraded the board into Vbulletin software.
I have almost 1 milion page views per month and about 90.000 posts.

What is happening is that i only have about 2 or 3 pages indexed by google.
My board is open to guest's and i rarely see googlebots online crawling. Other like yahoo are always crawling my board.

Will VBSEO help me regarding this problem?
Thank you.

Well - it sounds like Google dropped you from the index after noticing the massive changes. This was to be expected. I suspect they will eventually find your vBulletin content again, but I wouldn't leave anything up to chance.

I believe vBSEO along with the vBSEO Google/Yahoo Sitemap will help you to finally get Googlebot to return to your site.

The best time to do this is now. Since Google does not currently have your forum indexed, it will not notice another change from vBulletin dynamic URLs to vBSEO static URLs. This is similar to the advantage that new forums have when they install vBSEO prior to launch.

vBSEO will help you to achieve link consensus, and therefore eliminate alot of unnecessary paths to the same (or duplicate) content. This will allow Google PR to be distributed better, helping to get you a higher crawling priority.

A good idea is to partake in link exchanges with other related and popular websites and message boards. Preferably those with a high indexation rate, this is because Google and other search engine robots will take note of outbound links on these pages, and index these. Thus decreasing the time required to get indexed in comparison the usual method of the Add URL submission form.

Right, link building is critical when it comes to SEO. Make sure you work on gaining links-in while having your on-page/site SEO optimized with vBSEO.

Q: What is your forum-root Google PR? We've definitely been able to identify a faster/deeper indexing trend associated to higher forum-root PR....

ps- Welcome to vBSEO

thank you for your replies.
I have been using google sitemap now and after making a search at google "link: www.myurl.com" i get 195 results of pages linking to my forums.

I cannot understand why google doesn't even want to look at my forums.

I'm always looking at who's online members, and i never see googlebot at my board (i only saw it once looking at my forum homepage and never more). Yahoo glurp spider is always at my board with 3/4 bots.

how can i guarantee google indexes my board? how can i make sure vbseo will help me?

I'm using also linkexchange.
thank you for your help.

You can never be sure about what google does but vBSeo is the best tool you can use

Also are you using spiders.xml ? If you are not maybe you don't know if google is crawling your site.

the link: command is unreliable, and it is not the tool for what you want to look up. link: shows sites that link to you, IF THEY are indexed. I wouldn't use this as a judge of your site. I have 0 link: results on some sites that have every page indexed.

the site: command is what you want to use to see how many SERP's you have.

didn't know about the spiders.xml file... installing now.

What happens with googlebot is that when it comes to the forum, it gets stucked in only one thread for 10 minutes and then i only see the bot a couple of weeks later doing the same thing: crawling one thread and going away.

I will install spiders.xml and probably install vbseo also and see results.
thank you all for helping

after installing spiders hack i can see googlebot is coming sometimes to my forums but never following links.

What is the best command search to list all urls of my domain at google?
allinurl: myurl.com ?
allinurl: site:myurl.com ?


It's strange because google has in cache pages crawled in May 2005 and never updated those pages. They are still in google's cache.

sorry being boring with all this questions, but i've been worried with this :s
Why googlebot never follow the links on my forums? Why it gets stucked in a thread/forum when it starts crawling the board? i see other forums and googlebot is always following the links....

do you have a robots.txt file?
a sitemap submitted to google daily?
vbseo installed?

1) robots.txt: yes (although, many sites say robots.txt is not necessary because normal procedure of searchbots is to follow all links)

2) Sitemaps submitted daily: yes (although many sites say sitemaps should be submitted weekly or even monthly avoiding being "banned" from search engines)

3) vbSEO: no (not yet, thinking if it would be worth to buy it or not...)


I remembered one thing now: could this problem be because using vbadvanced in homepage as www.mydomain.com/index.php and using www.mydomain.com/forum/ as my forum directory and not in the root?

1- right. but there are things you DONT want SE's to index to "pollute" your listings. if you do a search on here for robots.txt you will find many examples of files various members are excluding.

2- Google will take a new sitemap ever second if you wanted to. It won't get you very far, but it won't get you banned either. I run mine nightly at 4am EST.

3- what are you waiting for see my sig. it's been a great month.

3b- no, vbadvanced won't have ill effects, nor will 1 level deep. many vbseo users run vba as well

I'm getting really bored with this problem about google.
I think my domain should be banned from google, because pages never appear at google. I think VBSEO can't help me, because the problem should be with google itself.

Google only indexes my homepage and updates it almost daily.
Then, never follow all other links and also there are only 3 or 4 pages cached in may2005 at google search engine.

Should i change my domain name to another one?
I'm getting desperate with this...

I see other boards in my country with 50 or 100 search engine bots online at the same time, including googlebot always online...

Well, if you have not achieved any significant level of current indexing, then changing your domain might be an option. It would be good if you could get an after market domain that already has some history, rather than a brand new domain.

However, have you done anything in the past that might have gotten you penalized in Google? When was your domain first purchased and launched (could you be sandboxed?). Have you tried a Google re-inclusion request?