Moved Forum to Subdomain - Googlebot Not Redirecting?

[esther196]

Last night, I moved our forum from a directory to a subdomain using 301s. Today, I see Googlebot hitting the old folder and being redirected, but I don't see the related entry on the subdomain. When I test the redirect on web-sniffer.net, everything appears correct.

Old URL: www. example . com / forum
New URL: forum.example.com/t-forum-post-123.html

Web-sniffer.net shows that the old URL is permanently redirected to the new URL, and the new URL returns a 200.

But, in the subdomain's access_log, I don't see the redirected entry (there have been hundreds of hits on our old domain). In fact, I only see about 5 Googlebot entries and they're all hitting showthread.php (rather than the vbseo URLs).

Any help at all will be greatly appreciated!!

[Brian Cummiskey]

Have you re-run your sitemap to be at the new url? And adjusted your webmaster tools areas? It can take a bit for google to understand the change and start re-indexing you. Any move, regardless of 301s, should expcet a 4-8 week 'bad' period from the bots while they adjust.

[esther196]

Yes, I've re-run my sitemap and verified that it includes URLs to the new subdomain. And I've added the subdomain to GWT and submitted the new sitemaps.

I understand it can take a while for Googlebot to re-index my entire forum, but something seems wrong because shouldn't I be able to see that Google was correctly redirected to the subdomain after it hits the old URL?

[esther196]

Just checked my access_log again and Googlebot is only hitting a few showthread.php pages. I haven't seen this before especially since installing vbseo; I'd expect it only to crawl the vbseo-ed URLs.

Thanks to anyone for any help!

[Brian Cummiskey]

It looks like google has alraedy started to index the new subdomain.
I wouldn't worry about what you are seeing and just let it roll for a few days.

[esther196]

Something just wasn't sitting right with me, and I finally figured it out. Hopefully this will help someone else out too.

I found it odd that Googlebot was grabbing my robots.txt and sitemaps yet showed 0 pages indexed in GWT (despite showing about 50 when using the site: operator). Also odd was the Googlebot wasn't crawling any rewritten URLs according to my access_log.

I finally used the robots.txt analysis tool in GWT against one of my forum URLs and realized the robots.txt was disallowing all *.htm files! I deleted that disallow and now I expect to see pages crawled and indexed.

I don't know where this robots.txt file came from. I assumed it was the standard one that came with vBulletin but maybe not.

[Mert Gökçeimam]

vBulletin or vBSEO doesn't include any robots.txt file at all.