URL's not followed... possible errors in redirect..

**Sonnie** · 09-19-2006 06:17 PM

Checking the Google Webmaster Tools it appears I may possible have a Redirect error...

This page lists URLs from your site that Googlebot had trouble crawling. Googlebot found these pages either in your Sitemap or by following links from other pages during a discovery crawl. Choose a category of errors to view:

URLs not followed (4726)

Here are just a sample of the 4726 URL's listed... they appear to be valid links when I click on them...

Code:

 http://www.hometheatershack.com/forums/archive/index.php/t-1001.html  	 Redirect error   [?]  	 Sep 14, 2006
http://www.hometheatershack.com/forums/archive/index.php/t-1002.html 	Redirect error   [?] 	Sep 13, 2006
http://www.hometheatershack.com/forums/archive/index.php/t-1003.html 	Redirect error   [?] 	Sep 15, 2006
http://www.hometheatershack.com/forums/archive/index.php/t-1006.html 	Redirect error   [?] 	Sep 15, 2006

**Oleg Ignatiuk** · 09-19-2006 08:25 PM

Strange.. the redirect looks fine for me. Do you see this for a long time?

**Sonnie** · 09-19-2006 09:35 PM

I don't understand what you mean by "a long time". I go to Google Webmaster Tools and it gives me a report of all the pages crawled at my site. Here's the latest report:

HTTP errors (0)
Not found (9)
URLs not followed (4845)
URLs restricted by robots.txt (33)
URLs timed out (0)
Unreachable URLs (14)

I can click on any of those and see the specific URL's. There just happens to be a bunch of them under "URLs not followed".

The report period begins on September 4, 2006. There are URL's from yesterday all the way back to that date.

Below is the explanations it gives:

This category lists URLs that we were unable to completely follow, along with some information as to why. In general, keep in the mind the following to ensure Google's spiders can follow links in your site:

* Use a text browser such as Lynx to examine your site, since many search engines see your site much as Lynx would. If features such as Javascript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.
* Don't use &ID= as a parameter in your URLs.
* If you use dynamic pages (for instance, the URL contains a ? character), be aware that not all search engine spiders crawl dynamic and static pages. It helps to keep the parameters short and the number of them few.
* If you are permanently redirecting from one page to another, use a permanent redirect (301).
* Where possible, use absolute rather than relative links. (For instance, when linking to another page in your site, link to www.example.com/mypage.html rather than simply mypage.html).

URLs not followed /Redirect error

Google was unable to completely follow the redirects from this page. Our guidelines suggest that you make every page on your site reachable from at least one static text link, so that we can access these pages easily. Below are some suggestions about using redirects:

* Minimize the number of redirects needed to follow a link from one page to another.
* Make sure your redirect timer is set for a relatively short period of time.
* Avoid using meta refreshes in the <head> statement of your pages.

That's a lot of pages that are not getting indexed... no wonder I don't have anymore pages on Google than I do.

**Oleg Ignatiuk** · 09-20-2006 08:41 AM

I don't understand what you mean by "a long time". I go to Google Webmaster Tools and it gives me a report of all the pages crawled at my site. Here's the latest report:

I meant whether you saw these errors before or not. Also, do you see similar errors for other (non-archive) links?

**Sonnie** · 09-20-2006 09:23 AM

This is the first time I've used the Google Webmaster Tools... but I know that at least go back to September 4.

They are not just the archives... here are a few more samples:

**Oleg Ignatiuk** · 09-20-2006 06:43 PM

Mm.. may be there were some server issues at your host?
Did anyone else experience this problem?

**Sonnie** · 09-24-2006 05:13 PM

I recently changed host since getting these errors and I'm still getting them...

URLs not followed (4966)

This has to be an issue with vBSEO. Do I need to open a support ticket?

**Juan Muriente** · 09-26-2006 12:29 AM

Hello Sonnie,

I've been researching further upon this issue and was unable to find a link between the rewrites and the error reported. At this point, I suspect it might be a bug with google webmaster tools. You might want to report it via their webmaster group for further feedback from Google (Google Groups: Google Webmaster Help)

Please keep us updated on the outcome, thanks-you.

**Sonnie** · 09-26-2006 03:01 AM

Have you checked this site (vbseo.com) to see if you have URL's not followed due to rewrite errors?

Wouldn't it make sense that if it was a Google Webmaster Tool bug that it would be happening to more than just my site?

I'm heard or seen of no one else having this problem, except me.

I opened a ticket on this over 24 hours ago and no one has responded to that ticket. No one has offered to look at my vbseocp.php and/or .htaccess files.

I'm curious as to how you could have researched this issue thuroughly.

**Keith Cohen** · 09-26-2006 11:05 AM

If the links works when clicked, then vBSEO is handling it correctly. It only makes sense that it's some issue with Google's tool. Are they continuing to show up, or are they all old dates like the samples posted?

**Juan Muriente** · 09-26-2006 11:12 AM

We are not showing this for vbseo.com - I'll post in the usergroup and see what they have to say.

**Sonnie** · 09-26-2006 05:52 PM

Originally Posted by Keith Cohen

Are they continuing to show up, or are they all old dates like the samples posted?

The last ones to show are from 9/20/06 which appears to be the last time the Googlebot crawled our site, which doesn't make sense.

Googlebot last successfully accessed your home page on Sep 20, 2006 .
We may not know about all the pages of your site. Submit a Sitemap to tell us more about your site.

It looks like it's not getting the sitemap, but I get an email everyday that says the sitemap was sent to Google and the ping was sucessful. Google spiders are on our site daily.

All of the problem URL's appear to be from the public_html/forums directory ... none are from the public_html.

Here's the .htaccess file in the /forums/ directory:

RewriteEngine On

RewriteCond %{HTTP_HOST} ^hometheatershack.com
RewriteCond %{REQUEST_URI} ^/forums/
RewriteRule (.*) http://www.hometheatershack.com/forums/$1 [R=301,L]

RewriteRule ^((urllist|sitemap).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

RewriteCond %{REQUEST_FILENAME} -f
RewriteCond %{REQUEST_FILENAME} (admincp|modcp|clientscript|cpstyles|images)/

RewriteRule ^(.*)$ $1 [L]
RewriteCond %{QUERY_STRING} !vbseourl=
RewriteCond %{REQUEST_FILENAME} !chat
RewriteRule ^(.*\.php)$ vbseo.php?vbseourl=$1&%{QUERY_STRING} [L]

RewriteRule ^(archive/.*)$ vbseo.php?vbseourl=$1&%{QUERY_STRING} [L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)$ vbseo.php?vbseourl=$1&%{QUERY_STRING} [L]

I have also posted in the Google Group... last night, but no helpful response as of yet.

**Keith Cohen** · 09-26-2006 05:59 PM

Do you have your sitemaps added to your webmaster account? That should show if the sitemaps have been downloaded. Also, the Download Log in vBSEO Sitemap should show Google hitting all your individual sitemaps.

**Sonnie** · 09-26-2006 06:19 PM

This is on the Download Sitemap Log for the last three days:

557 2006-09-24 19:46 sitemap_index.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

558 19:49 sitemap_1.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

559 2006-09-25 00:49 sitemap_1.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

560 00:49 sitemap_index.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

561 01:09 sitemap_1.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

562 16:15 sitemap.txt.gz Yahoo 74.6.72.233 Mozilla/5.0 (compatible; Yahoo! Slurp; Yahoo! Help -) Remove record

563 17:03 sitemap_index.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

564 17:23 sitemap_1.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

565 2006-09-26 00:49 sitemap_1.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

566 00:49 sitemap_index.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

567 01:14 sitemap_1.xml.gz Googlebot 66.249.72.45 Mozilla/5.0 (compatible; Googlebot/2.1; +Webmaster Help Center) Remove record

**Sonnie** · 09-26-2006 06:28 PM

There are no sitemaps listed on the Webmaster Account. My individual sites are listed but not sitemaps.

Do I need to add a sitemap? I didn't think this was necessary.

URL's not followed... possible errors in redirect..

LinkBack

Thread Tools

Display

URL's not followed... possible errors in redirect..

Similar Threads

Sitemap generating a lot more urls

Google Sitemap-Listing of old url's option-from 1.7

Redirect old URLs to index (phpBB)

Best way to redirect when moving URLS?

Posting Permissions