Results 1 to 6 of 6

Indexed URLS only about 10% of submitted URLS

This is a discussion on Indexed URLS only about 10% of submitted URLS within the General Discussion forums, part of the vBSEO Google/Yahoo Sitemap category; For the many weeks I have had vbseo and the sitemap plugin installed , and Google are only seeming to ...

  1. #1
    Member
    Real Name
    Alex D.
    Join Date
    Oct 2009
    Location
    Toronto, Ontario, Canada
    Posts
    41
    Liked
    0 times

    Indexed URLS only about 10% of submitted URLS

    For the many weeks I have had vbseo and the sitemap plugin installed , and Google are only seeming to index about 10% of my submitted sitemap:

    Sitemap Index Total: 279,613, Indexed: 25,320

    Sometimes it goes up to 27,000, but not more than that so far.

    Why is that?

  2. #2
    Senior Member
    Real Name
    Ceri May
    Join Date
    Jul 2009
    Location
    United Kingdom
    Posts
    1,726
    Liked
    15 times
    Blog Entries
    1
    The indexing can take anywhere from a few weeks to months before most of your links are indexed, I understand you have been running a few weeks but there are many factors into how quickly your site is indexed.

    Assuming that it is not just a time issue do you have 279,613 threads or are you also including your showpost links in the sitemap? Google will tend to ignore showpost pages as the content is generally duplicated.

    I would recommend submitting a "Critique Please" thread to see if there are any tips we can offer to increase your Crawl efficiency and Optimisation though just in case there is something to look at.

    Ceri

  3. #3
    vBSEO Staff Brian Cummiskey's Avatar
    Real Name
    Brian Cummiskey
    Join Date
    Jul 2009
    Location
    btwn NYC and Boston
    Posts
    12,782
    Liked
    648 times
    Blog Entries
    2
    Is a lot of the 279k bloat?

    member pages, tags, etc likely won't ever be a SERP and are best left out of the sitemap.
    Brian Cummiskey / Crawlability Inc.
    vBSEO 3.6.0 GOLD Released!
    Unveiling the NEW vBSEO Sitemap Generator 3.0. - available NOW for vBSEO Customers!


  4. #4
    Member
    Real Name
    Alex D.
    Join Date
    Oct 2009
    Location
    Toronto, Ontario, Canada
    Posts
    41
    Liked
    0 times
    We have like 5 million posts on the site. Showpost is not included in the 279k, I have disabled including that in the sitemap control panel. Also I have a robots text file that should keep the bots out of the following:

    User-agent: Mediapartners-Google*
    Disallow:

    Allow: /

    User-agent: *
    Disallow: /board/admincp/
    Disallow: /board/backup/
    Disallow: /board/calendar.php
    Disallow: /board/cron.php
    Disallow: /board/editpost.php
    Disallow: /board/joinrequests.php
    Disallow: /board/group.php
    Disallow: /board/login.php
    Disallow: /board/member.php
    Disallow: /board/misc.php
    Disallow: /board/modcp/
    Disallow: /board/moderator.php
    Disallow: /board/newreply.php
    Disallow: /board/newthread.php
    Disallow: /board/online.php
    Disallow: /board/printthread.php
    Disallow: /board/private.php
    Disallow: /board/profile.php
    Disallow: /board/register.php
    Disallow: /board/search.php
    Disallow: /board/sendmessage.php
    Disallow: /board/showgroups.php
    Disallow: /board/showpost.php
    Disallow: /board/subscription.php
    Disallow: /board/subscriptions.php
    Disallow: /board/threadrate.php
    Disallow: /board/usercp.php

    User-Agent: msnbot
    Crawl-Delay: 10

    User-Agent: Slurp
    Crawl-Delay: 10

    Sitemap: http://www.tribemagazine.com/board/sitemap_index.xml.gz

  5. #5
    Senior Member
    Real Name
    Marco Mamdouh
    Join Date
    May 2008
    Location
    Egypt
    Posts
    2,627
    Liked
    1 times
    Blog Entries
    1
    Google will never index all of the submitted URLs in sitemap, That's will increase by the time but it will never index the all URLs in your sitemap.
    Just include only showthreads and forumdisplay pages.

  6. #6
    Member
    Real Name
    Adam
    Join Date
    Aug 2009
    Posts
    37
    Liked
    0 times
    I've been including everything in my sitemap for months and I found out it was useless.

    As marco1 said above me, you better leave only showthread and forumdisplay in your board as these are the links which will be given a higher priority from Google.

Similar Threads

  1. Urls indexed
    By djilou in forum General Discussion
    Replies: 1
    Last Post: 11-08-2009, 08:01 PM
  2. url submitted different with my sitemap urls
    By ayah in forum General Discussion
    Replies: 1
    Last Post: 10-20-2009, 04:28 AM
  3. sitemap URLs vs indexed URLs in Google for a new site
    By jola in forum General Discussion
    Replies: 18
    Last Post: 10-01-2009, 10:59 AM
  4. URLs submitted?
    By doubler0204 in forum General Discussion
    Replies: 12
    Last Post: 08-27-2009, 09:08 PM
  5. Still no urls indexed . . .
    By smaddle in forum General Discussion
    Replies: 6
    Last Post: 12-05-2008, 05:54 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •