Page 1 of 2 1 2 LastLast
Results 1 to 15 of 17

Google Crawling and Indexing

This is a discussion on Google Crawling and Indexing within the Pre-Sales Questions forums, part of the Announcements & Pre-Sales category; Hello, I have been running a forum for about 6 years. Some months ago i upgraded the board into Vbulletin ...

  1. #1
    Junior Member
    Real Name
    GTX2
    Join Date
    Jul 2006
    Posts
    11
    Liked
    0 times

    Google Crawling and Indexing

    Hello,

    I have been running a forum for about 6 years.
    Some months ago i upgraded the board into Vbulletin software.
    I have almost 1 milion page views per month and about 90.000 posts.

    What is happening is that i only have about 2 or 3 pages indexed by google.
    My board is open to guest's and i rarely see googlebots online crawling. Other like yahoo are always crawling my board.

    Will VBSEO help me regarding this problem?
    Thank you.

  2. #2
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    36 times
    Blog Entries
    9
    Well - it sounds like Google dropped you from the index after noticing the massive changes. This was to be expected. I suspect they will eventually find your vBulletin content again, but I wouldn't leave anything up to chance.

    I believe vBSEO along with the vBSEO Google/Yahoo Sitemap will help you to finally get Googlebot to return to your site.

    The best time to do this is now. Since Google does not currently have your forum indexed, it will not notice another change from vBulletin dynamic URLs to vBSEO static URLs. This is similar to the advantage that new forums have when they install vBSEO prior to launch.

    vBSEO will help you to achieve link consensus, and therefore eliminate alot of unnecessary paths to the same (or duplicate) content. This will allow Google PR to be distributed better, helping to get you a higher crawling priority.

  3. #3
    Senior Member Michael's Avatar
    Real Name
    Michael Benson
    Join Date
    Sep 2005
    Location
    United Kingdom
    Posts
    776
    Liked
    0 times
    A good idea is to partake in link exchanges with other related and popular websites and message boards. Preferably those with a high indexation rate, this is because Google and other search engine robots will take note of outbound links on these pages, and index these. Thus decreasing the time required to get indexed in comparison the usual method of the Add URL submission form.

  4. #4
    vBSEO Staff Juan Muriente's Avatar
    Real Name
    Juan Carlos Muriente
    Join Date
    Jun 2005
    Location
    Puerto Rico
    Posts
    14,266
    Liked
    586 times
    Right, link building is critical when it comes to SEO. Make sure you work on gaining links-in while having your on-page/site SEO optimized with vBSEO.

    Q: What is your forum-root Google PR? We've definitely been able to identify a faster/deeper indexing trend associated to higher forum-root PR....

    ps- Welcome to vBSEO

  5. #5
    Junior Member
    Real Name
    GTX2
    Join Date
    Jul 2006
    Posts
    11
    Liked
    0 times
    thank you for your replies.
    I have been using google sitemap now and after making a search at google "link: www.myurl.com" i get 195 results of pages linking to my forums.

    I cannot understand why google doesn't even want to look at my forums.

    I'm always looking at who's online members, and i never see googlebot at my board (i only saw it once looking at my forum homepage and never more). Yahoo glurp spider is always at my board with 3/4 bots.

    how can i guarantee google indexes my board? how can i make sure vbseo will help me?

    I'm using also linkexchange.
    thank you for your help.

  6. #6
    vBSEO.com Webmaster Mert Gökçeimam's Avatar
    Real Name
    Lizard King
    Join Date
    Oct 2005
    Location
    Istanbul, Turkey, Turkey
    Posts
    23,111
    Liked
    622 times
    Blog Entries
    4
    You can never be sure about what google does but vBSeo is the best tool you can use

    Also are you using spiders.xml ? If you are not maybe you don't know if google is crawling your site.
    Mert Gökçeimam / Crawlability Inc.

    vBSEO 3.6.0 Alpha Önizlemesi - Including Like Tree
    Unveiling the NEW vBSEO Sitemap Generator 3.0 - available NOW for vBSEO Customers!


    Twitter:@Depkac
    Personal Blog : Mert Gökçeimam

  7. #7
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    the link: command is unreliable, and it is not the tool for what you want to look up. link: shows sites that link to you, IF THEY are indexed. I wouldn't use this as a judge of your site. I have 0 link: results on some sites that have every page indexed.

    the site: command is what you want to use to see how many SERP's you have.

  8. #8
    Junior Member
    Real Name
    GTX2
    Join Date
    Jul 2006
    Posts
    11
    Liked
    0 times
    didn't know about the spiders.xml file... installing now.

    What happens with googlebot is that when it comes to the forum, it gets stucked in only one thread for 10 minutes and then i only see the bot a couple of weeks later doing the same thing: crawling one thread and going away.

    I will install spiders.xml and probably install vbseo also and see results.
    thank you all for helping

  9. #9
    Junior Member
    Real Name
    GTX2
    Join Date
    Jul 2006
    Posts
    11
    Liked
    0 times
    after installing spiders hack i can see googlebot is coming sometimes to my forums but never following links.

    What is the best command search to list all urls of my domain at google?
    allinurl: myurl.com ?
    allinurl: site:myurl.com ?


    It's strange because google has in cache pages crawled in May 2005 and never updated those pages. They are still in google's cache.

  10. #10
    Junior Member
    Real Name
    GTX2
    Join Date
    Jul 2006
    Posts
    11
    Liked
    0 times
    sorry being boring with all this questions, but i've been worried with this :s
    Why googlebot never follow the links on my forums? Why it gets stucked in a thread/forum when it starts crawling the board? i see other forums and googlebot is always following the links....

  11. #11
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    do you have a robots.txt file?
    a sitemap submitted to google daily?
    vbseo installed?


  12. #12
    Junior Member
    Real Name
    GTX2
    Join Date
    Jul 2006
    Posts
    11
    Liked
    0 times
    1) robots.txt: yes (although, many sites say robots.txt is not necessary because normal procedure of searchbots is to follow all links)

    2) Sitemaps submitted daily: yes (although many sites say sitemaps should be submitted weekly or even monthly avoiding being "banned" from search engines)

    3) vbSEO: no (not yet, thinking if it would be worth to buy it or not...)


    I remembered one thing now: could this problem be because using vbadvanced in homepage as www.mydomain.com/index.php and using www.mydomain.com/forum/ as my forum directory and not in the root?
    Last edited by gtx2; 07-17-2006 at 07:18 PM.

  13. #13
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    1- right. but there are things you DONT want SE's to index to "pollute" your listings. if you do a search on here for robots.txt you will find many examples of files various members are excluding.

    2- Google will take a new sitemap ever second if you wanted to. It won't get you very far, but it won't get you banned either. I run mine nightly at 4am EST.

    3- what are you waiting for see my sig. it's been a great month.

    3b- no, vbadvanced won't have ill effects, nor will 1 level deep. many vbseo users run vba as well

  14. #14
    Junior Member
    Real Name
    GTX2
    Join Date
    Jul 2006
    Posts
    11
    Liked
    0 times
    I'm getting really bored with this problem about google.
    I think my domain should be banned from google, because pages never appear at google. I think VBSEO can't help me, because the problem should be with google itself.

    Google only indexes my homepage and updates it almost daily.
    Then, never follow all other links and also there are only 3 or 4 pages cached in may2005 at google search engine.

    Should i change my domain name to another one?
    I'm getting desperate with this...

    I see other boards in my country with 50 or 100 search engine bots online at the same time, including googlebot always online...

  15. #15
    Senior Member
    Real Name
    Joseph Ward
    Join Date
    Jun 2005
    Posts
    23,845
    Liked
    36 times
    Blog Entries
    9
    Well, if you have not achieved any significant level of current indexing, then changing your domain might be an option. It would be good if you could get an after market domain that already has some history, rather than a brand new domain.

    However, have you done anything in the past that might have gotten you penalized in Google? When was your domain first purchased and launched (could you be sandboxed?). Have you tried a Google re-inclusion request?

Page 1 of 2 1 2 LastLast

Similar Threads

  1. Replies: 1
    Last Post: 04-24-2006, 07:25 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •