Results 1 to 8 of 8

Bot constantly viewing tag list

This is a discussion on Bot constantly viewing tag list within the General Discussion forums, part of the vBSEO SEO Plugin category; I have set up my robot.txt and it is shown to be working fine but I have noticed google bot ...

  1. #1
    Member
    Real Name
    Craig
    Join Date
    Jul 2009
    Posts
    62
    Liked
    1 times

    Bot constantly viewing tag list

    I have set up my robot.txt and it is shown to be working fine but I have noticed google bot is regally viewing the "tag list", I think allot of time is taken from bot viewing this too often. I have a tag cloud on the forum, should I set this so that guests cant view it? or is it beneficial for the bot to look at this so much?

  2. #2
    Senior Member
    Real Name
    Michael Biddle
    Join Date
    Jan 2007
    Location
    Southern California
    Posts
    7,097
    Liked
    5 times
    Hello,

    You can block it off in your robots file.
    The Forum Hosting - Forum Hosting from the Forum Experts

  3. #3
    Member
    Real Name
    Craig
    Join Date
    Jul 2009
    Posts
    62
    Liked
    1 times
    Thanks for the reply,

    It doesn't actually show as a link in the who's on line (see pic). So would setting it to disallow actually work?

    Is this how it should be in my robot.txt

    Disallow: /viewing tag list/

    Many thanks.
    Attached Thumbnails Attached Thumbnails not-link.jpg  

  4. #4
    Senior Member
    Real Name
    Marco Mamdouh
    Join Date
    May 2008
    Location
    Egypt
    Posts
    2,627
    Liked
    3 times
    Blog Entries
    1
    First in vBSEO sitemap settings include tags choose No
    Second in your robots.txt add
    Code:
    Disallow: /tags/

  5. #5
    Member
    Real Name
    Craig
    Join Date
    Jul 2009
    Posts
    62
    Liked
    1 times
    Thanks Marco, vBSEO was already set to not include tags but I have added that the the robot.txt, will see how it goes from here.

    Once again many thanks, great support here

  6. #6
    Senior Member
    Real Name
    Marco Mamdouh
    Join Date
    May 2008
    Location
    Egypt
    Posts
    2,627
    Liked
    3 times
    Blog Entries
    1
    You are welcome

  7. #7
    Member
    Real Name
    Craig
    Join Date
    Jul 2009
    Posts
    62
    Liked
    1 times
    I have done this and it is still viewing the tag list all the time, if you look at the picture above the "Veiwing the tag list" doesn't appear to be a place on my forum, it isn't a link and if I enter it in the url eg www.mydomain/Viewing_Tag_List the page is not found, I cant see what the bot is looking at but it seems to spend 50 percent of its time in there.

  8. #8
    Senior Member
    Real Name
    Marco Mamdouh
    Join Date
    May 2008
    Location
    Egypt
    Posts
    2,627
    Liked
    3 times
    Blog Entries
    1
    First the tags correct URL is /tags/ and in who is online you will see viewing tags list so that's tell you the bot in tags list not in viewing tag list page
    Second yes you will see it in tags list but the bots will not index it
    there is different between view and index or view and didn't index which we have already done that in robots.txt file why disallow the bots to "index" it

Similar Threads

  1. Ban IP address from Viewing
    By stevectaylor in forum Off-Topic & Chit Chat
    Replies: 9
    Last Post: 02-23-2010, 09:16 PM
  2. Error when viewing post
    By brockied in forum Troubleshooting
    Replies: 4
    Last Post: 01-16-2009, 03:53 PM
  3. Spider Erorrs when viewing Tag List
    By MadMax in forum Troubleshooting
    Replies: 3
    Last Post: 09-07-2008, 05:06 PM
  4. Spiders viewing no permission msg
    By bela-meaad in forum Troubleshooting
    Replies: 1
    Last Post: 03-05-2007, 10:40 AM
  5. How easy to use? Do I need to constantly tweak settings?
    By sovaward in forum Pre-Sales Questions
    Replies: 11
    Last Post: 11-13-2006, 07:27 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •