Letting Adsense spider access private forums

[eJM]

The Adsense Site Authentication page leaves a lot to be desired. I still don't know how to allow the Adsense spider to access private forums. There are a number of forums on my site that are for flooring professionals only and they gain access only by being members of a special usergroup. Adsense spiders don't fall into that group.

I have Adsense ads in my private forums, but the spider is getting no-access messages. How can I set it up so that only the Adsense spider has access to show content relevant ads? I do NOT want the standard Google spider to access those forums.

Thank you for any help you can offer.

Jim McClain

[briansol]

Hello,
I'm done with Jim (eJM) and any information i ever gave him.

sorry if this effects you if you had the same question. This man is no longer worth my space on these forums.

Thank him for this dis-service.

[eJM]

Thanks, Brian, but I already came to that conclusion. The reason I came here is because the Adsense Authentication page doesn't give any usable examples of what to put in the Authentication URL box. I tried http://www.thefloorpro.com/community/login.php?do=login, but that only results in this error message:

Your submission could not be processed because a security token was missing.

If this occurred unexpectedly, please inform the administrator and describe the action you performed before you received this error.

Anyway, as I indicated in my original message, the page to add the Adsense access doesn't give directions. I bet a step-by-step for each box would be helpful to not only me, but a whole lot of other people who happen to have private forums with Adsense ads on them. I tried searching Google for this kind of information and got nothing helpful at all. Maybe vBSEO will be the top result for the terms I used (one of which is the title I used for this thread).

Thanks,

Jim

[briansol]

Hello,
I'm done with Jim (eJM) and any information i ever gave him.

sorry if this effects you if you had the same question. This man is no longer worth my space on these forums.

Thank him for this dis-service.

[Notorious]

Thanks, Brian, but I already came to that conclusion. The reason I came here is because the Adsense Authentication page doesn't give any usable examples of what to put in the Authentication URL box. I tried http://www.thefloorpro.com/community/login.php?do=login, but that only results in this error message:
Jim

site authentication for vb won't work as googlebot doesn't support cookies or session, at best you can get the bot to log in a single page.

You can use this hack KX - Spider Permissions - vBulletin.org Forum to set permissions for spiders.

[eJM]

Good point. i didn't even think of that.

I think the adsense auth tool is looking more for .htpassword style protected folders more so than a login.

Me thinks the only way to do this is to make a user called 'adsense bot' or whatever, and make that user have access to the private areas, and then via a product/plugin of some sort, check the IP of the guest visit, ID if it is a bot or not. If it is a bot, is it media partners? if so, set its user session.

i'll do some more reasearch tomorrow (late here now...) and maybe i'll think of something better.

I'm sure you're right about this applying more to protected folders. I did register a Adsense Spider as a member and added it to a group that has access (read only), but it would have to log in to be able to use it. Maybe the mod below...?

site authentication for vb won't work as googlebot doesn't support cookies or session, at best you can get the bot to log in a single page.

You can use this hack KX - Spider Permissions - vBulletin.org Forum to set permissions for spiders.

It's not GoogleBot, it's Mediapartners-Google*, according to the email I got from Google. I did add access to protected folders through robots.txt, but that won't get the spider into the private forums.

I'm going to check out the mod you linked to - it's for vB 3.8, but the author said it would work on 3.7.x too. I'll let you know.

Thanks,

Jim

[Notorious]

ah, I actually meant Mediapartners-Google,

[eJM]

ah, I actually meant Mediapartners-Google,

No matter, Notorious, that modification is black-hat SEO. There are no options to choose which spiders to give access to. Giving access to areas that unregistered visitors can't access is a violation of the Google Terms of Service. That mod gives access to private areas to any spider listed in the spiders_vbulletin.xml file. I have a modified version of that file too, so it would grant access to quite a few spiders.

If the author of that mod would provide better checks and options, it would be a good mod for those of us who want to follow the rules.

Jim

[Joe Ward]

site authentication for vb won't work as googlebot doesn't support cookies or session, at best you can get the bot to log in a single page.

You can use this hack KX - Spider Permissions - vBulletin.org Forum to set permissions for spiders.

Seems like this hack could work with some modifications.

[eJM]

I looked at the XML file to see if I could set it to look at a different spiders file, but I can't see anywhere in its code where it points to the spiders_vbulletin.xml file. If that were possible, I'd think it could be modified to look at a file that only had the Adsense spider listed in it. I've never written a plug-in before, so I don't know much about the content of this one. But it does appear to be pretty simple compared to many I've looked at.

Jim

[briansol]

Hello,
I'm done with Jim (eJM) and any information i ever gave him.

sorry if this effects you if you had the same question. This man is no longer worth my space on these forums.

Thank him for this dis-service.

[eJM]

Hi Brian,

I tried your plugin, but it doesn't seem to be working. It may be that it needs more time, but I'm guessing it should work right away.

I did edit your plugin a little, but only to correct spelling, add a URL to this thread and changed the version number to reflect the minor changes I made.

Do you think the conditional, if ($userinfo['spidertype'] == "Mediapartners-Google") could be the problem? Like I said before, I've never written a plug-in, so I don't know the coding at all. I did get an email from Google a week or so ago, directing me to add this to my robots.txt file:

User-agent: Mediapartners-Google* 
Disallow:

I was wondering how important that asterisk is?

Anyway, I've been known to break things I don't know anything about, so I hesitate to do any more editing. The one I did edit is below. Thanks for your efforts. I know this could help an awful lot of people if it worked.

Jim

[briansol]

Hello,
I'm done with Jim (eJM) and any information i ever gave him.

sorry if this effects you if you had the same question. This man is no longer worth my space on these forums.

Thank him for this dis-service.

[eJM]

Okay, I understand better. But it's not working as you'd hoped. Bummer.

Jim

[briansol]

Hello,
I'm done with Jim (eJM) and any information i ever gave him.

sorry if this effects you if you had the same question. This man is no longer worth my space on these forums.

Thank him for this dis-service.