How to create a vbulletin bot and scraper trap

First off the boring introduction part

Early last year I got hacked with a redirect code.
Thanks to the guys here, once the problem was found I was up and running again in a matter of minutes, so its only fair I release the bot trap here first

Everyone seems to have a downer on the likes of China, Russia etc
But how many know the truth behind where your content is going or by who its being scraped?

Some days I have been doing 3000+ unique and my earnings have been beyond dire
This then led me to do some experiments on finding ways of tracking where my hits have come from, what hits and a way of tracking and catching content scrapers

Your content is being used to help another person grow their websites.

I have been running some tests using nothing more than a free hack from vb ord by Paul M
Track Guest Visits - vBulletin.org Forum
Thats just a link to the 3.8 version, but there is also a 4 series version

And modifying my spiders_vbulletin.xml file

The latest version can be found on Vbulletin
WS Spiders List (for updated vBulletin "spiders_vbulletin.xml" files))

Boring bit over

Running the above you will have a nice show of all the spiders hitting you

As well as the spiders, you can add user agents to catch the content scrpaers

For those that dont know. The spiders_vbulletin.xml file in vb 3.8 is found under forum root > includes > xml and you will find it there

To add any of the following code. Open the file with notepad, wordpad etc and add it above </searchspiders> which is located near or at the very end of the file.

Save it, reupload it and then click on your Currently Active Users, which will then oad the latest changes.

These are some of my results tracking some user agents

<spider ident="FREE">
<name>Free</name>
</spider>

100% success rate on catching scrapers and spammers

<spider ident="Deepnet Explorer">
<name>Deepnet Explorer</name>
</spider>

100% success rate on catching scrapers and spammers

<spider ident="Gentoo">
<name>Gentoo</name>
</spider>

100% success rate. This will show a lot of softlayer, ThePlanet to name a few

<spider ident="DigExt">
<name>DigExt</name>
</spider>

100% success rate. Scrapers and spammers

<spider ident="Java">
<name>Java</name>
</spider>

70% success. But I found the moron that had been overloading my server with this one by scraping 150 pages at a time

<spider ident="Windows 98">
<name>Windows 98</name>
</spider>

How many people use windows 98? Another moron that was scraping me was along with a load of other unwanted people. Success rate 100%

<spider ident="WordPress">
<name>WordPress</name>
</spider>

This will show a lot of idiots running a wordpress add on that scrapes you for content.
You will see some trty and leave comments.
One guy was hitting me and scraping all content via the rss feeds and removing my link. Lets just say he had a divert placed to a rude sites rss feeds.

You cann the above in action on my forum, because I have the results on show.
Plus you will get an idea of some of the other user agents Im tracking to see what the results are like.

If anyone wants, I will add my own robots file. But it can change on an hourly basis if I find something else to track, or I get too many false positives.

The above are all good to use from my own experiments so far

If you find the above helps you, bung me a backlink
This is a project I have been working on for a few weeks and thought it only fair to share my findings and with any luck have others share theirs

This will give some idea on the success rate Im having using some of the above

02-01-2011, 09:46 AM WordPress Yes (2) showthread 74.204.184.144 Viewing Thread Smoking at Spanish cafes, bars banned
02-01-2011, 09:46 AM WordPress No newreply 74.204.184.144 *
02-01-2011, 05:42 AM WordPress Yes (1) showthread 74.52.31.162 Viewing Thread Spanish smokers enjoy last puffs ahead of new law
02-01-2011, 01:16 AM WordPress Yes (5) showthread 94.136.55.21 Viewing Thread Smoking at Spanish cafes, bars banned
02-01-2011, 01:10 AM WordPress No newreply 94.136.55.21 *

The above are from using the wordpress spider trap
I normally catch a lot more each day.

02-01-2011, 08:35 PM Gentoo Yes (1) showthread 69.163.239.121 Viewing Thread Barcelona aim to overcome defensive absences against struggling Levante
02-01-2011, 06:08 PM Gentoo Yes (1) showthread 97.74.24.226 Viewing Thread Flash mob witnesses bizarre marriage proposal at Vancouver airport
02-01-2011, 05:48 PM Gentoo Yes (1) showthread 174.122.2.34 Viewing Thread Why David Arquette looked like a clown at a Miami Beach party
02-01-2011, 05:00 PM Gentoo Yes (1) showthread 174.122.143.211 Viewing Thread Central Holidays Extends Early Booking Discount Promotion for New...
02-01-2011, 03:55 PM Gentoo Yes (1) showthread 74.52.60.98 Viewing Thread Smoking at Spanish cafes, bars banned
02-01-2011, 02:46 PM Gentoo Yes (1) showthread 208.88.227.218 Viewing Thread Flash mob witnesses bizarre marriage proposal at Vancouver airport
02-01-2011, 02:41 PM Gentoo Yes (3) showthread 78.46.102.40 Viewing Thread "Mommy, Mommy! can I have a bike for Christmas?
02-01-2011, 02:24 PM Gentoo Yes (1) showthread 174.120.148.130 Viewing Thread FLW College Fishing Southeast Division to open on Lake Okeechobee
02-01-2011, 02:00 PM Gentoo Yes (1) showthread 174.120.243.194 Viewing Thread Right hand man of Georgian mafia boss to be extradited to Spain from Russia
02-01-2011, 12:13 PM Gentoo Yes (1) showthread 74.52.179.98 Viewing Thread Top Spanish Properties This Week on Kyero
02-01-2011, 11:48 AM Gentoo Yes (1) showthread 174.121.37.162 Viewing Thread Salespeople, estate agents, property listers, telecanvassers.
02-01-2011, 10:14 AM Gentoo Yes (2) showthread 174.132.248.82 Viewing Thread More than 1,000 dead birds fall from sky in Ark.
02-01-2011, 08:02 AM Gentoo Yes (1) showthread 174.121.10.130 Viewing Thread Three men are trapped on an island. They find a genie's lamp and agree they will ...
02-01-2011, 02:46 AM Gentoo Yes (2) showthread 173.236.45.90 Viewing Thread CFS MERCEDES ML 320 AUTOMATIC 2002 PETROL SILVER 48,000KMS FSH
02-01-2011, 01:23 AM Gentoo Yes (1) showthread 74.220.215.216 Viewing Thread Damon "Wouldn't Change a Frame" of "True Grit"
02-01-2011, 12:55 AM Gentoo Yes (1) showthread 205.234.104.51 Viewing Thread Damon "Wouldn't Change a Frame" of "True Grit"
01-01-2011, 09:48 PM Gentoo Yes (1) showthread 74.220.215.86 Viewing Thread The Best Places for Tapas in Seville, Spain

The above is from tracking the Gentoo agent for 24hrs

There are other agents Im trialing to see what kind of results I get. This is from one of those

02-01-2011, 08:50 PM German server4 Yes (1) register 91.201.66.133 Member Registration attempt for UNSONTASP
02-01-2011, 07:21 PM German server4 Yes (4) showthread 83.43.205.75 Viewing Thread Global Insight: Spain's economic skies darken - Financial Times - Google jobs feed ne
02-01-2011, 06:44 PM German server4 Yes (1) showthread 190.51.17.163 Viewing Thread Adam Sandler Death Rumors Are Totally True
02-01-2011, 05:45 PM German server4 Yes (2) vbsoccer 91.210.105.63 Called with DO = 'schedule'
02-01-2011, 04:52 PM German server4 No misc 123.242.172.4 Called with DO = 'flashchat'
02-01-2011, 04:51 PM German server4 Yes (3) weather 58.26.168.174 *
02-01-2011, 04:24 PM German server4 Yes (2) showthread 66.249.82.196 Viewing Thread Ukrainians not interested in strawberry picking in Spain
02-01-2011, 03:48 PM German server4 Yes (6) showthread 74.125.16.66 Viewing Thread Celebrate the New Year in Spanish Style with La Senza - big news network
02-01-2011, 01:53 PM German server4 No misc 41.223.55.7 Called with DO = 'page'
02-01-2011, 11:15 AM German server4 Yes (1) index 219.136.240.166 Viewing Home Page
02-01-2011, 11:14 AM German server4 No showthread 195.110.6.134 Viewing Thread UGG Boots 50-70% Off, 100% Real sheepskin Boots, Big Fish
02-01-2011, 11:14 AM German server4 No showthread 166.111.120.63 Viewing Thread UGG Boots 50-70% Off, 100% Real sheepskin Boots, Big Fish
02-01-2011, 11:14 AM German server4 No showthread 213.251.154.218 Viewing Thread UGG Boots 50-70% Off, 100% Real sheepskin Boots, Big Fish
02-01-2011, 11:13 AM German server4 No showthread 195.201.131.5 Viewing Thread UGG Boots 50-70% Off, 100% Real sheepskin Boots, Big Fish
02-01-2011, 11:13 AM German server4 No showthread 77.122.172.104 Viewing Thread UGG Boots 50-70% Off, 100% Real sheepskin Boots, Big Fish
02-01-2011, 11:13 AM German server4 No showthread 41.133.101.149 Viewing Thread UGG Boots 50-70% Off, 100% Real sheepskin Boots, Big Fish
02-01-2011, 11:13 AM German server4 No showthread 124.234.144.7 Viewing Thread UGG Boots 50-70% Off, 100% Real sheepskin Boots, Big Fish
02-01-2011, 11:12 AM German server4 No showthread 113.108.198.45 Viewing Thread UGG Boots 50-70% Off, 100% Real sheepskin Boots, Big Fish
02-01-2011, 11:12 AM German server4 No showthread 117.126.3.54 Viewing Thread UGG Boots 50-70% Off, 100% Real sheepskin Boots, Big Fish
02-01-2011, 07:47 AM German server4 Yes (3) index 83.28.25.85 Viewing Home Page
02-01-2011, 06:21 AM German server4 No misc 111.160.70.145 Called with DO = 'flashchat'
02-01-2011, 06:21 AM German server4 Yes (4) index 111.160.70.144 Viewing Home Page
02-01-2011, 06:05 AM German server4 Yes (1) register 115.228.160.33 Member Registration attempt for Incagicioro
02-01-2011, 05:52 AM German server4 Yes (1) showthread 74.125.16.66 Viewing Thread "Wolf boy" - is there cure for him ?
02-01-2011, 04:48 AM German server4 Yes (1) showthread 201.194.3.198 Viewing Thread IFS Gamo MOD.400 air rifle
02-01-2011, 12:34 AM German server4 Yes (1) showthread 86.139.97.36 Viewing Thread Showjumper Dale Burnham recovering well after bad fall
01-01-2011, 10:09 PM German server4 Yes (1) showthread 83.54.63.107 Viewing Thread LOCH NESS GUN CLUB LTD, gun news, reviews, forums Spain

The above falls into the 90% success rate

This is the code Im using for that one

<spider ident="Windows NT 5.1; U">
<name>German server4</name>
</spider>

I originally found the user agent after looking at the agent a server hitting me from Germany was using. Hence my nick for that one.
At the same time there are a lot of Chinese and Russian IP's there

So far I have reduced my daily hits by about 1/5

I tend to ban them after they hit. That way I get a free one way link out of it and they dont get to come back for more.

Since doing this my junk email has dropped from about 200 a day to about 20 on a bad day

Another one I just checked on thats giving good results is this one

<spider ident="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)">
<name>Serverbot</name>
</spider>

Bet you cant guess why I called it that LOL

Proof of the results

02-01-2011, 08:16 PM Serverbot Yes (1) index 204.93.183.28 Viewing Home Page
02-01-2011, 06:06 PM Serverbot Yes (1) index 194.63.248.49 Viewing Home Page
02-01-2011, 12:46 AM Serverbot Yes (1) showthread 190.202.82.98 Viewing Thread Fury over racy images of women posing with dogs for charity calendar
01-01-2011, 09:53 PM Serverbot Yes (1) showthread 94.194.64.5 Viewing Thread Replacing A Provisional Licence?

The above are all from the last 24hrs

I have a lot more user agents that are being tested to see what others I can catch.

One worth adding is this

PHP Code:

        <spider ident="obot">
                <name>obot</name>
        </spider> 


I caught a fake bot operating from one country a week or so back. From memory it was the Ukraine
155 hits in about two minutes. I cant remember the bot name. But when you checked it out, the page it linked to was a holding page.

Over night I was in need of Viagra and a new fake rolex again.

On checking this user agent

PHP Code:

        <spider ident="Windows NT 5.2">
                <name>Windows NT 5.2</name>
        </spider> 


I discovered a scraper running from a Chinese ip address

03-01-2011, 10:42 AM Windows NT 5.2 Yes (54) forumdisplay 125.77.182.221

You get 90% google type hits show with that one, but you do find the sneaky Chinese and Russian hits trying to hide in there

Another one set up yesterday seemed pointless, then all of a sudden, its lived up to its name

PHP Code:

        <spider ident="Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3">
                <name>SoftLayer Server</name>
        </spider> 


Nothing all night with that one in place and then this

03-01-2011, 01:49 PM SoftLayer Server Yes (1) showthread 75.126.237.204 Viewing Thread PP stop Socialists establishing campaign office in Alicante
03-01-2011, 01:49 PM SoftLayer Server Yes (1) showthread 74.86.229.200 Viewing Thread Church thief rescued by helicopter
03-01-2011, 01:49 PM SoftLayer Server Yes (1) showthread 75.126.222.189 Viewing Thread Man dies after falling down health centre stairs
03-01-2011, 12:42 PM SoftLayer Server Yes (2) showthread 74.86.235.121 Viewing Thread EXTINGUISHED FIREFIGHTERS SUPPORT APANEE
03-01-2011, 12:42 PM SoftLayer Server Yes (1) showthread 75.126.213.177 Viewing Thread Short trip but difficult game for Real Madrid
03-01-2011, 12:41 PM SoftLayer Server Yes (2) showthread 75.126.222.224 Viewing Thread Barcelona kick off 2011 with hard fought win over levante
03-01-2011, 12:41 PM SoftLayer Server Yes (1) showthread 75.126.212.89 Viewing Thread RAISING MONEY FOR AECC
03-01-2011, 12:41 PM SoftLayer Server Yes (1) showthread 74.86.230.97 Viewing Thread LOCAL BUSINESSES SUPPORT MABS
03-01-2011, 12:41 PM SoftLayer Server Yes (1) showthread 74.86.229.207 Viewing Thread Spanish Manufacturing Improves In December
03-01-2011, 12:41 PM SoftLayer Server Yes (1) showthread 75.126.118.202 Viewing Thread PETS NEWS
03-01-2011, 12:41 PM SoftLayer Server Yes (1) showthread 75.126.225.83 Viewing Thread New winter opening hours at The Horse Trust

I love you oBot. Its just caught another false bot

User agent
vik-robot/Nutch-1.0 (vikspider; http://vik.com; chenlibiti@163.com) (dac852060f583703e8567cca732f736a)

Website
vik.com

Oh look, its for sale and a complete nothing.

The ip 184.72.7.221 traces back to Amazon

The website they are pretending to be are hosted on 208.87.33.150

These are the idiots Im after. The scrapers. The morons that run them should have their keyboards inserted where the sun dont shine

The following are some questions that got sent to me via PM from someone thats been following what I have posted on here about what I have been doing.

How does this stop people from scraping your content?

Its a bit like shutting the gate after the horse has bolted to an extent.
What Im doing is tracking user agents of known scrapers and just about anything else I can find to catch a person running a scraper script
What I have found is these people (if you can call them that) can hit several times a day and some only show as single page hits each time they hit
Others hit the same time each day
If your running bog standard Vbulletin, you only see the guestats that are online at the time you look.
Using Paul M's script you can see the last x amount of guests that have hit under each user agent.
Im managing to produce nice little groups of these user agents with a good deal of accuracy by trial and error

I thought it was a good thing to have spiders spidering content on my site. I did not know that they could steal content and post it on another site???

Goggle, Yahoo, MSN all send hits
A lot of these severs are set to scrape content. A bit like a copy and paste, but by an automated site thats just after scraping content.
There is a hack on the org to disable right click.
It took me one click on a browser to get around this on my own forum
From my won understanding of SEO, links from bad sites go against you.
These guys are hitting your site, you dont know if they are the good, the bad or the ugly.
I have also found a lot of sites that scrape your content and then charge people for an analysis of your data. So lets get this right, we all offer forums that are free to use, some moron is then scraping the data and charging for it. Can we charge that moron for scraping our data?

Another thing that can happen is a site that ranks better than you for keywords is scraping your content. Say you have a unique post on eggs and bacon, I believe the Americans would be more inclined for coffee and muffins.
Your content is unique. Your doing well for a day or two, along comes a pr6 site that scrapes that content. Who will get the hits for that content.

What about rss feeds? I thought supplying those was a good idea since vbseo does it. Can people actually remove my link from an rss feed and post content from my site without linking back to us?

rss feeds are simple to tweak to remove the link back to the site with the original content. If you set up a feed on a forum, I believe its the last line of text you remove and hey presto, thats your content with no link back to the original site
You can reduce the amount of words the feed produces.
I know Brian helped me at one stage with my own feeds.

I want to implement as much of your good work as I can, but since I know so little about how the spider thing works, I don't want to do something that will block my content from the 'good' spiders. Can you recommend what I should do?

So far through my own tests, I have managed to get half a dozen user agents that produce results of the scraper kind. These are the well known ones like "Free" "Deepnet Explorer" "Java"

If you look at my own forums, you will see just how many user agents Im finding, tracking for 48hrs and seeing what the results are. Links in my signature and I have the list on show, an option in Paul M's mod

Some give false positives. Others are producing some good results. MRS seems to be a nice return of idiots at present, Gento has been another good one along with Windows 98. Lets give all our old computers to third world countries so they can get online and screw our websites


Thanks for your post and this good information. I hope I can learn how to do what you are doing. I get so many visitors (hits) per month, but my adsense revenue is terrible. I suspect it must be what you are describing.

What I will do at some stage over the next day or two is get a copy of the spiders list and add just those that give positive results, the likes of Gento, Softlayer etc

What Im doing is tracking them. This does not ban them
What I have is a nice list of potential scrapers etc.
I then spend time going through each list that made, checking to see if they are servers or real people. Servers get banned via cpanel, which writes to my htaccess

From time to time, I watch my current guests online
This is where I found that a French site called Gaoland was hitting about every fifteen minutes. They were taking feeds and god knows what. A server does not hit your site for no reason. I did a google to find their full ip range, one of the few that I have gone to that length on.

I tried the following code in and out of my htaccess file over about eighteen hours
Eighteen hours too long for the amount of attempted member registrations

PHP Code:

RewriteEngine On
RewriteCond %{HTTP_REFERER} \.ru [NC,OR]
RewriteCond %{HTTP_REFERER} \.fr [NC,OR]
RewriteCond %{HTTP_REFERER} \.BR [NC,OR]
RewriteCond %{HTTP_REFERER} \.IN [NC,OR]
RewriteCond %{HTTP_REFERER} \.KP [NC,OR]
RewriteCond %{HTTP_REFERER} \.KR [NC,OR]
RewriteCond %{HTTP_REFERER} \.VN [NC,OR]
RewriteCond %{HTTP_REFERER} \.UA [NC,OR]
RewriteCond %{HTTP_REFERER} \.TR [NC,OR]
RewriteCond %{HTTP_REFERER} \.IT [NC,OR]
RewriteCond %{HTTP_REFERER} \.AR [NC,OR]
RewriteCond %{HTTP_REFERER} \.NG [NC,OR]
RewriteCond %{HTTP_REFERER} \.RO [NC,OR]
RewriteCond %{HTTP_REFERER} \.cn [NC]
RewriteRule .* - [F,L] 


It went bac in when I noticed how many of our slanty eyed friends and their vodka drinking buddies were back on wither trying to register or just borrowing content

It wont stop all the hits, but it doe stop a lot. As far as slowing the server, there is very little difference in speed between using that code and not using that code

My countries on block will be different to other peoples

If you just after China, Korea and Russia, add this to your htaccess file
Korea are another menace

PHP Code:

RewriteEngine On
RewriteCond %{HTTP_REFERER} \.ru [NC,OR]
RewriteCond %{HTTP_REFERER} \.KR [NC,OR]
RewriteCond %{HTTP_REFERER} \.cn [NC]
RewriteRule .* - [F,L] 


This is where the likes of Brian or Mert step in and point out an error in my SEO methods

Im not an expert on this. I have just hit that point where its been time to do something about it and so far, what I have done has made some of these idiots as easy as shooting fish in a barrel to identify

And from what I can see, no one else has been publishing the code to add to your spider list to make them easily visible

This gives some idea to the results of the last 24hrs using the above and one or two more Im trialing

OffByOne Spiders (3) < user agent used by content thieves and spam poster
Windows 98 Spiders (11) < dont give your old computers to third world countries
Gentoo Spiders (18)
DigExt Spiders (17)
Some Russian Geezer Spiders (2) < AKA user agent Free
Comment spammer Spiders (3)
WordPress Spiders (4)
Java Spiders (3)
SoftLayer Server Spiders (11)
Artabus Spiders (1)
Deepnet Explorer Spiders (1) < must be national Vodka drinking day, I normally get a lot more on that one and the one above
Serverbot Spiders (20)
Serverbot2 Spiders (2) Serverbot3 Spiders (11)
German server Spiders (11)
German server3 Spiders (1)
German server4 Spiders (18)
MRA Spiders (4) < been trailed from today along with the one below
MRA 4.3 Spiders (4)

For those that cant wait for a tidy version of the spiders xml file
I have added my present scruff one which also tracks mobile device users
Plus a new agent I discovered on hits from another Amazon ip
Something called Flipboa
None of the hits have swear words describing their user agents
Some of the user agents give a lot of false positives, but at the same time help break the main list of hits down to a better level for spotting other servers.

A rule of thumb o that is, if you see anything showing "home page" its about a 30/70 chance of finding another server hit

Another fake bot found of the vik-robot/Nutch-1.0 variety
This time its ip 184.72.20.195
Traces back to one of the many Amazon servers.

I have managed to get one of the Gento hits tracked down to a single website to see what type of website they are running. Only the one site shown on the reverse ip I did on it.

It looks as though they are running wordpress

As I say, I got a visit from their server. All 15 visits to be exact

03-01-2011, 10:05 PM 86.47.44.247 Viewing Thread POPPY APPEAL TOTAL REVEALED
03-01-2011, 10:04 PM 86.47.44.247 Viewing Thread POPPY APPEAL TOTAL REVEALED
03-01-2011, 10:03 PM 86.47.44.247 Viewing Thread POPPY APPEAL TOTAL REVEALED
03-01-2011, 09:58 PM 86.47.44.247 Viewing Thread POPPY APPEAL TOTAL REVEALED
03-01-2011, 09:44 PM 86.47.44.247 Viewing Thread POPPY APPEAL TOTAL REVEALED
03-01-2011, 07:34 PM 86.47.44.247 Viewing Thread The 2011 Weird Science Awards
03-01-2011, 07:34 PM 86.47.44.247 Viewing Thread 2011 Trends: Experts see modest jobs improvement for 2011 - North Bay Business Journa
03-01-2011, 07:34 PM 86.47.44.247 Viewing Thread Corporate tax rate keeping jobs overseas - Nashua Telegraph
03-01-2011, 07:34 PM 86.47.44.247 Viewing Thread San Javier Civil Guard allegedly used force's IT system to plan burglaries
03-01-2011, 07:34 PM 86.47.44.247 Viewing Thread Free "FREESAT" for communities
03-01-2011, 06:31 PM 86.47.44.247 Viewing Thread Spanish smoking ban 'puts jobs at risk' - Mirror.co.uk
03-01-2011, 06:31 PM 86.47.44.247 Viewing Thread The 2011 Weird Science Awards
03-01-2011, 06:31 PM 86.47.44.247 Viewing Thread 2011 Trends: Experts see modest jobs improvement for 2011 - North Bay Business Journa
03-01-2011, 06:31 PM 86.47.44.247 Viewing Thread Corporate tax rate keeping jobs overseas - Nashua Telegraph
03-01-2011, 06:31 PM 86.47.44.247 Viewing Thread Free "FREESAT" for communities

The promised bot list wont happen today. Unfortunately my eyesight is in a bad way. To many nights working until 3am
My own earnings are back on the up and a lot of the junk traffic is on the decline
This could just be luck
Im tracking a few more agents in the hope of finding another scraper agent to give away more server hits

Looking at the screen through squinted eyes. Nothing all day on the softlayer server tracker and then this. Like waiting for a bus, nothing all day and then this

04-01-2011, 05:56 PM SoftLayer Server Yes (1) showthread 174.36.122.61 Viewing Thread Albelli acquires Fotobook.co.uk
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 174.37.2.113 Viewing Thread Spain Sees Progress on Deficit and Unemployment
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 74.86.244.79 Viewing Thread Spain's ACS Secures Crucial Stake in Rival Hochtief
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 174.37.29.41 Viewing Thread Spain's ACS steps up battle for Hochtief - Summary
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 208.43.19.236 Viewing Thread Spanish court open probe into killings in Iraq
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 174.37.112.19 Viewing Thread Barcelona visits Athletic Bilbao in Copa del Rey
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 74.86.236.50 Viewing Thread Prostitution ring broken up in Alicante and Murcia
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 74.86.130.159 Viewing Thread Cádiz man finally compensated for rape he did not commit
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 74.86.235.126 Viewing Thread Spain's ACS takes 30 percent share Hochtief
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 174.37.7.95 Viewing Thread New SDXC card from Lexar
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 75.126.213.176 Viewing Thread Navy detains Spanish fishing vessel
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 208.43.19.230 Viewing Thread Spain's unemployment claims drop in December
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 208.43.51.135 Viewing Thread Spanish Box Office Drops 3% in 2010 to $874 Million
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 75.126.214.173 Viewing Thread Spain Sees Progress on Deficit and Unemployment - New York Times

Squinted eyes still in place and this is the code I have in place for catching the softlayer hits

<spider ident="Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3">
<name>SoftLayer Server</name>
</spider>

As you can see, it works rather well. This includes the above 14 and 26 of their relatives

04-01-2011, 07:56 PM SoftLayer Server Yes (1) showthread 75.126.225.92 Viewing Thread Countdown for Alicante chiringuitos
04-01-2011, 07:56 PM SoftLayer Server Yes (1) showthread 74.86.85.121 Viewing Thread Bare it All for a Bargain in Spain (VIDEO)
04-01-2011, 07:56 PM SoftLayer Server Yes (1) showthread 74.86.227.238 Viewing Thread Real Madrid's run in the first round
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 74.86.236.59 Viewing Thread Spanish jobless rate drops in five-month first
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 208.43.25.38 Viewing Thread Chinese vice premier arrives in Spain for visit
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 75.126.119.120 Viewing Thread Valencia to declare Alcoy Three Kings Parade as Intangible Cultural Heritage
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 74.86.244.72 Viewing Thread Spain's National Court calls Iraqi officer for questioning on raid on refugee camp
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 75.126.213.182 Viewing Thread Marbella restaurant warned that it must comply with anti-smoking law
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 75.126.119.114 Viewing Thread Spanish court open probe into killings in Iraq
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 174.37.62.161 Viewing Thread Spain's ACS grabs stake in German builder Hochtief
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 75.126.222.235 Viewing Thread Prostitution ring broken up in Alicante and Murcia
04-01-2011, 07:55 PM SoftLayer Server Yes (2) showthread 174.36.122.61 Viewing Thread Spain Plans Second Stress Test, Sees Limited FROB Use
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 75.126.222.190 Viewing Thread Spanish unemployment increased 4.5 per cent in 2010
04-01-2011, 07:55 PM SoftLayer Server Yes (1) showthread 74.86.229.195 Viewing Thread Spain Weather - Tuesday January 4 2010
04-01-2011, 06:56 PM SoftLayer Server Yes (1) showthread 208.43.51.140 Viewing Thread Thieves Tunnel Into Bank Over New Year
04-01-2011, 06:56 PM SoftLayer Server Yes (1) showthread 75.126.212.80 Viewing Thread Second Annual Magners Irish Cider Comedy Festival in Boston
04-01-2011, 06:56 PM SoftLayer Server Yes (1) showthread 74.86.119.80 Viewing Thread ebookers launches TV ad campaign
04-01-2011, 06:56 PM SoftLayer Server Yes (1) showthread 75.126.246.101 Viewing Thread Spain Business Brief - Tuesday January 4 2011
04-01-2011, 06:56 PM SoftLayer Server Yes (1) showthread 75.126.237.199 Viewing Thread Welcome to Local Spanish Forum Member
04-01-2011, 06:56 PM SoftLayer Server Yes (1) showthread 74.86.240.125 Viewing Thread Directflights claims 8% conversion rate ahead of full launch
04-01-2011, 06:56 PM SoftLayer Server Yes (1) showthread 74.86.230.172 Viewing Thread ETA cache found in France near Toulouse
04-01-2011, 06:56 PM SoftLayer Server Yes (1) showthread 208.43.25.45 Viewing Thread Widow Builds Vault, Could Get Corpses Back
04-01-2011, 06:55 PM SoftLayer Server Yes (1) showthread 174.36.122.48 Viewing Thread DNA Clears Man Who Spent 30 Years In Prison
04-01-2011, 06:55 PM SoftLayer Server Yes (1) showthread 75.126.246.97 Viewing Thread Ill. mom gives birth to 1 twin in 2010, 1 in 2011
04-01-2011, 06:55 PM SoftLayer Server Yes (1) showthread 75.126.118.211 Viewing Thread Spain's National Court calls Iraqi officer for questioning on raid on refugee camp
04-01-2011, 06:55 PM SoftLayer Server Yes (1) showthread 174.37.2.105 Viewing Thread Investment in media to present true picture of China
04-01-2011, 06:55 PM SoftLayer Server Yes (1) showthread 208.43.25.37 Viewing Thread Partial Solar Eclipse Visible Over Mideast, Europe
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 174.37.2.113 Viewing Thread Spain Sees Progress on Deficit and Unemployment
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 74.86.244.79 Viewing Thread Spain's ACS Secures Crucial Stake in Rival Hochtief
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 174.37.29.41 Viewing Thread Spain's ACS steps up battle for Hochtief - Summary
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 208.43.19.236 Viewing Thread Spanish court open probe into killings in Iraq
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 174.37.112.19 Viewing Thread Barcelona visits Athletic Bilbao in Copa del Rey
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 74.86.236.50 Viewing Thread Prostitution ring broken up in Alicante and Murcia
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 74.86.130.159 Viewing Thread Cádiz man finally compensated for rape he did not commit
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 74.86.235.126 Viewing Thread Spain's ACS takes 30 percent share Hochtief
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 174.37.7.95 Viewing Thread New SDXC card from Lexar
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 75.126.213.176 Viewing Thread Navy detains Spanish fishing vessel
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 208.43.19.230 Viewing Thread Spain's unemployment claims drop in December
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 208.43.51.135 Viewing Thread Spanish Box Office Drops 3% in 2010 to $874 Million
04-01-2011, 05:55 PM SoftLayer Server Yes (1) showthread 75.126.214.173 Viewing Thread Spain Sees Progress on Deficit and Unemployment - New York Times

So much for an early night.

They have been back with more ips. Same time, each time.

Its 3am, I want my bed, but seeing what else they hit with is just to good to see

Rather than post the full list, I will add the 40 or so more ips that have since hit

05-01-2011, 02:57 AM SoftLayer Server Yes (1) showthread 75.126.214.33 Viewing Thread True Grit &ndash; review
05-01-2011, 02:57 AM SoftLayer Server Yes (1) showthread 174.37.62.171 Viewing Thread Seven in 10 people from Madrid regularly read for pleasure
05-01-2011, 02:57 AM SoftLayer Server Yes (1) showthread 174.37.7.94 Viewing Thread Smoking law brings 700 complaints in two days
05-01-2011, 02:57 AM SoftLayer Server Yes (1) showthread 208.43.25.33 Viewing Thread the Best Resorts in Spain
05-01-2011, 02:57 AM SoftLayer Server Yes (1) showthread 74.86.226.28 Viewing Thread 'Four Injured Performers Is Just Not Strange,' Recovering 'Spider-Man' Dancer Says
05-01-2011, 02:57 AM SoftLayer Server Yes (1) showthread 174.37.29.48 Viewing Thread 'True Grit,' '127 Hours,' receive Writers Guild noms
05-01-2011, 01:57 AM SoftLayer Server Yes (1) showthread 174.37.112.73 Viewing Thread Spanish smoking ban: At last, Spain emerges from the Dark Ages, with a cough ... - Da
05-01-2011, 01:57 AM SoftLayer Server Yes (1) showthread 208.43.25.46 Viewing Thread EU Residents Downbeat About Local Job Market in 2010 - Gallup.com
05-01-2011, 01:57 AM SoftLayer Server Yes (1) showthread 75.126.212.85 Viewing Thread China lends a hand as Spain climbs from economic doldrums
05-01-2011, 01:57 AM SoftLayer Server Yes (1) showthread 74.86.244.76 Viewing Thread Athletic displays at 'Balagan' circus show in Spain
05-01-2011, 01:57 AM SoftLayer Server Yes (1) showthread 75.126.118.220 Viewing Thread Top Chinese official in talks on Spain's economic woes
05-01-2011, 01:57 AM SoftLayer Server Yes (1) showthread 74.86.124.142 Viewing Thread Golf-Old and new on McDowell's agenda for PGA Tour season opener
05-01-2011, 01:57 AM SoftLayer Server Yes (1) showthread 74.86.226.24 Viewing Thread Spanish government points to economic upturn - Sydney Morning Herald
05-01-2011, 01:57 AM SoftLayer Server Yes (1) showthread 174.37.2.111 Viewing Thread Ronaldo bags pair for Real Madrid
04-01-2011, 11:56 PM SoftLayer Server Yes (1) showthread 174.37.7.81 Viewing Thread Kodak Easyshare Touch, Mini and Sport Cameras
04-01-2011, 11:56 PM SoftLayer Server Yes (1) showthread 75.126.237.195 Viewing Thread Kodak Pulse digital frames
04-01-2011, 11:56 PM SoftLayer Server Yes (1) showthread 174.37.112.74 Viewing Thread New All-in-One Inkjet Printers from Kodak
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 208.43.19.233 Viewing Thread Spanish builder ACS closes in on German rival
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 75.126.212.94 Viewing Thread More birds fall from sky &mdash; this time in La.
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 74.86.227.233 Viewing Thread grandmother, grandson party hearty
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 75.126.217.183 Viewing Thread What if the Bible Really Is True
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 174.37.2.100 Viewing Thread Mont. boy recovering after impaled while sledding
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 75.126.246.108 Viewing Thread house hunting
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 174.37.112.69 Viewing Thread 'Unkillable' cow puts elderly man in hospital
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 75.126.214.32 Viewing Thread Could the days of the British "pint" be numbered
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 174.37.29.58 Viewing Thread Assistly Secures $3M in Funding Round from Bullpen Capital, Index Ventures, True Vent
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 75.126.119.117 Viewing Thread Tendulkar ton adds to the twist as India take slim lead
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 75.126.246.104 Viewing Thread La. has mass bird kill just days after Ark.
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 75.126.246.103 Viewing Thread `True Grit,' `127 Hours' receive writers noms
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 174.37.29.47 Viewing Thread Madison burglar gives tequila back
04-01-2011, 10:56 PM SoftLayer Server Yes (1) showthread 75.126.248.187 Viewing Thread Illinois mom gives birth to 1 twin in 2010, 1 in 2011
04-01-2011, 09:55 PM SoftLayer Server Yes (1) showthread 75.126.118.195 Viewing Thread looking for property.
04-01-2011, 09:55 PM SoftLayer Server Yes (1) showthread 74.86.85.124 Viewing Thread Popular grey Monet's Garden critically ill with foot infection
04-01-2011, 09:55 PM SoftLayer Server Yes (1) showthread 208.43.25.32 Viewing Thread Photographers gear up for CES gadget fest
04-01-2011, 09:55 PM SoftLayer Server Yes (1) showthread 75.126.213.186 Viewing Thread Box office now open for Blenheim Horse Trials
04-01-2011, 08:55 PM SoftLayer Server Yes (1) showthread 74.86.236.63 Viewing Thread Madrid Museum goers move from the Prado to the Reina Sofia
04-01-2011, 08:55 PM SoftLayer Server Yes (1) showthread 208.43.51.131 Viewing Thread Spain's El Niño lottery approaches yet again
04-01-2011, 08:55 PM SoftLayer Server Yes (1) showthread 75.126.70.131 Viewing Thread More than 1.5 tons of cannabis seized in Almería

Just goes to show, not all morons have Chinese or Russian ips

Please tell me this is some kind of Joke google are playing.
They have changed their server or something daft

Been to bed, got up and they are still coming

A few more bad ips for you

05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 174.37.7.90 Viewing Thread Prostitution ring broken up In Spain
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 174.37.29.39 Viewing Thread Three kings day in Spain 06/01/2011
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 208.43.25.47 Viewing Thread Husbands 'butt call' does not sit well with Illinois SWAT team
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 74.86.85.114 Viewing Thread New edition removes Mark Twain's 'offensive' words
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 75.126.214.167 Viewing Thread SohoOS Releases New Multilingual Version and Mobile App for the Micro Business Manage
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 74.86.130.158 Viewing Thread China to make multimillion pound investment in beleaguered Spain
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 74.86.240.124 Viewing Thread X Factor USA 2011 Simon Cowell threatens to halt Launch US X Factor if Cheryl Cole ca
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 75.126.118.206 Viewing Thread Real Madrid keep pace with Barcelona thanks to Cristiano Ronaldo double
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 75.126.222.183 Viewing Thread Man stole cop car to be deported
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 174.36.122.40 Viewing Thread &lsquo;What are you thankful for&rsquo;
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 75.126.70.133 Viewing Thread Lady Tigers win Spanish Fort Christmas Bash
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 74.86.244.66 Viewing Thread Ind. vet dissects strange bird deaths
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 174.37.62.188 Viewing Thread Pep Guardiola FC Barcelona Coach understands Gabriel Milito&rsquo;s desire to leave J
05-01-2011, 10:59 AM SoftLayer Server Yes (1) showthread 75.126.118.218 Viewing Thread A look into the "True Life" of food addicts
05-01-2011, 09:58 AM SoftLayer Server Yes (1) showthread 75.126.214.36 Viewing Thread Spanish Deficit Down as Unemployment Eases
05-01-2011, 07:59 AM SoftLayer Server Yes (1) showthread 75.126.225.88 Viewing Thread ACS Spanish construction group steps up battle for Hochtief to launch a full takeover
05-01-2011, 07:59 AM SoftLayer Server Yes (1) showthread 75.126.171.129 Viewing Thread Athletic Bilbao vs FC Barcelona Copa Del Rey King&rsquo;s Cup 05/01/2011
05-01-2011, 07:59 AM SoftLayer Server Yes (1) showthread 74.86.119.83 Viewing Thread Nadal wins in Doha
05-01-2011, 07:59 AM SoftLayer Server Yes (1) showthread 75.126.217.191 Viewing Thread United Kingdom: General Office Administrator / Registrar
05-01-2011, 07:58 AM SoftLayer Server Yes (1) showthread 174.37.112.23 Viewing Thread Málaga Football Club spending for future success
05-01-2011, 07:58 AM SoftLayer Server Yes (1) showthread 75.126.237.197 Viewing Thread Spain Papers Review - Wednesday January 5 2011
05-01-2011, 07:58 AM SoftLayer Server Yes (1) showthread 75.126.118.203 Viewing Thread Spanish economy government points to economic upturn
05-01-2011, 07:58 AM SoftLayer Server Yes (1) showthread 75.126.222.186 Viewing Thread Unemployment jobless rate in Spain drops fell for the first time for five months in D
05-01-2011, 07:58 AM SoftLayer Server Yes (1) showthread 75.126.225.91 Viewing Thread Chinese Vice Premier Li Keqiang visits Spain amid hopes of economic support keep buyi
05-01-2011, 07:58 AM SoftLayer Server Yes (1) showthread 75.126.119.109 Viewing Thread Prado Museum Madrid suffers drop in visitor numbers in 2010
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 74.86.119.82 Viewing Thread Swear law falls foul of police
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 74.86.124.138 Viewing Thread 10-year-old girl discovers a supernova
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 74.86.124.130 Viewing Thread Colts Follow Strange Path to Their Usual Spot
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 74.86.227.235 Viewing Thread Ill. mom gives birth to 1 twin in 2010, 1 in 2011
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 75.126.248.184 Viewing Thread World's fattest man's heartbreak hunger
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 75.126.119.118 Viewing Thread Fight over Facebook results in stabbing
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 174.37.29.56 Viewing Thread Strange names leadership team for Ala. AG's office
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 75.126.248.185 Viewing Thread True Pheromones Presents-True Love For Men
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 75.126.119.98 Viewing Thread MK Livni attacks Netanyahu for giving her chance to talk about Pollard
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 208.43.51.138 Viewing Thread 10-year-old girl discovers a supernova
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 74.86.244.209 Viewing Thread Swear law falls foul of police
05-01-2011, 06:58 AM SoftLayer Server Yes (1) showthread 74.86.244.212 Viewing Thread World's fattest man's heartbreak hunger
05-01-2011, 05:58 AM SoftLayer Server Yes (1) showthread 75.126.213.190 Viewing Thread Samsung connects phone and camera for remote control with SH100
05-01-2011, 05:57 AM SoftLayer Server Yes (1) showthread 75.126.246.110 Viewing Thread Spain's unemployment claims drop in December
05-01-2011, 04:57 AM SoftLayer Server Yes (1) showthread 74.86.152.37 Viewing Thread Golf-PGA Tour gives fans green light to use mobile phones
05-01-2011, 03:57 AM SoftLayer Server Yes (1) showthread 75.126.222.191 Viewing Thread Spanish star Javier Bardem
05-01-2011, 03:57 AM SoftLayer Server Yes (1) showthread 75.126.222.226 Viewing Thread Sainz, Despres still atop Dakar Rally
05-01-2011, 03:57 AM SoftLayer Server Yes (1) showthread 174.37.112.29 Viewing Thread PREVIEW-Golf-Competing in Maui a twin bonus for holder Ogilvy
05-01-2011, 03:57 AM SoftLayer Server Yes (1) showthread 174.37.112.30 Viewing Thread Spanish judge to probe Iraq refugee camp killings - lawyer
05-01-2011, 03:57 AM SoftLayer Server Yes (1) showthread 75.126.70.146 Viewing Thread Seven in 10 people from Madrid regularly read for pleasure
05-01-2011, 03:57 AM SoftLayer Server Yes (1) showthread 74.86.240.112 Viewing Thread Chinese vice premier visits Spain amid hopes of economic support

hi

thanks for your great effot but i have 2 questions?

1- how to know is this bad or good bot to your site.

2- if it is bad how to get rid of it? ie: what we must to do after detect it?


thanks