Thread: How do you detect "Hit Bots"

I have been hearing about Hit Bots and I guess the term is pretty self explanatory. A program that generates hits by using automated bots... But how do you detect if somebody is sending bots to your site?

Also, I have been noticing on my Google Analytics that I am getting one visit per key word phrase. Some phrases are actually ridiculous and I find it hard to believe that somebody found my site using these keywords. Is this a bot? Is Google Crawling itself? Thanks to any answers you may bring to the table.

Hitbot Detector Instructions:

Go to the kitchen and get some Aluminum Foil
Make yourself an antenna like this:

login to your trade script and refresh every 10 seconds until you see the antenna start to vibrate

Originally Posted by prono

I have been hearing about Hit Bots and I guess the term is pretty self explanatory. A program that generates hits by using automated bots... But how do you detect if somebody is sending bots to your site?

Also, I have been noticing on my Google Analytics that I am getting one visit per key word phrase. Some phrases are actually ridiculous and I find it hard to believe that somebody found my site using these keywords. Is this a bot? Is Google Crawling itself? Thanks to any answers you may bring to the table.

One way is to set up an invisible link on your site that the visitors can't see or click on and see if it gets clicked anyway. Check the site/ip that click was generated by and then block them.

Originally Posted by morient

One way is to set up an invisible link on your site that the visitors can't see or click on and see if it gets clicked anyway. Check the site/ip that click was generated by and then block them.

Wow, good trick... I was twisting up my foil until I read this..

Originally Posted by prono

Also, I have been noticing on my Google Analytics that I am getting one visit per key word phrase. Some phrases are actually ridiculous and I find it hard to believe that somebody found my site using these keywords. Is this a bot? Is Google Crawling itself? Thanks to any answers you may bring to the table.

they're probably real searches. the idea of using a hitbot is generally to credit the referring site as having sent a hit, so they'll typically come from trades, not from search engine referrers.

people search for lots of really weird phrases.

Originally Posted by JACOBKELL

I will tell you what aheib told me:use common sense.And it working as a charm.

What is that? Common sense is not always so common. That's what the magic eight ball has told me...

Originally Posted by AmateurFlix

they're probably real searches. the idea of using a hitbot is generally to credit the referring site as having sent a hit, so they'll typically come from trades, not from search engine referrers.

people search for lots of really weird phrases.

Thanks for your reply,

But I think you may have misunderstood my question, or maybe I did not phrase it correctly. I was not implying that people are using hit bots to use google to access my site. These were not to be confused with one another. Two separate questions..

1st question, how to detect hit bots
2nd question, does google use bots to crawl itself

Originally Posted by prono

2nd question, does google use bots to crawl itself

Google uses a bot to scan your sites... as do all other SE's.

When you block bots make sure they aren't the "useful" ones.

Originally Posted by nation-x

Hitbot Detector Instructions:

Go to the kitchen and get some Aluminum Foil
Make yourself an antenna like this:

login to your trade script and refresh every 10 seconds until you see the antenna start to vibrate

I would personally used a Maxtor array as a conductor since Chinese bots travels such a long distance and have weaker signals, but thats just me

1. there are some scripts with prepackaged hitbot detection methods. they work because no one but the author knows how they work. if you find a way to detect hitbots effectively, then you shouldn't post about it either because someone will write a hitbot to defeat your methods. except for some basic things like the hidden link technique mentioned, you won't find much information about this topic, and for good reason. making it easier for the average webmaster to figure this stuff out would make it much harder for everyone else to detect hitbots. it's a catch 22.

2. AFAIK, googlebot would not send a referer containing search terms to your site. I doubt if google crawls itself. It is entirely possible for someone else to write a spider which would crawl google's search results and end up on your site, although I doubt if that is happening with any significant frequency for many terms on your one site.

Originally Posted by morient

Google uses a bot to scan your sites... as do all other SE's.

When you block bots make sure they aren't the "useful" ones.

Again, I will try to rephrase this so it is better understood.

Does google send out those little invisible guys called bots to google.com... Wait, I may have lost somebody there so lets back up....

Google bots go here ----> Google
Google bots crawl Google
Yes or No?

I know that SE bots crawl my site but do they crawl themselves? Does Google crawl Google, Does yahoo crawl yahoo, does msn crawl msn?

Originally Posted by AmateurFlix

2. AFAIK, googlebot would not send a referer containing search terms to your site. I doubt if google crawls itself. It is entirely possible for someone else to write a spider which would crawl google's search results and end up on your site, although I doubt if that is happening with any significant frequency for many terms on your one site.

Ok, I see you understood what I was getting at.. Thanks for your reply...

Originally Posted by AmateurFlix

1. there are some scripts with prepackaged hitbot detection methods. they work because no one but the author knows how they work. if you find a way to detect hitbots effectively, then you shouldn't post about it either because someone will write a hitbot to defeat your methods. except for some basic things like the hidden link technique mentioned, you won't find much information about this topic, and for good reason. making it easier for the average webmaster to figure this stuff out would make it much harder for everyone else to detect hitbots. it's a catch 22.

Continous calculation of a trades statistical deviation is an alternative to the "hidden link(s)" approach. Humans tend to be sheeps, bots does not. Not failsafe but used together with common sense its a pretty darn good tool.

Originally Posted by willwank

I would personally used a Maxtor array as a conductor since Chinese bots travels such a long distance and have weaker signals, but thats just me

Try using titanium foil. It's more expensive and harder to find than aluminum foil, but it will improve your reception about ten fold.

Since we are on the topic of bots and bot like hits, maybe someone else has seen this. I am now blocking 4-5 IP's a week from teen-hot.com for bad hits. I just brought up my draupnir "no ref" logs and here is my top 3 hitters for the last 24 hours. The first two have whois info that shows Germany and Russia and the third is always high in the list and comes back as a Google crawl server.
I have been blocking these IP's (not the google one) as they show up since they also usually come in at about 50% productivity. Any thoughts as to what might be going on? the IP's change all the time and are usually in different C blocks and countries.

3965 42.5 % 91.96.193.213
1053 11.3 % 93.158.151.25
510 5.5 % 66.249.66.161

Cyberprod, I just had to block 91.205.124.9 for the same reason. ~10k noref hits, doesn't appear that they were generating any clicks at all.

googlebot has been going bonkers on my site lately as well, however I suspect that might have something to do with my adding ~50k pages to my domain.

I wrote a script to detect hitbots... but I decided not to release it... because I didn't want the drama... some of the sites that sent me 100% hitbot traffic are very well known. Another reason I didn't release it is because I think that if I did the cheaters would figure out a way around it anyway... so I have just been using it for my own use. I just had a long phone conversation with another developer who wrote something similar (for a trade script he was planning on releasing) and when he realized that as much as 40% of the traffic coming to his sites was fake... alot from zombie machines infected with trojans. he realized that the only way that his script would be successful would be if he didn't include that feature.... simply because noone could grow a site using the script with that feature in it.

It comes down to this... do you want to build a large traffic site or combat hitbots? I am here to tell you that a good percentage of the traffic pool on most skim (and probably noskim too) sites is fake traffic. When I started filtering the traffic my network went from 300k/day to 120k/day. It's kind of a no win situation.

Originally Posted by nation-x

.... simply because noone could grow a site using the script with that feature in it.

It comes down to this... do you want to build a large traffic site or combat hitbots? I am here to tell you that a good percentage of the traffic pool on most skim (and probably noskim too) sites is fake traffic.

QUOTED FOR TRUTH

I've opted to combat bots, be extremely selective and seo my sites, and for that my sites grow extremely slow compared to many others. But on the other hand, I challenge anyone with tgps/mgps to beat my signup ratios.