Bad Bot Blacklist

October 31, 2023

Throughout the course of a normal day, we constantly get hammered by bots trying to find known vulnerabilities in various software packages by hitting different URLs on your website. Some of these bots can be pretty aggressive and may hit dozens of urls per second as they scan.

These are mostly pretty harmless, but they do take up server resources and if enough bad bots hit at once, it could cause a bit of a slowdown. Mostly these bots are just annoying though.

Over the last two months, I've been building a new rate-limiter- this prevents actions like hitting 100 urls per second by first warning the user/bot that they are making too many requests, and then if they persist, we blacklist them.

This blacklist is handled by each website that uses our software individually. So if company A has a bad bot, company B won't necessarily know about it unless the bot moves to attack them as well. Not ideal! If we identify a threat once, we should be able to shield every instance of our software from it, right?

SO, what I've built and deployed this last week is a way for each running instance of the Kiva Logic software to 'phone home' to kivalogic.com to pass along all the IP addresses of these bad bots via an API. kivalogic.com records these ip addresses, where they came from and who sent them.

Then, and this is the best part, kivalogic.com contacts CloudFlare via their API and adds the IP address to the CloudFlare web application firewall, which absolutely stops any requests from those bad bots from ever hitting your website!

IP addresses don't always stay in the same place with the same owner/user, they can be sold, they can re-assigned, they can be re-used. To handle this, what we do is:

Allow CloudFlare to dicate what kind of block is used- either a challenge to the user (hey are you a human?), an automated javascript test, or a strict disconnect block.
We un-blacklist each IP address after 6 months.

It's fascinating to see where these attacks come from, and we are stopping attacks from all around the world- literally AROUND the world. Egypt, Pakistan, Western Europe, Eastern Europe, all sorts of places.