Filtering more than web traffic: shutting bad bots out of SSH

In September we started judging bots by their behaviour. Bad bots caught on one server now get shut out across all protected servers on our fleet. That work covered web traffic, but bots don't stop at your website.

Any server with SSH open gets a steady stream of bots trying common usernames and leaked passwords. Most of them bounce off, but they never stop knocking. So we're taking the same community-immunisation approach to SSH for our managed Cloud Container customers.

Same idea, new door

We're taking our bad bot blocking beyond the web, and SSH is the first stop.

For the past couple of weeks, we've been collecting SSH logs from traffic across our entire network to build a wider blocklist. There are already more than 1,000 addresses on the blocklist at any given moment.

A bot trying its luck on any one of our servers ends up blocked on every managed Cloud Container.

Deciding what to block

The system we’re using looks for patterns that give SSH bots away. There’s brute force entry for example, where bots attempt lots of failed logins in a short time. There’s also the similar but different slow brute force, which is a brute force spread out to stay under the radar.

Brute force detection uses the same “leaky bucket” idea as our web traffic blocking. Every IP address represents a potential bot, so it gets its own bucket. Then each failed login adds a splash while the bucket slowly drains. Fail too often or too fast, and it overflows and the address is blocked.

Slow brute force uses a bigger bucket that drains more slowly. A bot that paces itself to slip past the first bucket will still fill this one up.

Blocking and banning

In our previous article about judging bots on their behaviour we explained how our system puts CAPTCHA challenges in front of suspected bad bot traffic. Any visitors that cause the leaky bucket to overflow get an automatic CAPTCHA challenge put in front of them, and they don’t get to access the website until they’ve completed the CAPTCHA. 

Unlike banned bad bot web traffic, SSH doesn’t have a web page to display a CAPTCHA on. Instead, a blocked address is simply cut off. Its connections go nowhere. This “blocking” only covers SSH, so that IP address can still load your website as normal.

Blocks lift on their own, and the first one lasts an hour. Each time the same address is caught again, the next block gets longer: four hours, then eight hours, then 12 hours. If a day goes by without it being blocked again, it's back to an hour.

Going beyond the server

Our Cloud Container servers already use a protective tool called fail2ban to catch bad bots hitting SSH ports. When fail2ban detects five failed logins on a server within 10 minutes, the IP address trying to log in gets locked out of the server for the next 10 minutes.

It’s effective, but fail2ban’s blocking is limited to each server it runs on. Every server counts attempted logins by itself and builds its own blocklist. Bad bots caught on one server can move to the next server and start again with a clean slate. And when a bot is blocked, it can always try the server again 10 minutes later once the block is lifted.

Our new SSH bad bot blocking system, however, watches and protects the whole managed Cloud Container fleet at once. Bots attempting logins on different servers are counted together, not in isolation. When a bot's address is flagged, it’s blocked on every protected server—not just the server it tried to access.

By design, the new system also catches the quieter traffic that fail2ban ignores. For example, it looks for addresses that connect to the SSH port on dozens of servers without ever attempting a login (that's exactly what a scanner bot mapping out targets looks like).

Now we’ve got the two tools working together: fail2ban guards the server it runs on, our new system works across the protected fleet to share blocklists and block bots that hide from fail2ban.

Who's getting protected?

SSH blocking is only for managed Cloud Container customers. It relies on the server's firewall, so it only applies on servers where we already manage that for you. As with our web traffic protection, it's going to roll out in stages so we can keep a close eye on it.

On all other services, nothing changes, and SSH works just as it does today. If you're an unmanaged customer and would like SSH blocking, get in touch with our team and we'll talk you through your options.

Do you need to do anything?

No. If you're a managed Cloud Container customer, SSH port blocking will just start working once a bad bot reaches your server.

The system looks for SSH users entering incorrect credentials, but a typo or two won't get you blocked since we look for patterns, not single failed logins. 

There's one thing to watch out for, though. Anything automated that keeps failing to log in can look like an attack, like a deploy script still using an old password.

If you think you've been blocked, SSH connections will time out rather than being refused (your website will keep loading as normal). In that scenario, reach out to our Support Team with the IP address you're connecting from and they can lift the block. We're also working on letting you manage your own exceptions in the Control Panel.

And while you're here, it's a good time to switch to SSH keys if you haven't already. A bot working through a list of leaked passwords won't guess a key.

Server Management gets even more valuable

If you’re using Cloud Containers but don’t have Server Management, this serves as a good reminder of Server Management’s ever-improving value.

There’s a set of developer-focused features that are only available on managed Cloud Containers, including:

• Container cloning and syncing.

• Simple Cache, which removes the pain of manual cache management with a single switch.

• One-click backup restoration.

• Toggling Traffic Protection On/Off.

• Scheduled maintenance, including image upgrades, performed by our team.

And now you can add SSH bad bot protection to that list.

If you want to know more about Server Management, we’re always happy to talk.