IP Set Goodness

What are IP sets?

From the man page..

Depending on the type of the set, an  IP set may store IP(v4/v6) addresses, (TCP/UDP) port numbers, IP and MAC address pairs, IP address and port number pairs, etc.

In a nutshell though they can be thought of as a hashmap of IP addresses or ip:port pairs.

Why use them?

Speed! They are super fast. They also simplify your iptables rules.

Storing all the ip addresses / CIDR blocks in their own hash map allows much more efficient lookups from iptables. Normally iptables would be required to have one rule per-match and would need to run each rule in order to determine when to take a specific action.

For example

-A INPUT -s 1.2.3.4/32 -p tcp -m tcp -j DROP
-A INPUT -s 1.2.3.5/32 -p tcp -m tcp -j DROP
-A INPUT -s 1.2.3.4/32 -p udp -m udp -j DROP
-A INPUT -s 1.2.3.5/32 -p udp -m udp -j DROP

The above is obviously a contrived example and normally you would be able to use a much more sane CIDR block to handle ranges of addresses, but what if you wanted to block (or allow) a huge list of specific IPs which do NOT fit neatly into CIDR ranges? That would require one iptable rule for each ip and requires iptables to evaluate each rule, in sequential order, for each packet to determine if that packet should be allowed or blocked. As you can probably imagine it does not take very long for that to slow down iptables and hence all your packets.

In the past there was really no reasonable way to ban (or allow) big lists of addresses without things getting really slow, or without your iptables rules getting insanely long and unmanageable.

Using ipsets moves all the ip addresses & CIDR blocks out of your iptables rules list, which makes looking at long rule chains much cleaner and easier.

What I will cover in this post is the hash:net type, which allows you to store CIDR address ranges and ip addresses. Those have been the most useful to me in blocking wide ranges of addresses from ip block lists.

The full details of the different types of ipsets can be found in the ipset manpage.

How to use them

IPsets are managed using the ipset command.

You can create a new set with ipset create SET_NAME hash:net.

New blocks or addresses can be added to the set with ipset add SET_NAME 1.1.1/24.

When updating blocks it is best to add the addresses to a temporary set and then swap them once they are loaded. This allows the existing rules and iptables to function while updating a set. To swap one set for another it is as easy as ipset swap TMP_SET_NAME SET_NAME.

Saving sets to a file is also very easy with the save command ipset save SET_NAME > OUTPUT_FILE.

You need a corresponding rule in your iptables to match the set.

# will DROP packets on interface enp0s25 where the src matches something in the SET_NAME ipset
-A INPUT -i enp0s25 -m set --match-set SET_NAME src -j DROP

How to use them with fail2ban

The ban action simply needs to be set correctly in your fail2ban jail.conf

[DEFAULT]
banaction = iptables-ipset-proto6
banaction_allports = iptables-ipset-proto6-allports

Now fail2ban will use ipsets to store all those addresses that end up on the block list.

Some helper scripts

You can find some scripts to help dealing with ipsets on my GitHub in my Linux-Server-Scripts repo.

I currently automate the update of my block lists using a simple cron script.

What are some good IP lists?

These are the lists currently pulled by my script above. There are plenty out there to choose from.

Blocklist.de

www.blocklist.de is a free and voluntary service provided by a Fraud/Abuse-specialist, whose servers are often attacked via SSH-, Mail-Login-, FTP-, Webserver- and other services. The mission is to report any and all attacks to the respective abuse departments of the infected PCs/servers, to ensure that the responsible provider can inform their customer about the infection and disable the attacker.

• www.blocklist.de
• https://lists.blocklist.de/lists/all.txt

Internet Storm Center

The ISC relies on an all-volunteer effort to detect problems, analyze the threat, and disseminate both technical as well as procedural information to the general public.

• https://isc.sans.edu/api/ - Available data feeds
• https://isc.sans.edu/api/threatcategory/research/ - List of research related IPs

# here is a one-liner you can use to parse the xml output above into a tmp file of the ipv4 addresses
# replace ${TMP_FILE} with the actual filename
curl --compressed "https://isc.sans.edu/api/threatcategory/research/" 2>/dev/null |\
 xmllint --xpath 'threatcategory/research/ipv4' - |\
 sed -e 's/<ipv4>//g' |\
 sed -e 's/<\/ipv4>/\n/g' > ${TMP_FILE}

IPsum

IPsum is a threat intelligence feed based on 30+ different publicly available lists of suspicious and/or malicious IP addresses.

• https://github.com/stamparm/ipsum
• https://raw.githubusercontent.com/stamparm/ipsum/master/levels/3.txt