IP blacklist on nethserver


(jack) #1

Why not add the possibility to add an IP blacklist?
Manually or by list.

This is a Simple script i originally find on the IPFire forum that i have adapted to work on NS.
In order to make it work i need to add the blacklist option in /etc/shorewall/interfaces

nano /etc/shorewall/interfaces
(in my case)

# Force GREEN + RED mode with only one interface
net     eth0    dhcp,nosmurfs,optional

to:

 # Force GREEN + RED mode with only one interface
 net     eth0    dhcp,nosmurfs,optional,blacklist

At the moment (as the script config) the resulting blacklist is of about 15.300 IP.
But you can easly add or remove lists.

Hope this help. :wink:

The script:
#!/bin/bash

########## CONFIG
#___________________________
#
# A firewall blacklist composed from IP lists, providing maximum protection with minimum false positives.
# Suitable for basic protection on all internet facing servers, routers and firewalls.
# includes: bambenek_c2 dshield feodo fullbogons palevo spamhaus_drop spamhaus_edrop sslbl zeus_badips ransomware_rw)
FIREHOL_LEVEL1=1
#____________________________
# An ipset made from blocklists that track attacks, during about the last 48 hours.
# (includes: blocklist_de dshield_1d greensnow openbl_1d virbl)
FIREHOL_LEVEL2=1
#____________________________
# An ipset made from blocklists that track attacks, spyware, viruses.
# It includes IPs than have been reported or detected in the last 30 days.
# (includes: bruteforceblocker ciarmy dragon_http dragon_sshpauth dragon_vncprobe dshield_30d dshield_top_1000
# malc0de maxmind_proxy_fraud myip openbl_30d shunlist snort_ipfilter sslbl_aggressive talosintel_ipfilter zeus vxvault)
FIREHOL_LEVEL3=1
#____________________________
# An ipset made from blocklists that track attacks, but may include a large number of false positives.
# (includes: cleanmx_viruses blocklist_net_ua botscout_30d cruzit_web_attacks cybercrime haley_ssh
# iblocklist_hijacked iblocklist_spyware iblocklist_webexploit ipblacklistcloud_top iw_wormlist malwaredomainlist)
FIREHOL_LEVEL4=0
#____________________________
# A web server IP blacklist made from blocklists that track IPs that should never be your web users.
# (This list includes IPs that are servers hosting malware, bots, etc or users having a long criminal history.
# This list is to be used on top of firehol_level1, firehol_level2, firehol_level3 and possibly firehol_proxies
# or firehol_anonymous) . (includes: hphosts_emd hphosts_exp hphosts_fsa hphosts_hjk hphosts_psh hphosts_wrz
# maxmind_proxy_fraud myip pushing_inertia_blocklist stopforumspam_toxic)
FIREHOL_WEBSERVER=0
#____________________________
# An ipset that includes all the anonymizing IPs of the world. (includes: anonymous bm_tor dm_tor firehol_proxies tor_exits)
FIREHOL_ANONYMOUS=0
#____________________________
# An ipset made from blocklists that track abusers in the last 30 days.
# (includes: cleantalk_new_30d cleantalk_updated_30d php_commenters_30d php_dictionary_30d php_harvesters_30d
# php_spammers_30d stopforumspam sblam)
FIREHOL_ABUSERS30D=0
#____________________________
# List of ip related to advertising and tracking
YOYO_ADLIST=1
#____________________________
# EmergingThreats.net Command and Control IPs These IPs are updates every 24 hours and should be
# considered VERY highly reliable indications that a host is communicating with a known and active
# Bot or Malware command and control server
ET_BOTCC=1
#____________________________
# EmergingThreats.net compromised hosts
ET_COMPROMISED=1
#____________________________

##Enable to remove specified ip from list
ENABLE_REMOVING=1
#
###Change accordingly to your system
BASE="/etc"
FOLDER_BL="sysconfig"
BLACKLIST="$BASE/$FOLDER_BL/blocklist"
WHITELIST="$BASE/$FOLDER_BL/whitelist" #Manually create this file
#___________________________________

rm $BASE/$FOLDER_BL/bl.tmp
rm $BASE/$FOLDER_BL/bl1.tmp
rm $BASE/$FOLDER_BL/b2.tmp
rm $BASE/$FOLDER_BL/blocklist

if [ $FIREHOL_LEVEL1 -ne 0 ]; then
        wget -t 3 --no-verbose https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset -O $BASE/$FOLDER_BL/firehol_level1.tmp
        cat $BASE/$FOLDER_BL/firehol_level1.tmp > $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in firehol_level1  %s \n"  `cat $BASE/$FOLDER_BL/firehol_level1.tmp | wc -l`
        rm $BASE/$FOLDER_BL/firehol_level1.tmp
fi

if [ $FIREHOL_LEVEL2 -ne 0 ]; then
        wget -t 3 --no-verbose https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset -O $BASE/$FOLDER_BL/firehol_level2.tmp
        cat $BASE/$FOLDER_BL/firehol_level2.tmp >> $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in firehol_level2  %s \n"  `cat $BASE/$FOLDER_BL/firehol_level2.tmp | wc -l`
        rm $BASE/$FOLDER_BL/firehol_level2.tmp
fi

if [ $FIREHOL_LEVEL3 -ne 0 ]; then
        wget -t 3 --no-verbose https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset -O $BASE/$FOLDER_BL/firehol_level3.tmp
                cat $BASE/$FOLDER_BL/firehol_level3.tmp >> $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in firehol_level3  %s \n"  `cat $BASE/$FOLDER_BL/firehol_level3.tmp | wc -l`
        rm $BASE/$FOLDER_BL/firehol_level3.tmp
fi

if [ $FIREHOL_LEVEL4 -ne 0 ]; then
        wget -t 3 --no-verbose https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset -O $BASE/$FOLDER_BL/firehol_level4.tmp
        cat $BASE/$FOLDER_BL/firehol_level4.tmp >> $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in firehol_level4  %s \n"  `cat $BASE/$FOLDER_BL/firehol_level4.tmp | wc -l`
        rm $BASE/$FOLDER_BL/firehol_level4.tmp
fi

if [ $FIREHOL_WEBSERVER -ne 0 ]; then
        wget -t 3 --no-verbose https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_webserver.netset -O $BASE/$FOLDER_BL/firehol_webserver.tmp
        cat $BASE/$FOLDER_BL/firehol_webserver.tmp >> $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in firehol_webserver  %s \n"  `cat $BASE/$FOLDER_BL/firehol_webserver.tmp | wc -l`
        rm $BASE/$FOLDER_BL/firehol_webserver.tmp
fi

if [ $FIREHOL_ANONYMOUS -ne 0 ]; then
        wget -t 3 --no-verbose https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_anonymous.netset -O $BASE/$FOLDER_BL/firehol_anonymous.tmp
        cat $BASE/$FOLDER_BL/firehol_anonymous.tmp >> $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in firehol_anonymous  %s \n"  `cat $BASE/$FOLDER_BL/firehol_anonymous.tmp | wc -l`
        rm $BASE/$FOLDER_BL/firehol_anonymous.tmp
fi

if [ $FIREHOL_ABUSERS30D -ne 0 ]; then
        wget -t 3 --no-verbose https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_abusers_30d.netset -O $BASE/$FOLDER_BL/firehol_abusers_30d.tmp
        cat $BASE/$FOLDER_BL/firehol_abusers_30d.tmp >> $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in firehol_abusers_30d  %s \n"  `cat $BASE/$FOLDER_BL/firehol_abusers_30d.tmp | wc -l`
        rm $BASE/$FOLDER_BL/firehol_abusers_30d.tmp
fi

if [ $YOYO_ADLIST -ne 0 ]; then
        wget -t 3 --no-verbose "http://pgl.yoyo.org/adservers/iplist.php?ipformat=plain&showintro=0&mimetype=plaintext" -O $BASE/$FOLDER_BL/yoyo_ad.tmp
        cat $BASE/$FOLDER_BL/yoyo_ad.tmp >> $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in YOYO_ADLIST  %s \n"  `cat $BASE/$FOLDER_BL/yoyo_ad.tmp | wc -l`
        rm $BASE/$FOLDER_BL/yoyo_ad.tmp
fi

if [ $ET_BOTCC -ne 0 ]; then
        wget -t 3 --no-verbose "https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/et_botcc.ipset" -O $BASE/$FOLDER_BL/et_botcc.tmp
        cat $BASE/$FOLDER_BL/et_botcc.tmp >> $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in ET_BOTCC  %s \n"  `cat $BASE/$FOLDER_BL/et_botcc.tmp | wc -l`
        rm $BASE/$FOLDER_BL/et_botcc.tmp
fi

if [ $ET_COMPROMISED -ne 0 ]; then
        wget -t 3 --no-verbose "https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/et_compromised.ipset" -O $BASE/$FOLDER_BL/et_compromised.tmp
        cat $BASE/$FOLDER_BL/et_compromised.tmp >> $BASE/$FOLDER_BL/bl.tmp
        printf "\n Amount of lines in ET_COMPROMISED  %s \n"  `cat $BASE/$FOLDER_BL/et_compromised.tmp | wc -l`
        rm $BASE/$FOLDER_BL/et_compromised.tmp
fi



printf "\n Amount of lines in the combined blocklist before any cleanup is done %s \n"  `cat $BASE/$FOLDER_BL/bl.tmp | wc -l`
printf "\n Remove comments etc."

cat $BASE/$FOLDER_BL/bl.tmp | sort | uniq > $BASE/$FOLDER_BL/bl1.tmp
sed /#/d $BASE/$FOLDER_BL/bl1.tmp > $BASE/$FOLDER_BL/bl2.tmp

if [ $ENABLE_REMOVING -ne 0 ]; then
        ### use this to remove ipadress or range from the blocklist (if needed)
        ### Add ip(s), one per line, in $BASE/$FOLDER_BL/whitelist
        awk '{if (f==1) { r[$0] } else if (! ($0 in r)) { print $0 } } ' f=1 $BASE/$FOLDER_BL/whitelist f=2 $BASE/$FOLDER_BL/bl2.tmp > $BASE/$FOLDER_BL/blocklist
else
        mv $BASE/$FOLDER_BL/bl2.tmp > $BASE/$FOLDER_BL/blocklist
fi

printf "\n Amount of lines in final blocklist %s \n"  `cat $BASE/$FOLDER_BL/blocklist | wc -l`

rm -rf /etc/shorewall/blacklist
cp $BASE/$FOLDER_BL/blocklist /etc/shorewall/blacklist

shorewall refresh &>/dev/null

UPDATE:
Thanks to Giacomo, in order to make the edit at the shorewall config persistend do this:

mkdir -p /etc/e-smith/templates-custom/etc/shorewall/interfaces/

Then add this code to /etc/e-smith/templates-custom/etc/shorewall/interfaces/20nics

#
# 20nics
#
{
    use esmith::NetworksDB;
    my $ndb = esmith::NetworksDB->open_ro();
    my $mac_validation = $firewall{'MACValidation'} || 'disabled';
    my $mac_option = '';
    if ($mac_validation eq 'enabled') {
        $mac_option = ',maclist';
    }

    if (!defined($ndb->red)) {
        foreach my $i ($ndb->green) {
            my $role = $i->prop('role') || next;
            my $type = $i->prop('type') || '';
            next if ($role eq 'slave' || $role eq 'bridged' || $role eq 'pppoe');
            next if ($type eq 'alias');
            if ($role eq 'green') {
                $OUT .= "# Force GREEN + RED mode with only one interface\n";
                $OUT .= "net\t".$i->key."\tdhcp,nosmurfs,optional,blacklist";
            }
            $OUT .= ",bridge" if ($type eq 'bridge');
            $OUT .= "\n";
        }
    } else {
        foreach my $i ($ndb->interfaces) {
            my $role = $i->prop('role') || next;
            my $type = $i->prop('type') || '';
            next if ($role eq 'slave' || $role eq 'bridged' || $role eq 'pppoe');
            next if ($type eq 'alias');
            if ($role eq 'green') {
                $OUT .= "loc\t".$i->key."\tdhcp,nosmurfs,routeback".$mac_option;
            } elsif ($role eq 'red') {
                $OUT.="net\t".$i->key."\tdhcp,nosmurfs,optional,blacklist";
            } else {
                $role = substr($role,0,5); #truncate zone name to 5 chars
                if ($role eq 'blue') {
                    $OUT.="$role\t".$i->key."\tdhcp,nosmurfs,routeback".$mac_option;
                } else {
                    $OUT.="$role\t".$i->key."\tdhcp,nosmurfs,routeback";
                }
            }
            $OUT .= ",bridge" if ($type eq 'bridge');
            $OUT .= "\n";
        }
    }
}

UPDATE1:
Added Ransomware blocklist , readded yoyoadlist

UPDATE2:
Complete Rewrite, changed blocklist (Thanks to firehol) :slight_smile:
Create the whiteliste file if needed

UPDATE3:
Added 2 blocklist ET_BOTCC and ET_COMPROMISED


How to block Google Adsense?
Firewall Block Lists
Fail2Ban Feature Improvements
Someone try hack my nethserver?
(Alessio Fattorini) #2

It looks interesting but I don’t have seen any request for this, can you describe a scenario that makes sense for you?


(jack) #3

it protect your network from know malicius ip, you can enable the TOR blocklist TOR blocklist or with blocklist.de block the ip that has been reporting to attack apache installation or ssh,email,imap (see there…)
An other pro is that you can also block malwaredomainor the yoyo Adlist to block malaware, spyware, botnet and also advertise and tracking.

So, this can increase the security (if you use NS as gateway this protect all the LAN user)

Sorry for my english…
what do you think about it?

With this enabled my Fail2ban works much less :wink:

NOTE: if you use the IPS with the ET ruleset:

EmergingThreats Compomised IP
EmergingThreats Block IP

these list are already included, so remove from the script :wink:


(jack) #4

But the file /etc/shorewall/interfaces return to its originale state Every reboot…how can i make this edit persistent?


(James Nesbitt) #5

I have got this working in production so I know this works, the trick to make it persistent is the following:

  1. Create an ipset with the following 2 commands:

ipset -N blockips
ipset create blockips hash:net family inet hashsize 16384 maxelem 65536

  1. Add the following in the /etc/shorewall/blrules file

#ACTION SOURCE DEST PROTO DPORT
DROP net:+blockips fw

  1. At the end of the script, do the following:
    3.1) Save all of the IP addresses into a text file.
    3.2) Add each IP address in the file into the IPSet. Example code:
    while read line; do
    ipset -A -exist blockips $line
    done < '/root/blacklist/blacklisthitstmp3’
    3.3) Save the IPSet. Example code:
    ipset -S > /etc/shorewall/ipsets

  2. Create the /etc/e-smith/templates-custom/etc/shorewall/shorewall.conf/60options file with the following in it:

BLACKLIST=“ALL”

Got a lot of this out of a previous discussion on a topic similar to this located at Linking IPset into Shorewall. I suspect that I have missed one or two minor commands to complete this, but this should get you 98% of the way there. If I remember what I have missed out, I will post it here.


(Giacomo Sanchietti) #6

This template-custom enable blacklists for red interfaces.
See also: http://shorewall.net/blacklisting_support.htm -> “Static blacklisting”

Create the template-custom dir:

mkdir -p /etc/e-smith/templates-custom/etc/shorewall/interfaces/

Then add this code to /etc/e-smith/templates-custom/etc/shorewall/interfaces/20nics

#
# 20nics
#
{
    use esmith::NetworksDB;
    my $ndb = esmith::NetworksDB->open_ro();
    my $mac_validation = $firewall{'MACValidation'} || 'disabled';
    my $mac_option = '';
    if ($mac_validation eq 'enabled') {
        $mac_option = ',maclist';
    }

    if (!defined($ndb->red)) {
        foreach my $i ($ndb->green) {
            my $role = $i->prop('role') || next;
            my $type = $i->prop('type') || '';
            next if ($role eq 'slave' || $role eq 'bridged' || $role eq 'pppoe');
            next if ($type eq 'alias');
            if ($role eq 'green') {
                $OUT .= "# Force GREEN + RED mode with only one interface\n";
                $OUT .= "net\t".$i->key."\tdhcp,nosmurfs,optional";
            }
            $OUT .= ",bridge" if ($type eq 'bridge');
            $OUT .= "\n";
        }
    } else {
        foreach my $i ($ndb->interfaces) {
            my $role = $i->prop('role') || next;
            my $type = $i->prop('type') || '';
            next if ($role eq 'slave' || $role eq 'bridged' || $role eq 'pppoe');
            next if ($type eq 'alias');
            if ($role eq 'green') {
                $OUT .= "loc\t".$i->key."\tdhcp,nosmurfs,routeback".$mac_option;
            } elsif ($role eq 'red') {
                $OUT.="net\t".$i->key."\tdhcp,nosmurfs,optional,blacklist";
            } else {
                $role = substr($role,0,5); #truncate zone name to 5 chars
                if ($role eq 'blue') {
                    $OUT.="$role\t".$i->key."\tdhcp,nosmurfs,routeback".$mac_option;
                } else {
                    $OUT.="$role\t".$i->key."\tdhcp,nosmurfs,routeback";
                }
            }
            $OUT .= ",bridge" if ($type eq 'bridge');
            $OUT .= "\n";
        } 
    }
}

At the end:

signal-event firewall-adjust

(jack) #7

Thanks!!!

In my case (i have only 1 nic) i have to change this in order to make it working correctly:

        $OUT .= "# Force GREEN + RED mode with only one interface\n";
        $OUT .= "net\t".$i->key."\tdhcp,nosmurfs,optional";

with:

        $OUT .= "# Force GREEN + RED mode with only one interface\n";
        $OUT .= "net\t".$i->key."\tdhcp,nosmurfs,optional,blacklist";

(Giacomo Sanchietti) #8

I’ve been thinking on it for a while, but probably we can safely enable blacklisting on all interfaces by default.


(James Nesbitt) #9

@giacomo - that may be a good idea as it would only be used if someone populates the blacklist file


(jack) #10

i don’t know how shorewall works…in that case shorewall will check twice the packet (in case of 2 nic red and green)?..is this a waste of resource or not?


(Giacomo Sanchietti) #11

Shorewall optimizes the rules, so I don’t think we have a waste of resources.


(James Nesbitt) #12

@jackyes I tested out the custom template and it applied it automatically against the RED interface.


(jack) #13

yes if you have separated red and green interface, if you have only one nic you have to modify it accordingly.


(jack) #14

Can’t edit the first post…
Added Ransomware blocklist , readded yoyoadlist

#!/bin/bash -

###########################################################################
# 2014-03-03 by burningpenguin: scriptblacklistip
#
# add blocked IPs to the firewall to not be accessed from green/blue
###########################################################################
# 2014-03-03 initial version
# 2014-12-13 adjustments by Garp
# 2014-12-28 adjustments by Garp (blocklists added)
# 2014-12-29 Garp: Remove yoyo Ads, they are being blocked by using the
#                hostsfile and the url filter
# 2016-03-03 jackyes: Added some list and adapted to work on NethServer
#                (NOTE: add blacklist option to interface in  /etc/shorewall/interfaces)
# 2016-06-03 jackyes: Added Ransomware blocklist, readded yoyoadlist
#
###########################################################################
# usage
#  extend by adding this section per blacklist url
#   #get list from ??service??
#   printf "\n Get files: ??service???"
#   wget -O - http:???url??? > /etc/sysconfig/blacklisttmp
#   cat /etc/sysconfig/blacklisttmp >> /etc/sysconfig/blacklist
#
#
###########################################################################

mkdir /root/blacklist
rm -rf /root/blacklist/*

#get list from dan.me.uk torlist
##Download once @ max 30 min or be banned!
printf "\n Get files: TOR node list"
wget -O - --no-check-certificate https://www.dan.me.uk/torlist/ > /root/blacklist/blacklisttorlist
cat /root/blacklist/blacklisttorlist >> /root/blacklist/blacklist

#get list from CIA-Army
printf "\n Get files: CIA-Army"
wget -O -  http://cinsscore.com/list/ci-badguys.txt > /root/blacklist/blacklistcibadguys
cat /root/blacklist/blacklistcibadguys >> /root/blacklist/blacklist

#get list from Ransomware tracker
printf "\n Get files: Ransomware List"
wget -O -  https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt > /root/blacklist/RW_IPBL
cat /root/blacklist/RW_IPBL >> /root/blacklist/blacklist

#get list from YoYo Ad List
printf "\n Get files: YoYo AD LIST"
wget -O -  "https://pgl.yoyo.org/as/iplist.php?format=&showintro=0" > /root/blacklist/YOYOAD
cat /root/blacklist/YOYOAD >> /root/blacklist/blacklist

#get list from malc0de
printf "\n Get files: malc0de"
wget -O -  http://malc0de.com/bl/IP_Blacklist.txt > /root/blacklist/blacklistmalc0de
cat /root/blacklist/blacklistmalc0de >> /root/blacklist/blacklist

#get list from mailwaredomains
printf "\n Get files: mailwaredomains"
wget -O - http://www.malwaredomainlist.com/hostslist/ip.txt > /root/blacklist/blacklistmalwaredomains
cat /root/blacklist/blacklistmalwaredomains >> /root/blacklist/blacklist

#get list from openbl.org
printf "\n Get files: OpenBL 7 days"
wget -O - http://www.openbl.org/lists/base_7days.txt > /root/blacklist/blacklistopen7days
cat /root/blacklist/blacklistopen7days >> /root/blacklist/blacklist

#get list from emergingthreats
printf "\n Get files: EmergingThreats Compromised IPs"
wget -O -  http://rules.emergingthreats.net/blockrules/compromised-ips.txt > /root/blacklist/blacklistemercompr
cat /root/blacklist/blacklistemercompr >> /root/blacklist/blacklist

#get list from emergingthreats
printf "\n Get files: EmergingThreats Block IPs"
wget -O - --no-check-certificate https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt > /root/blacklist/blacklistemerblock
cat /root/blacklist/blacklistemerblock >> /root/blacklist/blacklist

#get list from palevotracker
printf "\n Get files: palevo"
wget -O - --no-check-certificate https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist > /root/blacklist/blacklistpalevo
cat /root/blacklist/blacklistpalevo >> /root/blacklist/blacklist

#get list from feodotracker Blocklist
printf "\n Get files: Feodo A"
wget -O - --no-check-certificate https://feodotracker.abuse.ch/blocklist/?download=ipblocklist > /root/blacklist/blacklistfeodoa
cat /root/blacklist/blacklistfeodoa >> /root/blacklist/blacklist

#get list from feodotracker BadIPS
printf "\n Get files: Feodo BadIPS"
wget -O - --no-check-certificate https://feodotracker.abuse.ch/blocklist/?download=badips > /root/blacklist/blacklistfeodob
cat /root/blacklist/blacklistfeodob >> /root/blacklist/blacklist

#get list from zeustracker
printf "\n Get files: zeus"
wget -O - --no-check-certificate https://zeustracker.abuse.ch/blocklist.php?download=badips > /root/blacklist/blacklistzeus
cat /root/blacklist/blacklistzeus >> /root/blacklist/blacklist

#get list from blacklist.de - all.txt
##VERY BIG!!
#printf "\n Get files: blacklistede all"
#wget -O - http://lists.blocklist.de/lists/all.txt > /root/blacklist/blacklistblall
#cat /root/blacklist/blacklistblall >> /root/blacklist/blacklist

#get list from blacklist.de - apache.txt
##VERY BIG!!
#printf "\n Get files: blacklistede apache"
#wget -O - http://lists.blocklist.de/lists/apache.txt > /root/blacklist/blacklistblapache
#cat /root/blacklist/blacklistblapache >> /root/blacklist/blacklist

#get list from blacklist.de - imap.txt
printf "\n Get files: blacklistede imap"
wget -O - http://lists.blocklist.de/lists/imap.txt > /root/blacklist/blacklistblimap
cat /root/blacklist/blacklistblimap >> /root/blacklist/blacklist

#get list from blacklist.de - ssh.txt
printf "\n Get files: blacklistede ssh"
wget -O - http://lists.blocklist.de/lists/ssh.txt > /root/blacklist/blacklistblssh
cat /root/blacklist/blacklistblssh >> /root/blacklist/blacklist

#get list from blacklist.de - bruteforcelogin.txt
printf "\n Get files: blacklistede ssh"
wget -O - http://lists.blocklist.de/lists/bruteforcelogin.txt > /root/blacklist/blacklistblbfl
cat /root/blacklist/blacklistblbfl >> /root/blacklist/blacklist



printf "\n Amount of lines in TOR list %s \n"  `cat /root/blacklist/blacklisttorlist | wc -l`
rm -rf /root/blacklist/blacklisttorlist

printf "\n Amount of lines in CIA-Badguys %s \n"  `cat /root/blacklist/blacklistcibadguys | wc -l`
rm -rf /root/blacklist/blacklistcibadguys

printf "\n Amount of lines in Ransomware list %s \n"  `cat /root/blacklist/RWIPBL | wc -l`
rm -rf /root/blacklist/RW_IPBL

printf "\n Amount of lines in YoYo Ad list %s \n"  `cat /root/blacklist/YOYOAD | wc -l`
rm -rf /root/blacklist/YOYOAD

printf "\n Amount of lines in malc0de %s \n"  `cat /root/blacklist/blacklistmalc0de | wc -l`
rm -rf /root/blacklist/blacklistmalc0de

printf "\n Amount of lines in MalwareDomains %s \n"  `cat /root/blacklist/blacklistmalwaredomains | wc -l`
rm -rf /root/blacklist/blacklistmalwaredomains

printf "\n Amount of lines in OpenBL 7 days list %s \n"  `cat /root/blacklist/blacklistopen7days | wc -l`
rm -rf /root/blacklist/blacklistopen7days

printf "\n Amount of lines in EmergingThreats Compomised IP's %s \n"  `cat /root/blacklist/blacklistemercompr | wc -l`
rm -rf /root/blacklist/blacklistemercompr

printf "\n Amount of lines in EmergingThreats Block IP's %s \n"  `cat /root/blacklist/blacklistemerblock | wc -l`
rm -rf /root/blacklist/blacklistemerblock

#printf "\n Amount of lines in SpyEye %s \n"  `cat /root/blacklist/blacklistspyeye | wc -l`
#rm -rf /root/blacklist/blacklistspyeye

printf "\n Amount of lines in Palevo %s \n"  `cat /root/blacklist/blacklistpalevo | wc -l`
rm -rf /root/blacklist/blacklistpalevo

printf "\n Amount of lines in Feodo A %s \n"  `cat /root/blacklist/blacklistfeodoa | wc -l`
rm -rf /root/blacklist/blacklistfeodoa

#printf "\n Amount of lines in Blacklist.de apache  %s \n"  `cat /root/blacklist/blacklistblapache | wc -l`
#rm -rf /root/blacklist/blacklistblapache

printf "\n Amount of lines in Blacklist.de imap  %s \n"  `cat /root/blacklist/blacklistblimap | wc -l`
rm -rf /root/blacklist/blacklistblimap

printf "\n Amount of lines in Blacklist.de ssh  %s \n"  `cat /root/blacklist/blacklistblssh | wc -l`
rm -rf /root/blacklist/blacklistblssh

printf "\n Amount of lines in Blacklist.de bruteorcelogin  %s \n"  `cat /root/blacklist/blacklistblbfl | wc -l`
rm -rf /root/blacklist/blacklistblbfl

#printf "\n Amount of lines in Blacklist.de all  %s \n"  `cat /root/blacklist/blacklistblall | wc -l`
#rm -rf /root/blacklist/blacklistblall

printf "\n Amount of lines in Feodo B %s \n"  `cat /root/blacklist/blacklistfeodob | wc -l`
rm -rf /root/blacklist/blacklistfeodob

printf "\n Amount of lines in Zeus %s \n"  `cat /root/blacklist/blacklistzeus | wc -l`
rm -rf /root/blacklist/blacklistzeus

# Insert any ip for your own testing pleasure
cat /root/blacklist/testbl >> /root/blacklist/blacklist
printf "\n Amount of lines in test list %s \n"  `cat /root/blacklist/testbl | wc -l`

#check amount of lines before sorting and cleanup
printf "\n Amount of lines in the combined blacklist before any cleanup is done %s \n"  `cat /root/blacklist/blacklist | wc -l`

printf "\n Sort ip's and make them unique"
cat /root/blacklist/blacklist | sort | uniq > /root/blacklist/blacklisttmp1

printf "\n Remove comments etc."
egrep -v '(^[[:space:]]*/|^[[:space:]]*#|^[[:space:]]*$)|/[0-9]|\:|/g' /root/blacklist/blacklisttmp1 > /root/blacklist/blacklisttmp2

# RD remove dakje-M from some IP's if it exists
sed 's/\r//g' /root/blacklist/blacklisttmp2 > /root/blacklist/blacklisttmp3

##Remove non ipv4 line
sed -rn '/((1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])/p' /root/blacklist/blacklisttmp3 > /root/blacklist/blacklist

#check amount of lines after sorting and clean-up
printf "\n Amount of lines in blacklisttmp2 %s \n"  `cat /root/blacklist/blacklisttmp2 | wc -l`

#check amount of lines after removing Windows line ends in downloaded files
printf "\n Amount of lines in blacklisttmp3 %s \n"  `cat /root/blacklist/blacklisttmp3 | wc -l`


#check amount of lines after removing non ipv4 line
printf "\n Amount of lines in final blacklist %s \n"  `cat /root/blacklist/blacklist | wc -l`

rm -rf /etc/shorewall/blacklist
cp /root/blacklist/blacklist /etc/shorewall/blacklist

shorewall refresh &>/dev/null

How to make NethServer an ideal server to set up on a VPS?
(Alessio Fattorini) #15

wikified, can you try again?


(jack) #16

Thank you!


(Joel Clendineng) #17

Any plans to implement custom lists in shorewall? I updated the init.d script, added 2 most lists pulled from local files, fail2ban blocks where I copy ips from the fail2ban list for perm blocking, and another file for any other ips I want to block. I also added code to delete the blrules file before running so the ips dont duplicate. Also added a shorewall update before the refresh, the update converts the blacklist ips into blrules file which the shorewall requires, as blacklist in interfaces has been made obsolete with an update to shorewall.


(Joel Clendineng) #18

I also, after I put this up, added a small part in the intro to enable chkconfig, so I could run a

chkconfig --level 345 block_script on

that basically makes it so that the script runs at startup, on a reboot it clears the previous blocklist file, reimports and updates/refreshes shorewall on a reboot. Also you can make a list of ips you want to block manually, fail2ban or I use ips from attacks blocked by suricata and add those to my block list, the idea being update the iptables every so often, Im trying to find a way to add ips without running the block_script every time I want to add ips but I will update when I simplify it. For now I have tested it with a couple ips of my own and can successfully block them that way.


(Joel Clendineng) #19

Edit 2, chkconfig was a horrible idea because who the heck needs to reload on reboot?? Instead I have a cron job to run 2 scripts, 1 for a full shorewall update that runs every week and 1 that only pulls any ips that I add to a specific file. Much better, and it doesnt need called as a service.

First here is the weekly full refresh/update of shorewall, it deletes the blrules file in etc/shorewall and forces a full update from all the sources, preventing duplicate ips in blrules.

Second one here is the daily, only pulls from 1 file, my list of ips I can add if I wish to add any, this does not delete the blrules file, only adds the ips from the file at the end and runs a refresh/updats on shorewall.

Finally in etc/crontab I added this to make them run at 12:30 in the morning daily and weekly respectively.

[details=Summary]"## cron for bl_full"
30 0 * * 7 root /root/bl_full

"## cron for bl_incremental"
30 0 * * * root /root/bl_incremental
(remove “” the reply makes “##” bold)
[/details]


(jack) #20

Added 2 more blocklist:
EmergingThreats.net Command and Control IPs
EmergingThreats.net compromised hosts

Updated script on the first post.

For those interested in a better way to crate a blocklist on centos/debian take a look at:


i wrote this script using ipset and iprange to compact the resulting blocklist as possible in order to increase the performance, i think can be easily adapted on nethserver (may be the next update :wink: )