Enabled Blacklists not visible in Cockpit

yummiweb · April 4, 2021, 4:48pm

Since Today i become many mails from Cron showing Blacklists errors like this:

[WARNING] Skipping invalid blacklist ‘proxyrss_7d’
[WARNING] Skipping invalid blacklist ‘proxyrss_1d’
[WARNING] Skipping invalid blacklist ‘zeus’
[WARNING] Skipping invalid blacklist ‘proxyrss_30d’
[WARNING] Skipping invalid blacklist ‘zeus_badips’
[WARNING] Skipping invalid blacklist ‘proxyrss’

Maybe this lists are temporary or no more reacheable or defective or something, so i want to disable this lists in IP blacklist.

I have tried this in Cockpit > Thread shield > IP blacklists section but the lists was not shown (anymore) via search field and not in the listing.

In /var/lib/nethserver/db/configuration file the (missed) blacklists was shown enabled as espected, so i decide to remove this lists by hand via editing this configuration file.

After “signal-event nethserver-blacklist-save” (or via Cockpit saving) no new Cron errors occured.

I think the IP blacklist section should shown all “enabled” lists entries and not only if the corresponding blacklists is found in the “master” blacklist-git (like GitHub - firehol/blocklist-ipsets: ipsets dynamically updated with firehol's update-ipsets.sh script).

Regards
yummiweb

stephdl · April 4, 2021, 5:31pm

this comes when you have enabled these blacklists but the blacklists are not more available in /usr/share/nethserver-blacklist/ipsets

try to download them again by

signal-event nethserver-blacklist-save ipsets

yummiweb · April 4, 2021, 6:05pm

After

signal-event nethserver-blacklist-save ipsets

the “missing” blacklists are not in /usr/share/nethserver-blacklist/ipsets and so not shown in the Cockpit > Thread shield > IP blacklists listing.

I understand (maybe), that the lists “gone” after an update in the corresponding git, but the removed lists should be visible (to disable) in Cockpit or removed straight in db/configuration.

It should be not necessary to remove this lists by hand in configuration db after the lists are “gone” in the git.

stephdl · April 4, 2021, 8:17pm

how to trade if it is a temporary missing blacklist, before to download we remove all the lists, then we fetch them

stephdl · April 4, 2021, 8:23pm

the relevant code

github.com

NethServer/nethserver-blacklist/blob/master/root/usr/share/nethserver-blacklist/load-ipsets#L107


file_netset="$BLDIR/$l".netset
file_ipset="$BLDIR/$l".ipset
name=$(echo "bl-$l" | cut -c 1-31) # limit to 31 chars
if [ -f $file_netset ]; then
    file=$file_netset
    type="hash:net"
elif [ -f $file_ipset ]; then
    file=$file_ipset
    type="hash:ip"
else
    warning "Skipping invalid blacklist '$l'"
    continue
fi
debug "Creating ipset $name"
# Create ipset and load IPs
/sbin/ipset create -exist $name $type maxelem ${MAXELEM:-131072}
if [ $? -gt 0 ]; then
    warning "Can't create $name ipset"
fi
cat $file | sed '/^#/d; /^$/d' | sed "s/^/add $name /" | /sbin/ipset restore
if [ $? -gt 0 ]; then

capote · April 4, 2021, 8:24pm

I had the same problem today.
A reset /usr/share/nethserver-blacklist/download dnss --debug doesn’t help
I uninstalled …

yum autoremove nethserver-blacklist

rm -f -r /usr/share/nethserver-blacklist

…and reinstalled thread shield.

stephdl · April 4, 2021, 8:27pm

Not sure your issue is relative to this one @capote, here the remote git has changed the blacklist content IIUC

dnutan · April 4, 2021, 8:53pm

What about, in case of subscribed blacklist are missing or not reachable:

keep sending the warning email;
when loading the list on UI, compare the blacklist config db against the available blacklists; if not there, mark theme as orphaned on the UI (maybe with a new “status” property). I haven’t checked the code, so maybe it is necessary to add the missing blacklist as fake… (if possible, or show them apart)
Let the admin keep/delete them.

capote · April 4, 2021, 9:07pm

It seems to me so, only reduced to the Zeus list.

Elleni · April 5, 2021, 10:41am

We also saw these messages regarding the zeus list recently. A check reveiled that this list was not activated anyway. What I then did was activating the list, and then deactivate it again. As I shortly had changed my upstream dns in pihole to filtered quad9 with dnssec and as I was not sure if this caused the problem I then modified it to unfiltered quad9 without dnssec in piHole . One of those two actions stopped the threatshield mails about the zeus list.

saitobenkei · April 6, 2021, 6:49am

Same problem here, i have some customers with threat shield enabled.

I have to modify manually every customers’ firewall configuration…

giacomo · April 6, 2021, 7:20am

I agree, this should be the correct behavior.
Let’s see if we can implement it.

stephdl · April 8, 2021, 2:56pm

Now the missing blacklist are show in the table to be removed if missing, kudo to @andre8244

stephdl · April 9, 2021, 6:15am

The final solution will be when the package will be tested and released, we are a tiny team, we need sysadmin to valid that the testing rpm is good without newer issues, please stay tuned.

stephdl · April 12, 2021, 4:18pm

please verify

giacomo · April 14, 2021, 9:55am

The fix has been released: excellent work @stephdl!