Thread Shield: Updating and Enabling of IP-Backlist failed

After months of trouble-free service, CRON fails to update the IP-Blacklist.

Cron sleep $(( ( RANDOM % 60 ) )); /usr/share/nethserver-blacklist/download ipsets [ERROR] Can't update blacklist repository: fetch failed

Also, when activating the service, an error occurs recently.

[root@ns-srv01 ~]# echo '{"status":"enabled","Url":"https://github.com/firehol/blocklist-ipsets.git","Whitelist":["xx.xxx.xxx.xxx","yy.yyy.yyy.yyy","192.168.0.0/16","169.254.0.0/16","172.16.0.0/16","224.0.0.0/24","239.0.0.0/24"],"Categories":["blocklist_de_bots","blocklist_de_bruteforce","botscout","botscout_1d","botscout_30d","botscout_7d","cleantalk","cleantalk_1d","cleantalk_30d","cleantalk_7d","cleantalk_new","cleantalk_new_1d","cleantalk_new_30d","cleantalk_new_7d","cleantalk_updated","cleantalk_updated_30d","dshield","dshield_1d","et_block","et_botcc","et_compromised","et_dshield","firehol_level1","firehol_level2","firehol_webclient"]}' | /usr/bin/setsid /usr/bin/sudo /usr/libexec/nethserver/api/nethserver-blacklist/ipsets/update | jq

{
“steps”: 4,
“pid”: 21796,
“args”: “ipsets”,
“event”: “nethserver-blacklist-save”
}
{
“step”: 1,
“pid”: 21796,
“action”: “S05generic_template_expand”,
“event”: “nethserver-blacklist-save”,
“state”: “running”
}
{
“progress”: “0.25”,
“time”: “0.107986”,
“exit”: 0,
“event”: “nethserver-blacklist-save”,
“state”: “done”,
“step”: 1,
“pid”: 21796,
“action”: “S05generic_template_expand”
}
{
“step”: 2,
“pid”: 21796,
“action”: “S20nethserver-blacklist-conf”,
“event”: “nethserver-blacklist-save”,
“state”: “running”
}
{
“progress”: “0.50”,
“time”: “0.159987”,
“exit”: 256,
“event”: “nethserver-blacklist-save”,
“state”: “done”,
“step”: 2,
“pid”: 21796,
“action”: “S20nethserver-blacklist-conf”
}
{
“step”: 3,
“pid”: 21796,
“action”: “S30firewall-adjust”,
“event”: “nethserver-blacklist-save”,
“state”: “running”
}
{
“progress”: “0.75”,
“time”: “3.138317”,
“exit”: 0,
“event”: “nethserver-blacklist-save”,
“state”: “done”,
“step”: 3,
“pid”: 21796,
“action”: “S30firewall-adjust”
}
{
“step”: 4,
“pid”: 21796,
“action”: “S90adjust-services”,
“event”: “nethserver-blacklist-save”,
“state”: “running”
}
{
“progress”: “1.00”,
“time”: “0.310871”,
“exit”: 0,
“event”: “nethserver-blacklist-save”,
“state”: “done”,
“step”: 4,
“pid”: 21796,
“action”: “S90adjust-services”
}
{
“pid”: 21796,
“status”: “failed”,
“event”: “nethserver-blacklist-save”
}
{
“id”: “1615135040”,
“type”: “EventFailed”,
“message”: “See /var/log/messages”

[root@ns-srv01 ~]# cat /var/log/messages | grep FAILED

Mar 7 17:22:05 ns-srv01 esmith::event[21187]: Action: /etc/e-smith/events/nethserver-blacklist-save/S20nethserver-blacklist-conf FAILED: 1 [0.171913]
Mar 7 17:22:10 ns-srv01 esmith::event[21187]: Event: nethserver-blacklist-save FAILED

How can I analyze and fix this?

Sincerely, Marko

I tried to uninstall/reinstall Thread Shield.

…same behavior and the former config and all logs still exists.
How can I completely uninstall Thread Shield?

I had the same problem but when enabling that… i changed my dns to google ones and it worked fine, i used use openDNS and still use them… but had that trouble once… at least it looks like!

1 Like

Thank you for your hint. But I use google-DNS permanently.
Sincerely, Marko

Anyone else has any ideas?

How can I completely uninstall Thread Shield?

Sincerely, Marko

Run:

/usr/share/nethserver-blacklist/download --debug ipsets

1 Like
[root@ns-srv01 ~]# /usr/share/nethserver-blacklist/download --debug ipsets

[DEBUG] Cloning repository
Cloning into ‘/usr/share/nethserver-blacklist/–debug’…
remote: Enumerating objects: 1470, done.
remote: Counting objects: 100% (1470/1470), done.
remote: Compressing objects: 100% (826/826), done.
remote: Total 1470 (delta 650), reused 1381 (delta 643), pack-reused 0
Receiving objects: 100% (1470/1470), 25.26 MiB | 4.72 MiB/s, done.
Resolving deltas: 100% (650/650), done.
[root@ns-srv01 ~]#

The error persists.

asked again: How can I completely uninstall Thread Shield?

@filippo_carletti and cc: @mrmarkuz

Hello,
is there really no way to completely remove thread shield and set it up from scratch?
best regards, Marko

It’s a module… remove it as usual. For deleting configuration o reset it, i don’t know if there’s any way. @capote are you trying to emulate cron?

Yes I did, same result.

I’d remove nethserver-blacklist and dependencies.
Check what’s going to be removed before uninstalling.

yum autoremove nethserver-blacklist

Maybe you need to manually remove the cron job in /etc/cron.d/nethserver-blacklist

2 Likes

Remove also /usr/share/nethserver-blacklist

3 Likes

that was the trick - Thank you

1 Like

Hi, I started to have the same problem yesterday: after weeks without any problem, suddenly I started to receive the following emails:
[ERROR] Can't update blacklist repository: fetch failed

I’m using firehol/block-list ipsets and it seems that at some point an IP address from github (140.82.121.4) was included in some of the ipsets (specifically blocklist_net_ua and firehol_level4).

To solve it, from nethserver terminal I did:
ipset add bl-whitelist 140.82.121.4
/usr/share/nethserver-blacklist/download ipsets

I had to do that from the terminal as adding the IP address to the whitelist from the web GUI and even removing the category complained about the blacklist repository, maybe because it was trying to download the ipsets before applying changes.

After that, I finally added github IP address to the whitelist via web GUI and everyting worked fine.

1 Like
/usr/share/nethserver-blacklist/download --debug ipsets
DEBUG] Pulling changes
DEBUG] Repository have been updated: reloading --debug
/usr/share/nethserver-blacklist/download: line 123: /usr/share/nethserver-blacklist/load---debug: No such file or directory

Using blacklist from:
firehol git

Ip blacklist is not updating, currently its old for about 5 days.
Trhread Shield v 1.1.7

[EDITED]

Sorry, my bad, the --debug option must be the last option.
The right command is /usr/share/nethserver-blacklist/download dnss --debug.

[DEBUG] Pulling changes
[DEBUG] Repository have been updated: reloading ipsets
[DEBUG] Resetting ipset bl-whitelist
[DEBUG] Resetting ipset bl-alienvault_reputation
[DEBUG] Resetting ipset bl-ISO_country_code_za
[DEBUG] Creating global whitelist
[DEBUG] Creating ipset bl-alienvault_reputation
[DEBUG] Creating ipset bl-ISO_country_code_za

You could also use bash -x.
Here’s the output of a working download for reference:

[root@ns7-com nethserver-blacklist]# bash -x /usr/share/nethserver-blacklist/download ipsets
+ TYPE=ipsets
+ PROP=blacklist
+ case $TYPE in
+ PROP=blacklist
++ /sbin/e-smith/config getprop blacklist Url
+ URL=https://github.com/firehol/blocklist-ipsets.git
++ /sbin/e-smith/config getprop subscription SystemId
+ SYSTEM_ID=
++ /sbin/e-smith/config getprop subscription Secret
+ SYSTEM_SECRET=
+ DEST_DIR=/usr/share/nethserver-blacklist/ipsets
+ DEBUG=0
++ getopt -o d --long debug -- ipsets
+ options=' -- '\''ipsets'\'''
+ '[' 0 -eq 0 ']'
+ eval set -- ' -- '\''ipsets'\'''
++ set -- -- ipsets
+ true
+ case "$1" in
+ shift
+ break
+ '[' -z https://github.com/firehol/blocklist-ipsets.git ']'
+ mkdir -p /usr/share/nethserver-blacklist/ipsets
++ echo https://github.com/firehol/blocklist-ipsets.git
++ grep ://
++ sed '-es,^\(.*://\).*,\1,g'
+ proto=https://
++ echo github.com/firehol/blocklist-ipsets.git
+ url=github.com/firehol/blocklist-ipsets.git
+ auth=
+ [[ ! -z '' ]]
+ quiet=
+ '[' 0 -eq 0 ']'
+ quiet=' --quiet '
+ '[' '!' -d /usr/share/nethserver-blacklist/ipsets/.git ']'
+ opts='--git-dir=/usr/share/nethserver-blacklist/ipsets/.git --work-tree=/usr/share/nethserver-blacklist/ipsets'
+ debug 'Pulling changes'
+ '[' 0 -eq 1 ']'
+ git --git-dir=/usr/share/nethserver-blacklist/ipsets/.git --work-tree=/usr/share/nethserver-blacklist/ipsets fetch --all
+ '[' 0 -gt 0 ']'
+ git --git-dir=/usr/share/nethserver-blacklist/ipsets/.git --work-tree=/usr/share/nethserver-blacklist/ipsets reset --hard origin/master
+ '[' 0 -eq 0 ']'
+ debug_flag=
+ '[' 0 -eq 1 ']'
+ debug 'Repository have been updated: reloading ipsets'
+ '[' 0 -eq 1 ']'
+ exec /usr/share/nethserver-blacklist/load-ipsets --reload
[root@ns7-com nethserver-blacklist]# echo $?
0
2 Likes

/usr/share/nethserver-blacklist/download ipsets --debug
[DEBUG] Pulling changes
[DEBUG] Repository have been updated: reloading ipsets
[DEBUG] Resetting ipset bl-whitelist
[DEBUG] Resetting ipset bl-blocklist_de_apache
[DEBUG] Resetting ipset bl-blocklist_de_bots
[DEBUG] Resetting ipset bl-blocklist_de_bruteforce
[DEBUG] Resetting ipset bl-blocklist_de_imap
[DEBUG] Resetting ipset bl-blocklist_de_mail
[DEBUG] Resetting ipset bl-dshield_top_1000
[DEBUG] Resetting ipset bl-firehol_abusers_1d
[DEBUG] Resetting ipset bl-spamhaus_drop
[DEBUG] Resetting ipset bl-spamhaus_edrop
[DEBUG] Creating global whitelist
[DEBUG] Creating ipset bl-blocklist_de_apache
[DEBUG] Creating ipset bl-blocklist_de_bots
[DEBUG] Creating ipset bl-blocklist_de_bruteforce
[DEBUG] Creating ipset bl-blocklist_de_imap
[DEBUG] Creating ipset bl-blocklist_de_mail
[DEBUG] Creating ipset bl-dshield_top_1000
[DEBUG] Creating ipset bl-firehol_abusers_1d
[DEBUG] Creating ipset bl-spamhaus_drop
[DEBUG] Creating ipset bl-spamhaus_edrop

But, at web gui still showing last update was 5 days ago.

if the Maintainer of the list does not provide anything new, there can be no new list.

@mrmarkuz

Exactly my problem - I think there is a larger bug in the firewall