[SOLVED]Threat Shields "DNS Blacklist" fails on repo update

Support
1

I recently moved my repos from github to gitlab, and tried editing my upstream repo for the Nethserver DNS blocklist to the new repo.

In cockpit, I get the following error message:

Screenshot 2020-11-18 at 09.31.58

Journalctl output is as follows:

-- Logs begin at Tue 2020-11-03 06:20:25 EST. --
Nov 18 09:28:04 nethserver.triggeredstudios.com pihole-ftl[17605]: [2020-11-18 09:28:04.299 17605] Listening on Unix socket
Nov 18 09:28:04 nethserver.triggeredstudios.com pihole-ftl[17605]: [2020-11-18 09:28:04.299 17605] Changing /FTL-lock (16) to 0:0
Nov 18 09:28:04 nethserver.triggeredstudios.com dnsmasq[17605]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify dumpfile
Nov 18 09:28:04 nethserver.triggeredstudios.com pihole-ftl[17605]: [2020-11-18 09:28:04.299 17605] Changing /FTL-strings (16) to 0:0
Nov 18 09:28:04 nethserver.triggeredstudios.com dnsmasq[17605]: using only locally-known addresses for domain use-application-dns.net
Nov 18 09:28:04 nethserver.triggeredstudios.com pihole-ftl[17605]: [2020-11-18 09:28:04.299 17605] Changing /FTL-co
Nov 18 09:28:04 nethserver.triggeredstudios.com dnsmasq[17605]: using nameserver 8.8.4.4#53
Nov 18 09:28:04 nethserver.triggeredstudios.com dnsmasq[17605]: using nameserver 8.8.8.8#53
Nov 18 09:28:04 nethserver.triggeredstudios.com dnsmasq[17605]: using nameserver 127.0.0.1#53 for domain triggeredstudios.com
Nov 18 09:28:04 nethserver.triggeredstudios.com dnsmasq[17605]: read /etc/hosts - 14 addresses
Nov 18 09:29:18 nethserver.triggeredstudios.com dnsmasq[17605]: exiting on receipt of SIGTERM
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972] Using log file /var/log/pihole-FTL.log
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972] ########## FTL started! ##########
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972] FTL branch: master
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972] FTL version: v5.0
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972] FTL commit: 3d7c095
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972] FTL date: 2020-05-10 18:58:38 +0100
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972] FTL user: root
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972]    SOCKET_LISTENING: only local
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.669 19972]    AAAA_QUERY_ANALYSIS: Show AAAA queries
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    MAXDBDAYS: max age for stored queries is 7 days
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    RESOLVE_IPV6: Resolve IPv6 addresses
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    RESOLVE_IPV4: Resolve IPv4 addresses
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    DBINTERVAL: saving to DB file every minute
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    DBFILE: Using /etc/pihole/pihole-FTL.db
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    MAXLOGAGE: Importing up to 1.0 hours of log data
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    PRIVACYLEVEL: Set to 0
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    IGNORE_LOCALHOST: Show queries from localhost
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    BLOCKINGMODE: Null IPs for blocked domains
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    DBIMPORT: Importing history from database
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.670 19972]    PIDFILE: Using /var/run/pihole-FTL.pid
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972]    PORTFILE: Using /var/run/pihole-FTL.port
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972]    SOCKETFILE: Using /var/run/pihole/FTL.sock
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972]    SETUPVARSFILE: Using /etc/pihole/setupVars.conf
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972]    MACVENDORDB: Using /etc/pihole/macvendor.db
Nov 18 09:29:18 nethserver.triggeredstudios.com dnsmasq[19972]: started, version pi-hole-2.81 cachesize 150
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972]    GRAVITYDB: Using /etc/pihole/gravity.db
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972]    PARSE_ARP_CACHE: Active
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972]    CNAME_DEEP_INSPECT: Active
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972]    DELAY_STARTUP: No delay requested.
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972]    BLOCK_ESNI: Enabled, blocking _esni.{blocked domain}
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.671 19972] Finished config file parsing
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.672 19972] WARNING: Starting pihole-FTL as user root is not recommended
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.673 19972] Database version is 5
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.673 19972] Database successfully initialized
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972] New upstream server: 8.8.4.4 (0/128)
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972] Imported 7 queries from the long-term database
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]  -> Total DNS queries: 7
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]  -> Cached DNS queries: 2
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]  -> Forwarded DNS queries: 5
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]  -> Blocked DNS queries: 0
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]  -> Unknown DNS queries: 0
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]  -> Unique domains: 3
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]  -> Unique clients: 4
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]  -> Known forward destinations: 1
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972] WARN: Opening of setupVars.conf failed!
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]       Make sure it exists and is readable
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.674 19972]       Message: No such file or directory
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.675 19972] PID of FTL process: 19972
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.675 19972] Listening on port 4711 for incoming IPv4 telnet connections
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.676 19972] Listening on port 4711 for incoming IPv6 telnet connections
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.676 19972] Listening on Unix socket
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.676 19972] Changing /FTL-lock (16) to 0:0
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.676 19972] Changing /FTL-strings (16) to 0:0
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.677 19972] Changing /FTL-counters (16) to 0:0
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.677 19972] Changing /FTL-domains (16) to 0:0
Nov 18 09:29:18 nethserver.triggeredstudios.com dnsmasq[19972]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify dumpfile
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18 09:29:18.677 19972] Changing /FTL-clients (16) to 0:0
Nov 18 09:29:18 nethserver.triggeredstudios.com dnsmasq[19972]: using nameserver 8.8.4.4#53
Nov 18 09:29:18 nethserver.triggeredstudios.com pihole-ftl[19972]: [2020-11-18
Nov 18 09:29:18 nethserver.triggeredstudios.com dnsmasq[19972]: using nameserver 8.8.8.8#53
Nov 18 09:29:18 nethserver.triggeredstudios.com dnsmasq[19972]: using nameserver 127.0.0.1#53 for domain triggeredstudios.com
Nov 18 09:29:18 nethserver.triggeredstudios.com dnsmasq[19972]: read /etc/hosts - 14 addresses

config show ftl:

ftl=service
    Bypass=
    Categories=
    Roles=green,orange
    TCPPorts=1153
    UDPPorts=1153
    Url=https://github.com/ccsuttles/dns-community-blacklist
    access=green
    status=enabled

I can change the URL in cockpit back to the old github repo and everything works.

Both repos are public. Whether this is an issue with changing the repo, or with gitlab compatability i’m not sure, as logs don’t really reveal much to me.

Github repo: https://github.com/ccsuttles/dns-community-blacklist
Gitlab repo: https://gitlab.com/christophercsuttles/dns-community-blacklist

2
Output of the failed command 
#  echo '{"status":"enabled","Url":"https://gitlab.com/christophercsuttles/dns-community-blacklist","Bypass":[],"Categories":[],"Roles":[]}' | /usr/bin/setsid /usr/bin/sudo /usr/libexec/nethserver/api/nethserver-blacklist/dnss/update | jq
{
  "steps": 4,
  "pid": 14821,
  "args": "dnss",
  "event": "nethserver-blacklist-save"
}
{
  "step": 1,
  "pid": 14821,
  "action": "S05generic_template_expand",
  "event": "nethserver-blacklist-save",
  "state": "running"
}
{
  "progress": "0.25",
  "time": "0.082855",
  "exit": 0,
  "event": "nethserver-blacklist-save",
  "state": "done",
  "step": 1,
  "pid": 14821,
  "action": "S05generic_template_expand"
}
{
  "step": 2,
  "pid": 14821,
  "action": "S20nethserver-blacklist-conf",
  "event": "nethserver-blacklist-save",
  "state": "running"
}
{
  "progress": "0.50",
  "time": "1.074059",
  "exit": 256,
  "event": "nethserver-blacklist-save",
  "state": "done",
  "step": 2,
  "pid": 14821,
  "action": "S20nethserver-blacklist-conf"
}
{
  "step": 3,
  "pid": 14821,
  "action": "S30firewall-adjust",
  "event": "nethserver-blacklist-save",
  "state": "running"
}
{
  "progress": "0.75",
  "time": "2.382758",
  "exit": 0,
  "event": "nethserver-blacklist-save",
  "state": "done",
  "step": 3,
  "pid": 14821,
  "action": "S30firewall-adjust"
}
{
  "step": 4,
  "pid": 14821,
  "action": "S90adjust-services",
  "event": "nethserver-blacklist-save",
  "state": "running"
}
{
  "progress": "1.00",
  "time": "0.337736",
  "exit": 0,
  "event": "nethserver-blacklist-save",
  "state": "done",
  "step": 4,
  "pid": 14821,
  "action": "S90adjust-services"
}
{
  "pid": 14821,
  "status": "failed",
  "event": "nethserver-blacklist-save"
}
{
  "id": "1605733602",
  "type": "EventFailed",
  "message": "See /var/log/messages"
}

/var/log/messages excerpt:

Nov 18 21:16:42 server esmith::event[16946]: expanding /etc/pihole/dnsmasq.conf
Nov 18 21:16:42 server esmith::event[16946]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.094498]
Nov 18 21:16:43 server esmith::event[16946]: error: RPC failed; result=22, HTTP code = 404
Nov 18 21:16:43 server esmith::event[16946]: fatal: The remote end hung up unexpectedly
Nov 18 21:16:43 server esmith::event[16946]: [ERROR] Can't download blacklist repository
Nov 18 21:16:43 server esmith::event[16946]: Action: /etc/e-smith/events/nethserver-blacklist-save/S20nethserver-blacklist-conf FAILED: 1 [1.094413]
(...)
Nov 18 21:16:46 server esmith::event[16946]: Event: nethserver-blacklist-save FAILED

Same error happens trying git clone from command line.
Possible workaround/solution could be to increase the http buffer used by git:

git config --global http.postBuffer 52428800

Obs.: Changing http.postBuffer might also require to set up the Nginx configuration file for gitlab to accept larger body sizes for the client, by tuning the value of client_max_body_size.

However, there is a workaround if you have access to the Gitlab machine or to a machine in its network, and that is by making use of git bundle .

3

Changing the post buffer had no effect, but appending “.git” to the end of the gitlab url in the "Download URL " section in cockpit settings does work.

Example: https://gitlab.com/christophercsuttles/dns-community-blacklist.git

2 Likes