It appears I'm having a little issue with Threat Shield

NethSecurity
,
1

If you follow my steps here and look at the active sources you’ll kinda see why I don’t seem to have any control of TS ip or dns.


root@nsecurity01:~# /etc/init.d/adblock status
::: adblock runtime information
  + adblock_status  : enabled
  + adblock_version : 4.1.5
  + blocked_domains : 772056
  + active_sources  : malware_lvl2, malware_privacy_lvl2
  + dns_backend     : dnsmasq (-), /tmp/dnsmasq.d
  + run_utils       : download: /usr/libexec/wget-ssl, sort: /usr/libexec/sort-coreutils, awk: /usr/bin/gawk
  + run_ifaces      : trigger: -, report: -
  + run_directories : base: /tmp, backup: /tmp/adblock-Backup, report: /tmp/adblock-Report, jail: /tmp
  + run_flags       : backup: ✔, flush: ✘, force: ✔, search: ✘, report: ✘, mail: ✘, jail: ✘
  + last_run        : restart, 0m 31s, 2003/1631/1670, 2024-08-16T18:56:41-07:00
  + system          : Shuttle Inc SG41, NethSecurity 8-23.05.4-ns.1.2.0 r24012-d8dd03c46f
root@nsecurity01:~# echo '{"blocklist": "malware_privacy_lvl2", "enabled": false}' | /usr/libexec/rpcd/ns.threatshield call dns-edit-blocklist | jq
{
  "message": "success"
}
root@nsecurity01:~# uci commit adblock && service adblock restart
root@nsecurity01:~# /etc/init.d/adblock status
::: adblock runtime information
  + adblock_status  : running
  + adblock_version : 4.1.5
  + blocked_domains : 0
  + active_sources  : adaway, adguard, disconnect, malware_lvl2, yoyo
  + dns_backend     : dnsmasq (-), /tmp/dnsmasq.d
  + run_utils       : download: /usr/libexec/wget-ssl, sort: /usr/libexec/sort-coreutils, awk: /usr/bin/gawk
  + run_ifaces      : trigger: -, report: -
  + run_directories : base: /tmp, backup: /tmp/adblock-Backup, report: /tmp/adblock-Report, jail: /tmp
  + run_flags       : backup: ✔, flush: ✘, force: ✔, search: ✘, report: ✘, mail: ✘, jail: ✘
  + last_run        : -
  + system          : Shuttle Inc SG41, NethSecurity 8-23.05.4-ns.1.2.0 r24012-d8dd03c46f
root@nsecurity01:~# /etc/init.d/adblock status
::: adblock runtime information
  + adblock_status  : running
  + adblock_version : 4.1.5
  + blocked_domains : 0
  + active_sources  : adaway, adguard, disconnect, malware_lvl2, yoyo
  + dns_backend     : dnsmasq (-), /tmp/dnsmasq.d
  + run_utils       : download: /usr/libexec/wget-ssl, sort: /usr/libexec/sort-coreutils, awk: /usr/bin/gawk
  + run_ifaces      : trigger: -, report: -
  + run_directories : base: /tmp, backup: /tmp/adblock-Backup, report: /tmp/adblock-Report, jail: /tmp
  + run_flags       : backup: ✔, flush: ✘, force: ✔, search: ✘, report: ✘, mail: ✘, jail: ✘
  + last_run        : -
  + system          : Shuttle Inc SG41, NethSecurity 8-23.05.4-ns.1.2.0 r24012-d8dd03c46f
root@nsecurity01:~# /etc/init.d/adblock status
::: adblock runtime information
  + adblock_status  : enabled
  + adblock_version : 4.1.5
  + blocked_domains : 733556
  + active_sources  : malware_lvl2
  + dns_backend     : dnsmasq (-), /tmp/dnsmasq.d
  + run_utils       : download: /usr/libexec/wget-ssl, sort: /usr/libexec/sort-coreutils, awk: /usr/bin/gawk
  + run_ifaces      : trigger: -, report: -
  + run_directories : base: /tmp, backup: /tmp/adblock-Backup, report: /tmp/adblock-Report, jail: /tmp
  + run_flags       : backup: ✔, flush: ✘, force: ✔, search: ✘, report: ✘, mail: ✘, jail: ✘
  + last_run        : restart, 0m 55s, 2003/1639/1679, 2024-08-16T20:36:19-07:00
  + system          : Shuttle Inc SG41, NethSecurity 8-23.05.4-ns.1.2.0 r24012-d8dd03c46f
2

And not to be confused with TS Adblock the enable and disable instructions are the same here; ns-threat_shield | NethSecurity

And where is all the community ad blocking lists, I don’t see anything here but looking at the status outputs above the sources must be somewhere because they do list out even if they’re not consistently applied.


root@nsecurity01:~# /etc/init.d/adblock status
::: adblock runtime information
  + adblock_status  : enabled
  + adblock_version : 4.1.5
  + blocked_domains : 0
  + active_sources  : -
  + dns_backend     : dnsmasq (-), /tmp/dnsmasq.d
  + run_utils       : download: /usr/libexec/wget-ssl, sort: /usr/libexec/sort-coreutils, awk: /usr/bin/gawk
  + run_ifaces      : trigger: -, report: -
  + run_directories : base: /tmp, backup: /tmp/adblock-Backup, report: /tmp/adblock-Report, jail: /tmp
  + run_flags       : backup: ✔, flush: ✘, force: ✔, search: ✘, report: ✘, mail: ✘, jail: ✘
  + last_run        : reload, 0m 3s, 2003/1700/1739, 2024-08-16T17:15:24-07:00
  + system          : Shuttle Inc SG41, NethSecurity 8-23.05.4-ns.1.2.0 r24012-d8dd03c46f
root@nsecurity01:~# /usr/libexec/rpcd/ns.threatshield call dns-list-blocklist | jq
{
  "data": [
    {
      "name": "malware_lvl2",
      "type": "community",
      "enabled": true,
      "confidence": 8,
      "description": "Threat Intelligence Feed"
    },
    {
      "name": "malware_privacy_lvl2",
      "type": "community",
      "enabled": false,
      "confidence": 8,
      "description": "Multi Light: malware & privacy"
    },
    {
      "name": "malware_privacy_lvl3",
      "type": "community",
      "enabled": false,
      "confidence": 6,
      "description": "Multi Normal: malware and privacy"
    },
    {
      "name": "malware_privacy_lvl4",
      "type": "community",
      "enabled": false,
      "confidence": 5,
      "description": "Multi Pro: malware and privacy"
    },
    {
      "name": "adult",
      "type": "community",
      "enabled": false,
      "confidence": -1,
      "description": "Adult"
    },
    {
      "name": "doh_vpn_tor_proxy",
      "type": "community",
      "enabled": false,
      "confidence": -1,
      "description": "Prevent DNS bypass"
    },
    {
      "name": "gambling",
      "type": "community",
      "enabled": false,
      "confidence": -1,
      "description": "Gambling"
    },
    {
      "name": "piracy",
      "type": "community",
      "enabled": false,
      "confidence": -1,
      "description": "Piracy"
    }
  ]
}

This has been quite the time sink and I haven’t even tried to copy over any other configuration from my current ngfw, not port forwards, nothing.

3

Threat Shield IP is configured via web UI, see also Threat shield IP — NethSecurity documentation

Threat Shield DNS needs to be configured via CLI, see Threat Shield DNS — NethSecurity documentation

I guess you wanted to enable “malware_privacy_lvl2” but you didn’t set “enabled” to true. Here’s the right command:

echo '{"blocklist": "malware_privacy_lvl2", "enabled": true}' | /usr/libexec/rpcd/ns.threatshield call dns-edit-blocklist | jq

Threat Shield needs to be enabled in the UI under Threat Shield/Settings, then the block lists should be there:

grafik
grafik1888×1439 259 KB

4

I wanted that source back off and that’s why I set the command to false.

The first status check should two malware source turned on, that’s what I did following the instructions, that illustrated in my first status call, it shows 772 thousand odd domains.
I then disabled one of the malware sources with false.
The next status check shows 4 sources that are not in the sources list and shows running with 0 domains.
Another status check about 30 sec later shows the same.
A third status check about 30 sec later shows status enabled instead of running, 733 thousand domains, and only the single malware source I left enabled. So what happened to the other 4 sources that were listed in the previous two status checks?

Do you really think I was able to post all the above various commands without referencing the docs you posted.

Anyone who’s running a router / filtering appliance for any network should be able to definitively see the applied settings on a consistent basis.

At this point I’m looking at a couple of websites on a client behind the Nsecurity that clearly demonstrate there is little or no blocking going on.

Another issue with blocking traffic is sites will be broken because elements necessary to the site loading will be blocked. The log does not show clear blocks and does not show it in any way that can be quickly addressed for a bypass and I say this from running and administrating many ngfw installs that make it clear and easy to manage traffic control.

5

It seems I completely misunderstood your post. Let me give it another try…

I can confirm that only the Threat shield DNS sources remain. After a service restart all sources are listed but after some time the Threat shield IP sources are removed.
Threat Shield IP uses banip where Threat Shield DNS uses adblock so it makes sense that the Threat Shield IP sources are removed from adblock.

From the Threat shield DNS admin manual:

Even if not recommended, it’s possible to use Adblock without Threat Shield DNS. For more detailed configuration options, please refer to the developer manual.

Maybe the manual configuration of ts-dns produced issues as it’s not recommended?

I tested Threat Shield DNS with the gambling list and bwin.com or interwetten.com was blocked immediately.

I couldn’t reproduce, could you please share not working sites/blocklists?
I got some partially loaded site fragments immediately after the Threat Shield service was restarted but then it worked.

You’re right. I wasn’t able to find a logfile showing the queries or to enable the DNS report option.
@giacomo do you know how to show the Threat Shield blockings? Maybe we need to enable adblock debug logging?

6

There are so many elements of Threat Shield that are making little sense to me. I’ve honestly lost track.

During my tests, msn.com was completely blocked, the browser itself gave an error message it couldn’t be found. That was TS IP before I even started messing with TS Adblock. I believe it was the TS IP adaway category. Or perhaps it was yoyo. I found it in the sources lists somewhere either on the nsecurity GitHub readme or on wrt’s site. Yet somehow nothing I have enabled at this time is blocking doubleclick … how in today’s world does a so call ad blocklist not have doubleclick?

If you look at theverge.com the ads there are still displayed.

This makes no sense to me.

Let’s look at what’s current;



root@nsecurity01:~# /etc/init.d/adblock status
::: adblock runtime information
  + adblock_status  : enabled
  + adblock_version : 4.1.5
  + blocked_domains : 669235
  + active_sources  : malware_lvl2
  + dns_backend     : dnsmasq (-), /tmp/dnsmasq.d
  + run_utils       : download: /usr/libexec/wget-ssl, sort: /usr/libexec/sort-coreutils, awk: /usr/bin/gawk
  + run_ifaces      : trigger: -, report: -
  + run_directories : base: /tmp, backup: /tmp/adblock-Backup, report: /tmp/adblock-Report, jail: /tmp
  + run_flags       : backup: ✔, flush: ✘, force: ✔, search: ✘, report: ✘, mail: ✘, jail: ✘
  + last_run        : reload, 0m 28s, 2003/1627/1670, 2024-08-17T13:03:10-07:00
  + system          : Shuttle Inc SG41, NethSecurity 8-23.05.4-ns.1.2.0 r24012-d8dd03c46f
root@nsecurity01:~# /etc/init.d/banip status
::: banIP runtime information
  + status            : active (nft: ✔, monitor: ✔)
  + version           : 1.0.0-5
  + element_count     : 268148
  + active_feeds      : allowlistv4MAC, allowlistv6MAC, allowlistv4, allowlistv6, becyberv4, iblockadsv4, cinsscorev4, ipthreatv4, iblockspyv4, urlvirv4, webclientv4, threatviewv4, yoyov4, yoyov6, blocklistv4MAC, blocklistv6MAC, blocklistv4, blocklistv6
  + active_devices    : wan: eth1 / wan-if: wan, wan / vlan-allow: - / vlan-block: -
  + active_uplink     : 192.168.23.58/24
  + nft_info          : priority: -100, policy: memory, loglevel: warn, expiry: 30m, limit (icmp/syn/udp): 10/10/100
  + run_info          : base: /tmp, backup: /tmp/banIP-backup, report: /tmp/banIP-report
  + run_flags         : auto: ✘, proto (4/6): ✔/✔, log (pre/inp/fwd/lan): ✘/✘/✘/✘, dedup: ✔, split: ✘, custom feed: ✘, allowed only: ✘
  + last_run          : action: reload, log: tail, fetch: curl, duration: 1m 4s, date: 2024-08-17 20:45:54
  + system_info       : cores: 2, memory: 1685, device: Shuttle Inc SG41, NethSecurity 8-23.05.4-ns.1.2.0 r24012-d8dd03c46f

Somewhere in there msn.com is completely blocked.
Adblock shows the 1 source I have enabled out of like 9 or so. They’re all listed in my previous post. One particular point about this is interesting, the “Adblock” source list used by TS DNS doesn’t seem to have a source that blocks ads.

So, it appears that if I want to block advertising on the network I must use TS IP. Well, if you look in the gui the source names do not seem to match the names listed in the status call, there are two “malware related IPs” I have selected and I couldn’t tell you which of those match up with the sources listed in the banip status check.

So I’m going to stop here because of the shear number of things that are quite unclear about attempting to setup and administrate a threat module in a firewall.

Except I am going to touch on logging, ie; being able to identify quickly and easily what is or is not being blocked for any aspect of any site being accessed is of paramount importance for a threat module on a network’s firewall.

7

Also… this, despite all the traffic I’ve pushed through the fw in the last couple of hours with, as you can see in the status calls above, threatshield ip enabled and threat shield dns enabled and actively blocking msn.com, does not work… and… why is this little bragging panel even necessary?

Screenshot 2024-08-18 at 6.01.19 PM

What would be helpful is say, top blocked domains, or… top blocked categories, something tangible, something that can be acted upon.

8

To search if an IP is in a TS IP blocklist:

/etc/init.d/banip search <IP>

To check if a domain is in TS DNS blocklist:

root@keepout:~# /etc/init.d/adblock query msn.com
:::
::: domain 'msn.com' in active blocklist
:::
  + ac3.msn.com
  + ads.msn.com
  + ads1.msn.com
  + ads2.msn.com
  + adsyndication.msn.com
  + anrfrm.msn.com
  + metric.appex-rf.msn.com
  + arc1.msn.com
  + confiant.msn.com
  + [...]

:::
::: domain 'msn.com' in backups and black-/whitelist
:::
  + adb_list.malware_lvl2.gz      anrfrm.msn.com
  + adb_list.malware_privacy_lvl2.gzac3.msn.com
  + adb_list.malware_privacy_lvl2.gzads.msn.com
  + adb_list.malware_privacy_lvl2.gzads1.msn.com
  + adb_list.malware_privacy_lvl2.gz[...]

There may be some blocklists that block too much/less for your need. Adaway and agguard should be good to block ads.

Where are the ads?

grafik
grafik1920×1584 150 KB

AFAIK it just shows the count of IPs blocked from WAN side. Threat shield IP (banip) also blocks bad IPs from www.

1 Like
9

Adblock has way to get a little report about DNS queries: it starts a tcpdump file and then, when requested, it generates the report.
Still the dump is rotated when reaches a certain limit, which is good enough only for installations with very little clients.

If you want to see all queries, you can enable the logging:

uci set dhcp.@dnsmasq[0].logqueries=1
uci commit dhcp
/etc/init.d/dnsmasq restart

Please bear in mind that it will be very noisy, this is what it produces when executing curl https://google.it:

Aug 19 09:40:31 pippo dnsmasq[1]: 1 127.0.0.1/38121 query[A] google.it from 127.0.0.1
Aug 19 09:40:31 pippo dnsmasq[1]: 1 127.0.0.1/38121 forwarded google.it to 192.168.100.1
Aug 19 09:40:31 pippo dnsmasq[1]: 1 127.0.0.1/38121 forwarded google.it to 10.10.0.1
Aug 19 09:40:31 pippo dnsmasq[1]: 1 127.0.0.1/38121 forwarded google.it to 192.168.122.1
Aug 19 09:40:31 pippo dnsmasq[1]: 2 ::1/38121 query[A] google.it from ::1
Aug 19 09:40:31 pippo dnsmasq[1]: 3 127.0.0.1/38121 query[AAAA] google.it from 127.0.0.1
Aug 19 09:40:31 pippo dnsmasq[1]: 3 127.0.0.1/38121 forwarded google.it to 192.168.100.1
Aug 19 09:40:31 pippo dnsmasq[1]: 3 127.0.0.1/38121 forwarded google.it to 10.10.0.1
Aug 19 09:40:31 pippo dnsmasq[1]: 3 127.0.0.1/38121 forwarded google.it to 192.168.122.1
Aug 19 09:40:31 pippo dnsmasq[1]: 4 ::1/38121 query[AAAA] google.it from ::1
Aug 19 09:40:32 pippo dnsmasq[1]: 1 127.0.0.1/38121 reply google.it is 142.251.209.35
Aug 19 09:40:32 pippo dnsmasq[1]: 2 ::1/38121 reply query is duplicate
Aug 19 09:40:32 pippo dnsmasq[1]: 3 127.0.0.1/38121 reply google.it is 2a00:1450:4002:411::2003
Aug 19 09:40:32 pippo dnsmasq[1]: 4 ::1/38121 reply query is duplicate
1 Like