IP"table"s full?

[gpunk (remix)]

Nethserver 7.8.2003
Module ip/dns-shield

Do we need to modify sysctl/kernel params maybe ?
Yes I have enabled ALL ip and dns cat/rules …

Sep 26 19:53:50 admin.bureau.cash kernel: Set bl-stopforumspam_90d is full, maxelem 131072 reached
Sep 26 19:53:50 admin.bureau.cash esmith::event[16575]: ipset v7.1: Error in line 131073: Hash is full, cannot add more elements
Sep 26 19:53:50 admin.bureau.cash esmith::event[16575]: [WARNING] Can't load bl-stopforumspam_90d ipset
Sep 26 19:53:52 admin.bureau.cash esmith::event[16575]: Action: /etc/e-smith/events/nethserver-blacklist-save/S20nethserver-blacklist-conf SUCCESS [38.424376]
Sep 26 19:53:52 admin.bureau.cash esmith::event[20594]: Event: nethserver-firewall-base-save nethserver-blacklist-save
Sep 26 19:53:52 admin.bureau.cash esmith::event[20594]: Action: /etc/e-smith/events/nethserver-firewall-base-save/S02providers-cleanup SUCCESS [0.132783]

[dnutan (Marc)]

Seems the same problem described here regarding ipsets (solution would be to cherry pick a more reduced set of lists):

Threat shield: Hash is full, cannot add more elements

Each set can actually have max 131072 elements: https://github.com/NethServer/nethserver-blacklist/blob/master/root/usr/share/nethserver-blacklist/load-ipsets#L111 For now the limit is hard-coded. Extending such limit, could eventually bring to performance loss. Also bear in mind that many ipsets has duplicated IPs. I think that you did the right choice: do not use extremely large ipsets.

[gpunk (remix)]

Well this is a little against the spirit of opensource etc I think,

I also think it is always best to leave the maximum freedom/choice to the user, as opposite to closed source software and OSs …

Because in my case, for example, I have 128G of ram, 32CPUs … do you think this machine cant handle more ? especially in the era of gigabit NICs …

And of course a little warning as evrywhere else about abusing this parameter can be written … in red …

I will do some tests after modifying this paramter, an keep you posted – I might be wrong

[mrmarkuz (Markus Neuberger)]

There’s a nice howto about configuring blacklists.

[capote (Marko)]

@mrmarkuz

you will have same IPs multiple times because some lists are collections of other lists.

Is there no consolidation like with PiHole? PiHole filters duplicated entries and consolidates the list to unique entries.
Nobody can handle thousands auf list entries manually.

[mrmarkuz (Markus Neuberger)]

You are right, the IPs are consolidated in the ipset, where duplicates are not added. I’m going to edit my post.

[gpunk (remix)]

Well, it is possible, a simple sort and uniq can do it with a shell , I did it lately with 4G file, I pulled out of it 200 email addresses it went down to a 4K file .

[capote (Marko)]

@gpunk

do it with a shell

Manual consolidation should not be necessary.

[gpunk (remix)]

I never said manually, but using a simple shell interpreter,
the one that runs most of unics scripts