Crowdsec log flooded

MadPatrick · October 14, 2024, 2:33pm

NethServer Version: 8
Module: Crowdsec

Hi,

I know i’ve a lot of question, but i’m still struggling with NS packages

I see every 10 seconds this in the crowdsec log

2024-10-14T16:25:03+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:25:03Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:25:03 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 88.970784ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:25:13+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:25:13Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:25:13 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 82.66505ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:25:23+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:25:23Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:25:23 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 83.527731ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:25:33+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:25:33Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:25:33 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 87.866835ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:25:33+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:25:33Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:25:33 UTC] \"GET /v1/heartbeat HTTP/1.1 200 405.837µs \"crowdsec/v1.6.3-4851945a-docker\" \""
2024-10-14T16:25:43+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:25:43Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:25:43 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 96.75658ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:25:53+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:25:53Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:25:53 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 85.63575ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:26:03+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:26:03Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:26:03 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 98.351291ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:26:13+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:26:13Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:26:13 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 84.184317ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:26:23+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:26:23Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:26:23 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 86.079231ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:26:33+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:26:33Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:26:33 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 88.519125ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:26:33+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:26:33Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:26:33 UTC] \"GET /v1/heartbeat HTTP/1.1 200 469.344µs \"crowdsec/v1.6.3-4851945a-docker\" \""
2024-10-14T16:26:43+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:26:43Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:26:43 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 92.719564ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:26:53+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:26:53Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:26:53 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 96.696593ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:27:03+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:27:03Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:27:03 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 82.82194ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:27:13+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:27:13Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:27:13 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 92.125656ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:27:23+02:00 [1:crowdsec2:crowdsec2] time="2024-10-14T14:27:23Z" level=info msg="127.0.0.1 - [Mon, 14 Oct 2024 14:27:23 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 87.199172ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" \""
2024-10-14T16:27:25

Is this normal ?

When I google this i see more post, but no solution.

for example

github.com/crowdsecurity/cs-firewall-bouncer

`set-only` mode causes crowdsec-firewall-bouncer to flood crowdsec-firewall-bouncer.log

opened 08:31PM - 28 May 23 UTC

closed 04:00PM - 31 May 23 UTC

jarppiko

# Summary If `crowdsec-firewall-bouncer` uses `mode=nftables` and `set-only=t…rue`, metrics collection floods `crowdsec-firewall-bouncer.log` with lines like this every 10 seconds: ``` $ tail /var/log/crowdsec-firewall-bouncer.log time="28-05-2023 13:36:43" level=error msg="can't collect dropped packets for ipv4 from nft: while running /usr/sbin/nft -j list chain ip filter crowdsec-chain-input: exit status 1" time="28-05-2023 13:36:53" level=error msg="can't collect dropped packets for ipv4 from nft: while running /usr/sbin/nft -j list chain ip filter crowdsec-chain-input: exit status 1" time="28-05-2023 13:37:03" level=error msg="can't collect dropped packets for ipv4 from nft: while running /usr/sbin/nft -j list chain ip filter crowdsec-chain-input: exit status 1" time="28-05-2023 13:37:13" level=error msg="can't collect dropped packets for ipv4 from nft: while running /usr/sbin/nft -j list chain ip filter crowdsec-chain-input: exit status 1" time="28-05-2023 13:37:23" level=error msg="can't collect dropped packets for ipv4 from nft: while running /usr/sbin/nft -j list chain ip filter crowdsec-chain-input: exit status 1" ``` `set-only` mode is intended for more complex firewall setups (e.g. in routers) where Crowdsec's the automated firewall rules cannot be used (e.g. there are multiple interfaces and `crowdsec` should control only some of those). # Root cause This bug was probably introduced in #231 since the PR seemed to consider only "auto-mode" (`set-only=false`). `nftables.metrics.CollectMetrics()` (line 118) does not handle `set-only` mode since it assumes there is a `nftables` chain with name `chain-hook` (where `hook` is `prerouting`, `input`, `output`, `forward` or `ingress`) to query from. In `set-only` mode `crowdsec` manages only blocked IPs in a named `nftables` `set` and the `chain` param is not used. # Workaround It is not possible to disable metrics collection. Therefore, as a workaround an admin has to create/rename a chain according to an undocumented naming convention `$nftables.ipv4.chain-$nftables_hooks.hook` (referring to bouncer config file yaml fields here). For example, if `nftables` config in bouncer config looks like this: ``` # /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml ... nftables: ipv4: enabled: true set-only: true table: filter chain: existing-chain-without-its-input-postfix nftables_hooks: - input ``` Then there needs to be a `nftables` chain in the right table with name : `existing-chain-without-its-input-postfix-input`. # Solution options I could come up with couple of options how to solve this issue. 1. Add `disable_metrics` option 2. Disable metrics collection in `set-only` mode automatically 3. Add support for metrics collection in `set-only` mode # Technical considerations ## 1. Add `disable_metrics` bouncer config option This would require admin to manually fix the issue. Commit [9619b01](https://github.com/crowdsecurity/cs-firewall-bouncer/pull/282/commits/9619b0150cbb2d7cb025aa82e2e4069f6fceef8d) introduced this functionality, but in the same PR #282 the commit was later reversed ([d17dafa](https://github.com/crowdsecurity/cs-firewall-bouncer/pull/282/commits/d17dafa8657645f85c57a51781f2d101eb2b6d2e)). Maybe @mmetc knows why the change was reversed. ## 2. Disable metrics collection in `set-only` mode This can be done by editing one line in `collectDropped()` in `nftables/metrics.go`. ``` func (c *nftContext) collectDropped(path string, hooks []string) (int, int, int) { if c.conn == nil || c.setOnly { // "|| c.setOnly" added return 0, 0, 0 } ... ``` ## 3. Add support for metrics collection in `set-only` mode This should not be a big change. * Modify `collectDroppedPackets()` in `metrics.go` to take chain name directly instead of doing string concatenation magic within the function. * Modify `collectDropped()` in `metrics.go` to handle `set-only=true` mode: * `set-only=false`: create chain name here with the string concatenation magic * `set-only=true`: use chain name directly from bouncer config ### Drawback A drawback of this implementation is that it would collect metrics only from *one chain*. Collecting metrics from all chains / hooks in `set-only` mode would require a way to tell the bouncer what chains are being used. If `nftables_hooks` config option could be changed to contain actual chain names and the string concatenation could be scrapped, then the `nftables_hooks` config option could be used also in `set-only` mode to configure metrics collection right. *Disclaimer: I am the author of `set-only` mode*

LayLow · October 14, 2024, 5:35pm

I can confirm and it is a lot of noise, unless it is required to evaluate the log to block IP’s?

stephdl · October 15, 2024, 8:21am

Oct 15 10:19:28 R2-pve.rocky9-pve2.org crowdsec1[101347]: time=“2024-10-15T08:19:28Z” level=info msg=“127.0.0.1 - [Tue, 15 Oct 2024 08:19:28 UTC] "GET /v1/decisions/stream HTTP/1.1 200 4.951014ms "crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5" "”

this is a normal communication between the crowdsec local intelligence and the its bouncer (the container which interacts with the firewall)

stephdl · October 15, 2024, 8:25am

see all current ban, even done by the central intelligence (add --all)

[root@R2-pve ~]# runagent -m crowdsec1 podman exec -ti crowdsec1 cscli decisions list --all

see all ban done, even not active

[root@R2-pve ~]# runagent -m crowdsec1 podman exec -ti crowdsec1 cscli alerts list --all

stephdl · October 15, 2024, 8:30am

journalctl --grep ‘localhost/crowdsec’
journalctl --grep ‘decision added’

stephdl · October 15, 2024, 8:56am

it seems we can quiet the log

stephdl · October 15, 2024, 12:45pm

hello mates

could please test the version of crowdsec

github.com/NethServer/dev

Crowdsec: excessive Logs from Container Communication

opened 12:27PM - 15 Oct 24 UTC

stephdl

**Issue Description** The communication between the two containers, local int…elligence and bouncer, occurs every 10 seconds, along with additional checks to the heartbeat and central intelligence. This results in excessive log entries that are not necessary since the functionality works as expected. this is some examples ``` Oct 15 11:44:42 R2-pve.rocky9-pve2.org crowdsec2[136638]: time="2024-10-15T09:44:42Z" level=info msg="127.0.0.1 - [Tue, 15 Oct 2024 09:44:42 UTC] \"GET /v1/decisions/stream HTTP/1.1 200 5.055043ms \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5\" Oct 15 11:48:04 R2-pve.rocky9-pve2.org crowdsec2[136638]: time="2024-10-15T09:48:04Z" level=info msg="127.0.0.1 - [Tue, 15 Oct 2024 09:48:04 UTC] \"GET /v1/heartbeat HTTP/1.1 200 203.138µs \"crowdsec/v1.6.3-4851945a-docker\" \"" ``` **Proposed Solution** The high volume of logs is due to the server’s log level being set to info. A potential solution would be to adjust the log level to error to reduce unnecessary log entries. **Alternative Solution** Leave the configuration unchanged, as it is functioning correctly. The issue is not a bug but rather an enhancement request. **Additional Context** CrowdSec version: 1.0.10 **References** [Community discussion on log flooding](https://community.nethserver.org/t/crowdsec-log-flooded/24746/6) ---- Thank you MadPatrick

MadPatrick · October 15, 2024, 4:54pm

Hi,
Thank you for the quick fix.
I’ve upgraded and the flooding is gone.

I\ll keep an eye on it for the next days

stephdl · October 15, 2024, 5:26pm

You need also to demonstrate that the log lines of a ban are still there

LayLow · October 15, 2024, 6:02pm

What is the correct command for installing the dev version please (Github does not tell) ? Is there a risk on a prod server?

TIA

transocean · October 15, 2024, 6:34pm

Hi @stephdl

I can confirm that.

2024-10-15T20:26:41+02:00 [1:crowdsec1:crowdsec1] time=“2024-10-15T18:26:41Z” level=info msg=“(localhost/crowdsec) LePresidente/http-generic-403-bf by ip xxx.xxx.101.119 (DE/3320) : 4m ban on Ip xxx.xxx.101.119”

Regards…

Uwe

transocean · October 15, 2024, 6:39pm

@LayLow

You can update the app directly from the app instance with a mouseclick on the top of the right side There where the 3 points are… There you get a offer to update the app vie test repo.

Regards…

Uwe

LayLow · October 15, 2024, 6:58pm

Ah! Another hidden feature to me!

Thanks @transocean

transocean · October 15, 2024, 7:05pm

@davidep has already described this here.

MadPatrick · October 15, 2024, 7:06pm

I don’t why, but i don’t have much bans.
After 1 week I had 2 bans.Still strange to my opinion becuase with fail2ban i had a few per day
Maybe Crowdsec operated differently and perhaps I am worrying about nothing

transocean · October 15, 2024, 7:12pm

@MadPatrick

May be there is a firewall in front of your server with active geo ip blocker?

Regards…

Uwe

MadPatrick · October 15, 2024, 7:20pm

I’ve Nethsecurity installed in front of NS8
But this was the same as with ClearOS and NS7 with Fail2ban.
No change on the firewall

LayLow · October 15, 2024, 7:49pm

Yep, I remembered after my post, too many changes lately Thx

LayLow · October 15, 2024, 8:01pm

I can confirm.

After install the dev version log silent, info level

2024-10-15T21:47:38+02:00 [1:crowdsec1:crowdsec1-firewall-bouncer] time="2024-10-15T19:47:38Z" level=info msg="Processing new and deleted decisions . . ."
2024-10-15T21:47:39+02:00 [1:crowdsec1:crowdsec1-firewall-bouncer] time="2024-10-15T19:47:39Z" level=info msg="198 decisions deleted"
2024-10-15T21:47:40+02:00 [1:crowdsec1:crowdsec1-firewall-bouncer] time="2024-10-15T19:47:40Z" level=info msg="50941 decisions added"

and

Banned due to multiple ssh false login attempts:

2024-10-15T21:53:47+02:00 [1:crowdsec1:crowdsec1] time="2024-10-15T19:53:47Z" level=info msg="Ip 193.187.xxx.234 performed 'crowdsecurity/ssh-bf' (7 events over 14.49895905s) at 2024-10-15 19:53:47.447410715 +0000 UTC"
2024-10-15T21:53:48+02:00 [1:crowdsec1:crowdsec1] time="2024-10-15T19:53:48Z" level=info msg="(localhost/crowdsec) crowdsecurity/ssh-bf by ip 193.187.xxx.234 (XX/13xx87) : 4m ban on Ip 193.187.xxx.234"
2024-10-15T21:53:48+02:00 [1:crowdsec1:crowdsec1] time="2024-10-15T19:53:48Z" level=info msg="Signal push: 1 signals to push"
2024-10-15T21:53:50+02:00 [1:crowdsec1:crowdsec1-firewall-bouncer] time="2024-10-15T19:53:50Z" level=info msg="1 decision added"

Thanks.

stephdl · October 15, 2024, 9:39pm

In logs I trust.

Show me the iterations of bots trying to login. Then we will be sure we have something to fix

Watch your logs !