NS8 >> stats.grafana.org

capote · January 20, 2025, 7:29pm

NS8 produces massive DNS-requests to this subdomain.

Is it possible to disable this behavior on the NS8-Server?

Ps.:

core1	Knoten 1	3.4.1
ldapproxy1	Knoten 1	1.1.0
loki1	Knoten 1	1.2.2
samba1	Knoten 1	2.3.0
traefik1	Knoten 1	2.2.5

mrmarkuz · January 20, 2025, 7:35pm

Do you have Grafana installed on the NS8?

It seems some of it’s plugins want to send usage stats. It could also be loki…

EDIT:

Here are methods for disabling it for loki:

github.com/grafana/loki

Loki Reports to stats.grafana.org despite `analytics.reporting_enabled: false`

opened 04:57PM - 19 Sep 23 UTC

tomswartz07

type/bug

**Describe the bug** Loki attempts to report usage analytics to `stats.grafana.…org` despite the `reporting_enabled` value being set to False. Additionally, if the `analytics.usage_stats_url` is set to a null string: `""`, Loki will not honor this value and report to `stats.grafana.org`, as well. **To Reproduce** Steps to reproduce the behavior: 1. Started Loki (SHA: `a63be545e811` Docker image `latest` tag) 2. Ensure that `stats.grafana.org` is blocked by PiHole/DNS service. 3. Wait several hours. 4. Observe that `reporter.go` emits hundreds of errors indicating that the connection is refused (rightfully so, in my case) ``` level=info ts=2023-09-19T16:01:39.855333051Z caller=reporter.go:303 msg="failed to send usage report" retries=0 err="Post \"https://stats.grafana.org/loki-usage-report\": dial tcp 0.0.0.0:443: connect: connection refused" level=info ts=2023-09-19T16:01:40.859555584Z caller=reporter.go:303 msg="failed to send usage report" retries=1 err="Post \"https://stats.grafana.org/loki-usage-report\": dial tcp 0.0.0.0:443: connect: connection refused" level=info ts=2023-09-19T16:01:43.863365631Z caller=reporter.go:303 msg="failed to send usage report" retries=2 err="Post \"https://stats.grafana.org/loki-usage-report\": dial tcp 0.0.0.0:443: connect: connection refused" level=info ts=2023-09-19T16:01:50.906529945Z caller=reporter.go:303 msg="failed to send usage report" retries=3 err="Post \"https://stats.grafana.org/loki-usage-report\": dial tcp 0.0.0.0:443: connect: connection refused" ``` The indicated line in the error: `reporter.go:303` [appears to refer to a backoff](https://github.com/grafana/loki/blob/main/pkg/analytics/reporter.go#L303) which shouldn't be triggered in the first place if reporting is disabled. In the span of a 30 minutes, this generated 228 instances of this error, and 380 failed DNS lookups. ![image](https://github.com/grafana/loki/assets/582862/1b049124-defc-4cda-be45-1b97998ac12f) **Expected behavior** Loki should not attempt to report stats if analytics.reporting_enabled is set to false. **Environment:** - Infrastructure: Docker - Deployment tool: Docker **Screenshots, Promtail config, or terminal output** Amended loki.config file: This config file is mounted at `/etc/loki/local-config.yml` as a read-only file within the docker container. ``` auth_enabled: false analytics: reporting_enabled: false usage_stats_url: "" server: http_listen_address: IP_ADDRESS http_listen_port: 9080 grpc_listen_port: 9096 common: path_prefix: /tmp/loki storage: filesystem: chunks_directory: /tmp/loki/chunks rules_directory: /tmp/loki/rules replication_factor: 1 ring: instance_addr: 127.0.0.1 kvstore: store: inmemory schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h ruler: alertmanager_url: http://IP_ADDRESS:9093 ``` Deployed in Docker, with the following ENV Vars: ``` "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TZ=America/New_York" ], ``` I can confirm that I have both restarted and created a new container after applying the expected settings, but Loki does not appear to honor these parameters. To be completely transparent: I've stopped using Loki unless and until this issue can be resolved. This issue represents a breach of trust, and considering the purpose of this tool is to handle sensitive log files, I trust you understand the implications of this issue.

capote · January 20, 2025, 9:33pm

Thank you Markus
No, I don’t have installed Grafana.

As workaround I added 0.0.0.0 stats.grafana.org to /etc/hosts.

@davidep @stephdl
The dev’s should disable it by default. I am surprised that such a long-known mistake has not been corrected in the meantime.

Sincerely, Marko

mrmarkuz · January 20, 2025, 9:41pm

It can be disabled in loki:

runagent -m loki1

Edit loki config file:

nano ../loki-config.yaml

and add following content after auth-enabled: false:

analytics:
  reporting_enabled: false
  usage_stats_url: ""

Restart loki-server:

systemctl --user restart loki-server

capote · January 20, 2025, 9:58pm

I had already tried this also in the meantime.

root@daho-ns8:~# runagent -m loki1 podman exec -ti loki-server vi /etc/loki/local-config.yaml

But the loki-config.yaml is write-protected.

runagent -m loki1 podman exec -ti loki-server ls -sla /etc/loki/local-config.yaml
     4 -rw-r--r--    1 root     root          1835 Nov  5 07:46 /etc/loki/local-config.yaml

mrmarkuz · January 20, 2025, 10:05pm

You’re running it from inside the container.
The loki container by default doesn’t login as root user when using podman exec. You could use podman exec -ti --user 0:0 loki-server sh to enter as root, then the file should be writable.
But that’s not necessary, the config file is in /home/loki1/.config/loki-config.yaml, it should work as in the instructions I posted.

In one line:

runagent -m loki1 nano ../loki-config.yaml

capote · January 20, 2025, 10:14pm

Ahhhh. Thank you!
So I was almost on the right track. But only almost.

I summarize:

runagent -m loki1 nano /home/loki1/.config/loki-config.yaml

auth_enabled: false

analytics:
  reporting_enabled: false
  usage_stats_url: ""

Restart loki-server:

runagent -m loki1 systemctl --user restart loki-server

mrmarkuz · January 20, 2025, 10:25pm

You’re welcome, please report if it worked.

It was the right way but this is a special case.

capote · January 20, 2025, 10:27pm

I can confirm, it works.

hucky · February 4, 2025, 2:31pm

same problem here but the workaround leads to
File “/usr/local/bin/runagent”, line 92, in
os.execvp(args.COMMAND, [args.COMMAND] + args.ARGS)
File “”, line 574, in execvp
File “”, line 616, in _execvpe
File “”, line 607, in _execvpe
FileNotFoundError: [Errno 2] No such file or directory
any other ideas?

capote · February 20, 2025, 4:21pm

and now, the problem is back

I added …

analytics:
  reporting_enabled: false
  usage_stats_url: ""

…again and only left out auth_enabled: false, because it is already in the first line

hucky · February 20, 2025, 5:21pm

same error here, insane dns questions like i post in this link:

Andy_Wismer · February 20, 2025, 5:38pm

@hucky

Why are you using IPv6 for this query?
Or you’re not aware your server is partially using IPv6?

My 2 cents
Andy

mrmarkuz · February 20, 2025, 6:03pm

I corrected my post as it was misleading.

@hucky please try this to get rid of the queries: NS8 >> stats.grafana.org - #4 by mrmarkuz

jgelauff · March 15, 2025, 8:12pm

i had to reverse this to get logs working again in the cluster_admin after the last core (3.6.0) update

hucky · May 4, 2025, 10:47am

at my side, with the workaround from post #4 it did not work, grafana spams again my protokoll

2025-05-04 12:47:37	A	stats.grafana.org	sogo.domain.de	4.5 µs
2025-05-04 12:47:37	AAAA	stats.grafana.org	sogo.domain.de	11.4 µs
2025-05-04 12:47:29	A	stats.grafana.org	sogo.domain.de	4.1 µs
2025-05-04 12:47:29	AAAA	stats.grafana.org	sogo.domain.de	11.4 µs
2025-05-04 12:47:27	A	stats.grafana.org	sogo.domain.de	5.0 µs
2025-05-04 12:47:27	AAAA	stats.grafana.org	sogo.domain.de	12.4 µs
2025-05-04 12:47:25	A	stats.grafana.org	sogo.domain.de	4.5 µs
2025-05-04 12:47:25	AAAA	stats.grafana.org	sogo.domain.de	11.4 µs

capote · May 4, 2025, 11:02am

same at me. I assume that the adjustments will be overwritten during updates.

hucky · May 4, 2025, 11:41am

i did the same again how @mrmarkuz described it at #4 but it does not work

capote · May 4, 2025, 11:53am

It works for me. The DNS-requests stops.

loki1@daho-ns8:~/.config/state$ cat ../loki-config.yaml
auth_enabled: false
analytics:
  reporting_enabled: false
  usage_stats_url: ""

server:
  http_listen_port: 3100
...
...
...

hucky · May 4, 2025, 12:10pm

@capote thank you, will have a look at it, maybe i had an error.