NethServer Version: NS8
Module: Backup
After the last update i got an error when the backup running.
Any Ideas?
Any Ideas?
1 Like
Traefik was one of the recently updated core apps.
Could you please share some logs? You could put traefik as single app to an own backup schedule, run it and check the cluster logs for errors.
Let’s also check if there’s a hanging restic container:
runagent -m traefik1 podman ps -a
The same problem here. It’s possible that this has something to do with yesterday’s core update.
Single backup from Traefik does not work either.
Here are the logs:
2025-03-18T16:53:01+01:00 [1:traefik1:traefik] 192.168.154.80 - - [18/Mar/2025:15:53:01 +0000] “GET / HTTP/1.1” 302 0 “-” “-” 8769 “nextcloud1-https@file” “http://127.0.0.1:20004” 41ms
2025-03-18T16:53:01+01:00 [1:traefik1:traefik] 192.168.154.80 - - [18/Mar/2025:15:53:01 +0000] “GET /login HTTP/1.1” 200 18476 “-” “-” 8770 “nextcloud1-https@file” “http://127.0.0.1:20004” 60ms
2025-03-18T16:53:04+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:04 +0000] “POST /cluster-admin/api/cluster/tasks HTTP/2.0” 201 322 “-” “-” 8771 “cluster-admin-https@file” “http://127.0.0.1:9311” 8ms
2025-03-18T16:53:04+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:04 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/context HTTP/2.0” 200 336 “-” “-” 8772 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:04+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:04 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/context HTTP/2.0” 200 336 “-” “-” 8773 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:04+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:04 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/context HTTP/2.0” 200 336 “-” “-” 8775 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:04+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:04 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/context HTTP/2.0” 200 336 “-” “-” 8774 “cluster-admin-https@file” “http://127.0.0.1:9311” 7ms
2025-03-18T16:53:05+01:00 [1:traefik1:agent@traefik1] task/module/traefik1/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89: run-backup/50run_backup is starting
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/module/traefik1/task/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89/context HTTP/2.0” 200 239 “-” “-” 8777 “cluster-admin-https@file” “http://127.0.0.1:9311” 5ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/context HTTP/2.0” 200 336 “-” “-” 8776 “cluster-admin-https@file” “http://127.0.0.1:9311” 8ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/module/traefik1/task/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89/context HTTP/2.0” 200 239 “-” “-” 8778 “cluster-admin-https@file” “http://127.0.0.1:9311” 12ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/module/traefik1/task/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89/context HTTP/2.0” 200 239 “-” “-” 8781 “cluster-admin-https@file” “http://127.0.0.1:9311” 5ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/module/traefik1/task/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89/context HTTP/2.0” 200 239 “-” “-” 8780 “cluster-admin-https@file” “http://127.0.0.1:9311” 8ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/context HTTP/2.0” 200 336 “-” “-” 8779 “cluster-admin-https@file” “http://127.0.0.1:9311” 12ms
2025-03-18T16:53:05+01:00 [1:traefik1:agent@traefik1] cp: cannot stat ‘manual_flags’: No such file or directory
2025-03-18T16:53:05+01:00 [1:traefik1:agent@traefik1] [ERROR] module-dump-state failed. Command ‘[‘module-dump-state’, ‘2’]’ returned non-zero exit status 1.
2025-03-18T16:53:05+01:00 [1:traefik1:agent@traefik1] [ERROR] module-backup aborted.
2025-03-18T16:53:05+01:00 [1:traefik1:agent@traefik1] task/module/traefik1/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89: action “run-backup” status is “aborted” (1) at step 50run_backup
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/module/traefik1/task/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89/context HTTP/2.0” 200 239 “-” “-” 8782 “cluster-admin-https@file” “http://127.0.0.1:9311” 5ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/module/traefik1/task/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89/context HTTP/2.0” 200 239 “-” “-” 8783 “cluster-admin-https@file” “http://127.0.0.1:9311” 5ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/module/traefik1/task/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89/status HTTP/2.0” 200 505 “-” “-” 8784 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/module/traefik1/task/c3f5bb3c-d331-4fca-9b29-d9a3b291ff89/status HTTP/2.0” 200 505 “-” “-” 8785 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “POST /cluster-admin/api/cluster/tasks HTTP/2.0” 201 241 “-” “-” 8786 “cluster-admin-https@file” “http://127.0.0.1:9311” 8ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/context HTTP/2.0” 200 250 “-” “-” 8787 “cluster-admin-https@file” “http://127.0.0.1:9311” 3ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/context HTTP/2.0” 200 250 “-” “-” 8788 “cluster-admin-https@file” “http://127.0.0.1:9311” 7ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/context HTTP/2.0” 200 336 “-” “-” 8789 “cluster-admin-https@file” “http://127.0.0.1:9311” 5ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/context HTTP/2.0” 200 250 “-” “-” 8790 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/context HTTP/2.0” 200 336 “-” “-” 8792 “cluster-admin-https@file” “http://127.0.0.1:9311” 7ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/context HTTP/2.0” 200 250 “-” “-” 8791 “cluster-admin-https@file” “http://127.0.0.1:9311” 12ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/status HTTP/2.0” 200 557 “-” “-” 8793 “cluster-admin-https@file” “http://127.0.0.1:9311” 3ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/c60c1569-675b-4e61-99b1-44d059d1f9a3/status HTTP/2.0” 200 557 “-” “-” 8794 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/context HTTP/2.0” 200 250 “-” “-” 8796 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/context HTTP/2.0” 200 250 “-” “-” 8797 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/context HTTP/2.0” 200 250 “-” “-” 8798 “cluster-admin-https@file” “http://127.0.0.1:9311” 7ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/context HTTP/2.0” 200 250 “-” “-” 8795 “cluster-admin-https@file” “http://127.0.0.1:9311” 12ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/status HTTP/2.0” 200 1056 “-” “-” 8799 “cluster-admin-https@file” “http://127.0.0.1:9311” 5ms
2025-03-18T16:53:05+01:00 [1:traefik1:traefik] 192.168.154.151 - - [18/Mar/2025:15:53:05 +0000] “GET /cluster-admin/api/cluster/task/9daf79b9-b417-4505-8c9b-b22f9cac5e32/status HTTP/2.0” 200 1056 “-” “-” 8800 “cluster-admin-https@file” “http://127.0.0.1:9311” 4ms
2025-03-18T16:53:08+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:08 +0000] “PROPFIND /remote.php/dav/files/d13ef92d-54ce-42db-82e9-fe73ea18a131/ HTTP/1.1” 207 270 “-” “-” 8801 “nextcloud1-https@file” “http://127.0.0.1:20004” 57ms
2025-03-18T16:53:12+01:00 [1:traefik1:traefik] 192.168.154.150 - - [18/Mar/2025:15:53:12 +0000] “PROPFIND /remote.php/dav/files/d13ef92d-54ce-42db-82e9-fe73ea18a131/ HTTP/1.1” 207 261 “-” “-” 8802 “nextcloud1-https@file” “http://127.0.0.1:20004” 48ms
runagent -m traefik1 podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4cfbb0a71d65 Docker Hub Container Image Library | App Containerization traefik 10 hours ago Up 10 hours 80/tcp traefik
And _api_server.yml is no longer present after the core update.
However, the cluster-admin page is once again accessible via the Internet, although this is not supposed to be the case.
1 Like
It seems the directory “manual_flags” is missing, I assume because it’s not created on update and you don’t have custom HTTP routes.
Please check the files and directories in traefik:
[root@ns8rockytest ~]# runagent -m traefik1 ls -l
total 40
-rw-r--r--. 1 traefik1 traefik1 103 Jul 25 2024 CACHEDIR.TAG
drwxr-xr-x. 2 traefik1 traefik1 56 Mar 17 10:06 acme
-rw-------. 1 traefik1 traefik1 278 Aug 5 2024 agent.env
-rw-------. 1 traefik1 traefik1 212 Mar 12 19:28 apitoken.cache
drwxr-xr-x. 2 traefik1 traefik1 4096 Mar 17 10:07 configs
drwxr-xr-x. 2 traefik1 traefik1 4096 Mar 17 10:06 configs.v2
drwxr-xr-x. 2 traefik1 traefik1 106 Dec 17 22:39 custom_certificates
drwxr-xr-x. 2 traefik1 traefik1 106 Mar 17 10:06 custom_certificates.v2
-rw-r--r--. 1 traefik1 traefik1 0 Jan 25 23:56 dump
-rw-r--r--. 1 traefik1 traefik1 954 Mar 17 10:06 environment
drwxr-xr-x. 2 traefik1 traefik1 31 Jan 30 10:36 manual_flags
-rw-r--r--. 1 traefik1 traefik1 1960 Aug 5 2024 selfsigned.crt
-rw-------. 1 traefik1 traefik1 3268 Aug 5 2024 selfsigned.key
drwxr-xr-x. 6 traefik1 traefik1 100 Mar 18 16:28 state-backup
-rw-r--r--. 1 traefik1 traefik1 568 Mar 17 10:06 traefik.yaml
-rw-r--r--. 1 traefik1 traefik1 534 Mar 17 10:06 traefik.yaml.v2
If the directory “manual_flags” or other dirs don’t exist, just create them:
runagent -m traefik1 mkdir manual_flags
Now the backup hopefully should work again.
EDIT:
This issue should be solved soon, see also User portal, unprotected access from the outside - #14 by davidep or Expose IP allow list configuration in cluster web interface · Issue #7348 · NethServer/dev · GitHub
4 Likes
You are the best… 
2 Likes
Thanks guys for pointing out the Bug, I changed the topic category accordingly.
I’d propose to add the creation of the directory to update-module.d but let’s wait if there are better approaches.
cc @davidep
EDIT:
Maybe just adding ExecStartPre=mkdir -p manual_flags to the service file is the better option.
2 Likes
thank you, works again
2 Likes
Hi Markus, after looking at the code I confirm the Traefik 3.0.0 regression.
Your workaround that manually creates the missing “manual_flags” directory is correct!
A even simpler alternative is to create a temporary HTTP route from UI, then delete it. Please suggest it as alternative in your solution 
The missing directory is now also created during the module installation, but pre-3.0.0 versions do it only when manual HTTP routes are configured.
Both proposals should work. A third option is creating the directory in module-dump-state, or ignoring the error, so we do not need a service restart to fix the backup failure.
See also:
2 Likes