Sftpgo ns8 not start and not configured?

france · April 10, 2024, 9:30am

Hi everyone, I can’t restart sftpgo.

2024-04-10T11:27:42+02:00 [1:webserver1:nginx] 2024/04/10 09:27:42 [notice] 1#1: OS: Linux 5.14.0-362.24.1.el9_3.x86_64
2024-04-10T11:27:42+02:00 [1:webserver1:nginx] 2024/04/10 09:27:42 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 524288:524288
2024-04-10T11:27:42+02:00 [1:webserver1:nginx] 2024/04/10 09:27:42 [notice] 1#1: start worker processes
2024-04-10T11:27:42+02:00 [1:webserver1:nginx] 2024/04/10 09:27:42 [notice] 1#1: start worker process 15
2024-04-10T11:27:42+02:00 [1:webserver1:nginx] 2024/04/10 09:27:42 [notice] 1#1: start worker process 16
2024-04-10T11:27:42+02:00 [1:webserver1:nginx] 2024/04/10 09:27:42 [notice] 1#1: start worker process 17
2024-04-10T11:27:42+02:00 [1:webserver1:nginx] 2024/04/10 09:27:42 [notice] 1#1: start worker process 18
2024-04-10T11:27:42+02:00 [1:webserver1:podman] 2149f785bdb2d51d223cc846feb103034ae37fcb56d76e660e37753ee0665aa1
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] Started Podman  nginx.service.
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] Starting Podman  sftpgo.service...
2024-04-10T11:27:42+02:00 [1:webserver1:podman] time="2024-04-10T11:27:42+02:00" level=warning msg="Unmounting container \"sftpgo\" while attempting to delete storage: unmounting \"/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged\": invalid argument"
2024-04-10T11:27:42+02:00 [1:webserver1:podman] Error: removing storage for container "sftpgo": unmounting "/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged": invalid argument
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] sftpgo.service: Control process exited, code=exited, status=125/n/a
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] sftpgo.service: Failed with result 'exit-code'.
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] Failed to start Podman  sftpgo.service.
2024-04-10T11:27:42+02:00 [1:webserver1:agent@webserver1] Job for sftpgo.service failed because the control process exited with error code.
2024-04-10T11:27:42+02:00 [1:webserver1:agent@webserver1] See "systemctl --user status sftpgo.service" and "journalctl --user -xeu sftpgo.service" for details.
2024-04-10T11:27:42+02:00 [1:webserver1:agent@webserver1]   File "/home/webserver1/.config/actions/configure-module/50start_services", line 14, in <module>
2024-04-10T11:27:42+02:00 [1:webserver1:agent@webserver1] Traceback (most recent call last):
2024-04-10T11:27:42+02:00 [1:webserver1:agent@webserver1]     agent.run_helper("systemctl", "--user", "restart", "sftpgo.service").check_returncode()
2024-04-10T11:27:42+02:00 [1:webserver1:agent@webserver1]   File "/usr/lib64/python3.11/subprocess.py", line 502, in check_returncode
2024-04-10T11:27:42+02:00 [1:webserver1:agent@webserver1] subprocess.CalledProcessError: Command '('systemctl', '--user', 'restart', 'sftpgo.service')' returned non-zero exit status 1.
2024-04-10T11:27:42+02:00 [1:webserver1:agent@webserver1]     raise CalledProcessError(self.returncode, self.args, self.stdout,
2024-04-10T11:27:42+02:00 [1:webserver1:agent@webserver1] task/module/webserver1/b21e99fd-c929-4a14-b327-41c5beceb68d: action "configure-module" status is "aborted" (1) at step 50start_services
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] sftpgo.service: Scheduled restart job, restart counter is at 1.
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] Stopped Podman  sftpgo.service.
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] Starting Podman  sftpgo.service...
2024-04-10T11:27:42+02:00 [1:webserver1:podman] time="2024-04-10T11:27:42+02:00" level=warning msg="Unmounting container \"sftpgo\" while attempting to delete storage: unmounting \"/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged\": invalid argument"
2024-04-10T11:27:42+02:00 [1:webserver1:podman] Error: removing storage for container "sftpgo": unmounting "/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged": invalid argument
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] sftpgo.service: Control process exited, code=exited, status=125/n/a
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] sftpgo.service: Failed with result 'exit-code'.
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] Failed to start Podman  sftpgo.service.
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] sftpgo.service: Scheduled restart job, restart counter is at 2.
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] Stopped Podman  sftpgo.service.
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] Starting Podman  sftpgo.service...
2024-04-10T11:27:42+02:00 [1:webserver1:podman] time="2024-04-10T11:27:42+02:00" level=warning msg="Unmounting container \"sftpgo\" while attempting to delete storage: unmounting \"/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged\": invalid argument"
2024-04-10T11:27:42+02:00 [1:webserver1:podman] Error: removing storage for container "sftpgo": unmounting "/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged": invalid argument
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] sftpgo.service: Control process exited, code=exited, status=125/n/a
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] sftpgo.service: Failed with result 'exit-code'.
2024-04-10T11:27:42+02:00 [1:webserver1:systemd] Failed to start Podman  sftpgo.service.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] sftpgo.service: Scheduled restart job, restart counter is at 3.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] Stopped Podman  sftpgo.service.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] Starting Podman  sftpgo.service...
2024-04-10T11:27:43+02:00 [1:webserver1:podman] time="2024-04-10T11:27:43+02:00" level=warning msg="Unmounting container \"sftpgo\" while attempting to delete storage: unmounting \"/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged\": invalid argument"
2024-04-10T11:27:43+02:00 [1:webserver1:podman] Error: removing storage for container "sftpgo": unmounting "/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged": invalid argument
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] sftpgo.service: Control process exited, code=exited, status=125/n/a
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] sftpgo.service: Failed with result 'exit-code'.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] Failed to start Podman  sftpgo.service.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] sftpgo.service: Scheduled restart job, restart counter is at 4.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] Stopped Podman  sftpgo.service.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] Starting Podman  sftpgo.service...
2024-04-10T11:27:43+02:00 [1:webserver1:podman] time="2024-04-10T11:27:43+02:00" level=warning msg="Unmounting container \"sftpgo\" while attempting to delete storage: unmounting \"/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged\": invalid argument"
2024-04-10T11:27:43+02:00 [1:webserver1:podman] Error: removing storage for container "sftpgo": unmounting "/home/webserver1/.local/share/containers/storage/overlay/ca0466378f9da2f131537423a93f066eeea133127df3625aaf2d163f4837d9c7/merged": invalid argument
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] sftpgo.service: Control process exited, code=exited, status=125/n/a
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] sftpgo.service: Failed with result 'exit-code'.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] Failed to start Podman  sftpgo.service.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] sftpgo.service: Scheduled restart job, restart counter is at 5.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] Stopped Podman  sftpgo.service.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] sftpgo.service: Start request repeated too quickly.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] sftpgo.service: Failed with result 'exit-code'.
2024-04-10T11:27:43+02:00 [1:webserver1:systemd] Failed to start Podman  sftpgo.service.
2024-04-10T11:28:07+02:00 [1:webserver1:agent@webserver1] task/module/webserver1/42d24765-e3c4-4585-b261-271af7ba04f2: get-configuration/20read is starting
2024-04-10T11:28:07+02:00 [1:webserver1:agent@webserver1] task/module/webserver1/6f1e1786-816d-4448-9806-13b167fd52bd: get-status/20read is starting
2024-04-10T11:28:07+02:00 [1:webserver1:agent@webserver1] task/module/webserver1/42d24765-e3c4-4585-b261-271af7ba04f2: action "get-configuration" status is "completed" (0) at step validate-output.json
2024-04-10T11:28:07+02:00 [1:webserver1:agent@webserver1] task/module/webserver1/6f1e1786-816d-4448-9806-13b167fd52bd: action "get-status" status is "completed" (0) at step validate-output.json

<7>dump_env() is deprecated and implemented as a no-op
<7>systemctl --user restart webserver.service
<7>systemctl --user enable sftpgo.service
<7>systemctl --user restart sftpgo.service
Job for sftpgo.service failed because the control process exited with error code.
See "systemctl --user status sftpgo.service" and "journalctl --user -xeu sftpgo.service" for details.
Traceback (most recent call last):
  File "/home/webserver1/.config/actions/configure-module/50start_services", line 14, in <module>
    agent.run_helper("systemctl", "--user", "restart", "sftpgo.service").check_returncode()
  File "/usr/lib64/python3.11/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '('systemctl', '--user', 'restart', 'sftpgo.service')' returned non-zero exit status 1.

mrmarkuz · April 10, 2024, 10:33am

I think it’s following issue:

github.com/containers/podman

podman failed to remove a container created with buildah

opened 08:21AM - 03 Aug 23 UTC

closed 03:57PM - 14 Aug 23 UTC

sstosh

kind/bug locked - please file new issue/PR

### Issue Description podman failed to remove a container created with buildah …and prints an error message. ``` Error: removing storage for container "<container name>": unmounting "<graphroot>/overlay/<layer id>/merged": invalid argument ``` This issue is reproduced after applying a commit c83589a8529de213b17dc69b9c7ad3e5ec86494a (#18835). The root cause seems to be a `containers/common` or `containers/storage` repository. ### Steps to reproduce the issue <details> <summary>latest main branch: Failed</summary> ```console # podman version Client: Podman Engine Version: 4.7.0-dev API Version: 4.7.0-dev Go Version: go1.20.6 Git Commit: ebf368dd692c498b1e921be0dc546fb3ad50548a Built: Thu Aug 3 17:07:18 2023 OS/Arch: linux/amd64 # buildah from quay.io/libpod/testimage:20221018 Trying to pull quay.io/libpod/testimage:20221018... Getting image source signatures Copying blob 578f06cc66c5 done Copying blob a3ed95caeb02 done Copying config f5a99120db done Writing manifest to image destination testimage-working-container # podman rm -f testimage-working-container WARN[0000] Unmounting container "testimage-working-container" while attempting to delete storage: unmounting "/var/lib/containers/storage/overlay/2b1d7e0f744eccac26db50d73725324c10e221014fc1a018b1061419c61be8f9/merged": invalid argument Error: removing storage for container "testimage-working-container": unmounting "/var/lib/containers/storage/overlay/2b1d7e0f744eccac26db50d73725324c10e221014fc1a018b1061419c61be8f9/merged": invalid argument ``` </details> <details> <summary>commit c83589a8529de: Failed</summary> ```console # podman version Client: Podman Engine Version: 4.6.0-dev API Version: 4.6.0-dev Go Version: go1.20.6 Git Commit: c83589a8529de213b17dc69b9c7ad3e5ec86494a Built: Thu Aug 3 16:13:34 2023 OS/Arch: linux/amd64 # buildah from quay.io/libpod/testimage:20221018 Trying to pull quay.io/libpod/testimage:20221018... Getting image source signatures Copying blob a3ed95caeb02 done Copying blob 578f06cc66c5 done Copying config f5a99120db done Writing manifest to image destination testimage-working-container # podman rm -f testimage-working-container WARN[0000] Unmounting container "testimage-working-container" while attempting to delete storage: unmounting "/var/lib/containers/storage/overlay/7f8faf07d665232b80b8f1e10cbaa29b90f5dd7b7bfb4f84a6a77c256776cf3b/merged": invalid argument Error: removing storage for container "testimage-working-container": unmounting "/var/lib/containers/storage/overlay/7f8faf07d665232b80b8f1e10cbaa29b90f5dd7b7bfb4f84a6a77c256776cf3b/merged": invalid argument ``` </details> <details> <summary>commit decbab9223a2: Succeeded</summary> ```console # podman version Client: Podman Engine Version: 4.6.0-dev API Version: 4.6.0-dev Go Version: go1.20.6 Git Commit: decbab9223a22e9dba6918ce0a908215ae8b7cbe Built: Thu Aug 3 16:19:52 2023 OS/Arch: linux/amd64 # buildah from quay.io/libpod/testimage:20221018 Trying to pull quay.io/libpod/testimage:20221018... Getting image source signatures Copying blob a3ed95caeb02 done Copying blob 578f06cc66c5 done Copying config f5a99120db done Writing manifest to image destination testimage-working-container # podman rm -f testimage-working-container testimage-working-container ``` </details> ### Describe the results you received ```console # podman rm -f testimage-working-container WARN[0000] Unmounting container "testimage-working-container" while attempting to delete storage: unmounting "/var/lib/containers/storage/overlay/7f8faf07d665232b80b8f1e10cbaa29b90f5dd7b7bfb4f84a6a77c256776cf3b/merged": invalid argument Error: removing storage for container "testimage-working-container": unmounting "/var/lib/containers/storage/overlay/7f8faf07d665232b80b8f1e10cbaa29b90f5dd7b7bfb4f84a6a77c256776cf3b/merged": invalid argument ``` ### Describe the results you expected ```console # podman rm -f testimage-working-container testimage-working-container ``` ### podman info output ```yaml host: arch: amd64 buildahVersion: 1.32.0-dev cgroupControllers: - cpuset - cpu - io - memory - hugetlb - pids - misc cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.1.7-2.fc38.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.7, commit: ' cpuUtilization: idlePercent: 99.9 systemPercent: 0.06 userPercent: 0.04 cpus: 36 databaseBackend: boltdb distribution: distribution: fedora variant: server version: "38" eventLogger: journald freeLocks: 2048 hostname: fedora38 idMappings: gidmap: null uidmap: null kernel: 6.2.9-300.fc38.x86_64 linkmode: dynamic logDriver: journald memFree: 7655833600 memTotal: 8301731840 networkBackend: netavark networkBackendInfo: backend: netavark dns: package: aardvark-dns-1.7.0-1.fc38.x86_64 path: /usr/libexec/podman/aardvark-dns version: aardvark-dns 1.7.0 package: netavark-1.7.0-1.fc38.x86_64 path: /usr/libexec/podman/netavark version: netavark 1.7.0 ociRuntime: name: crun package: crun-1.8.5-1.fc38.x86_64 path: /usr/bin/crun version: |- crun version 1.8.5 commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed rundir: /run/user/0/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL os: linux pasta: executable: /usr/bin/pasta package: passt-0^20230625.g32660ce-1.fc38.x86_64 version: | pasta 0^20230625.g32660ce-1.fc38.x86_64 Copyright Red Hat GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. remoteSocket: path: /run/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.0-12.fc38.x86_64 version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.3 swapFree: 8301572096 swapTotal: 8301572096 uptime: 0h 8m 19.00s plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan - ipvlan volume: - local registries: search: - registry.fedoraproject.org - registry.access.redhat.com - docker.io - quay.io store: configFile: /usr/share/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.mountopt: nodev,metacopy=on graphRoot: /var/lib/containers/storage graphRootAllocated: 16039018496 graphRootUsed: 10907877376 graphStatus: Backing Filesystem: xfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "true" imageCopyTmpDir: /var/tmp imageStore: number: 1 runRoot: /run/containers/storage transientStore: false volumePath: /var/lib/containers/storage/volumes version: APIVersion: 4.7.0-dev Built: 1691050038 BuiltTime: Thu Aug 3 17:07:18 2023 GitCommit: ebf368dd692c498b1e921be0dc546fb3ad50548a GoVersion: go1.20.6 Os: linux OsArch: linux/amd64 Version: 4.7.0-dev ``` ### Podman in a container No ### Privileged Or Rootless None ### Upstream Latest Release Yes ### Additional environment details KVM ### Additional information It may be related to the following issue. https://github.com/containers/podman/issues/18874#issuecomment-1591940810 ```console # bin/podman rm -f 335f WARN[0000] Unmounting container "335f" while attempting to delete storage: unmounting "/var/lib/containers/storage/overlay/e3149559a6e1ae65a7b0b2e140ceb2bb7e7bdc06b9654bd5d46ba5d3adf7ad46/merged": invalid argument Error: removing storage for container "335f": unmounting "/var/lib/containers/storage/overlay/e3149559a6e1ae65a7b0b2e140ceb2bb7e7bdc06b9654bd5d46ba5d3adf7ad46/merged": invalid argument ```

Be sure to have a backup before removing overlay folders:

runagent -m webserver1 rmdir ../../.local/share/containers/storage/overlay/*/merged

Restart service:

runagent -m webserver1 systemctl --user restart webserver

france · April 10, 2024, 10:48am

runagent -m systemctl --user restart webserver
usage: runagent [-h] [-m MODULE_ID] [-c] [COMMAND] …
runagent: error: unrecognized arguments: --user

mrmarkuz · April 10, 2024, 10:50am

Please try again, I corrected my post. I forgot “webserver1”

france · April 10, 2024, 11:08am

Sorry it works!

france · April 10, 2024, 11:10am

But keep giving this error

dnutan · April 10, 2024, 11:16am

Probably some service still loading or failed.
Check the status of services on the webserver app instance.

france · April 10, 2024, 11:45am

I noticed that in the first instance the delete command gave error

runagent -m webserver1 rmdir ../../.local/share/containers/storage/overlay/*/merged
rmdir: failed to remove '../../.local/share/containers/storage/overlay/*/merged': No such file or directory

@Mrmarkuz anyway the delete command returns me the error and the merged folders are present . Where do I need to run the command from?

mrmarkuz · April 10, 2024, 12:24pm

Sorry, you need to be root, so this one should work:

rmdir /home/webserver1/.local/share/containers/storage/overlay/*/merged