Error: allocating lock for new container: allocation failed; exceeded num_locks (2048)

kemboielvis22 · January 14, 2025, 8:52pm

am getting this error when installing a module whats the issue

2025-01-14T23:46:11+03:00 [1:kobotoolbox7:systemd] Starting Podman kobotoolbox.service...
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Created slice cgroup user-libpod_pod_786a8f3786c4d1642e5bb1593a807a04ee8aa91da470cecb924d15efe3a0d51a.slice.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Removed slice cgroup user-libpod_pod_786a8f3786c4d1642e5bb1593a807a04ee8aa91da470cecb924d15efe3a0d51a.slice.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:podman] Error: allocating lock for new container: allocation failed; exceeded num_locks (2048)
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Control process exited, code=exited, status=125/n/a
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:podman] Error: reading pod ID file: open /run/user/1009/kobotoolbox.pod-id: no such file or directory
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Control process exited, code=exited, status=125/n/a
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Failed with result 'exit-code'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Failed to start Podman kobotoolbox.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  postgres-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] postgres-app.service: Job postgres-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  redis-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] redis-app.service: Job redis-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  kpi-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kpi-app.service: Job kpi-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  nginx-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] nginx-app.service: Job nginx-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  enketo_express-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] enketo_express-app.service: Job enketo_express-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  kpi_worker-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kpi_worker-app.service: Job kpi_worker-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  mongo-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] mongo-app.service: Job mongo-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:agent@kobotoolbox7] See "systemctl --user status kobotoolbox.service" and "journalctl --user -xeu kobotoolbox.service" for details.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:agent@kobotoolbox7] Job for kobotoolbox.service failed because the control process exited with error code.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:agent@kobotoolbox7] task/module/kobotoolbox7/f5a7a8ac-5283-4c8a-9083-b502454a708a: action "configure-module" status is "aborted" (1) at step 80start_services
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Scheduled restart job, restart counter is at 1.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Stopped Podman kobotoolbox.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Starting Podman kobotoolbox.service...
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Created slice cgroup user-libpod_pod_c74e3efda835b143eb9c93d31a2de1645d502f1672a2907bc2edf94e357561d5.slice.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Removed slice cgroup user-libpod_pod_c74e3efda835b143eb9c93d31a2de1645d502f1672a2907bc2edf94e357561d5.slice.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:podman] Error: allocating lock for new container: allocation failed; exceeded num_locks (2048)
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Control process exited, code=exited, status=125/n/a
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:podman] Error: reading pod ID file: open /run/user/1009/kobotoolbox.pod-id: no such file or directory
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Control process exited, code=exited, status=125/n/a
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Failed with result 'exit-code'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Failed to start Podman kobotoolbox.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  postgres-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] postgres-app.service: Job postgres-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  redis-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] redis-app.service: Job redis-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  kpi-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kpi-app.service: Job kpi-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  nginx-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] nginx-app.service: Job nginx-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  enketo_express-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] enketo_express-app.service: Job enketo_express-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  kpi_worker-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] kpi_worker-app.service: Job kpi_worker-app.service/start failed with result 'dependency'.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  mongo-app.service.
2025-01-14T23:46:12+03:00 [1:kobotoolbox7:systemd] mongo-app.service: Job mongo-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Scheduled restart job, restart counter is at 2.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Stopped Podman kobotoolbox.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Starting Podman kobotoolbox.service...
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Created slice cgroup user-libpod_pod_3f308d3aabd39e425edf0fdd9ddd01552071ac36fa818c18af04e886b8f1476b.slice.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Removed slice cgroup user-libpod_pod_3f308d3aabd39e425edf0fdd9ddd01552071ac36fa818c18af04e886b8f1476b.slice.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:podman] Error: allocating lock for new container: allocation failed; exceeded num_locks (2048)
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Control process exited, code=exited, status=125/n/a
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:podman] Error: reading pod ID file: open /run/user/1009/kobotoolbox.pod-id: no such file or directory
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Control process exited, code=exited, status=125/n/a
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Failed with result 'exit-code'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Failed to start Podman kobotoolbox.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  postgres-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] postgres-app.service: Job postgres-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  redis-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] redis-app.service: Job redis-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  kpi-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kpi-app.service: Job kpi-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  nginx-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] nginx-app.service: Job nginx-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  enketo_express-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] enketo_express-app.service: Job enketo_express-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  kpi_worker-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kpi_worker-app.service: Job kpi_worker-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  mongo-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] mongo-app.service: Job mongo-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Start request repeated too quickly.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kobotoolbox.service: Failed with result 'exit-code'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Failed to start Podman kobotoolbox.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  postgres-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] postgres-app.service: Job postgres-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  redis-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] redis-app.service: Job redis-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  kpi-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kpi-app.service: Job kpi-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  nginx-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] nginx-app.service: Job nginx-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  enketo_express-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] enketo_express-app.service: Job enketo_express-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  kpi_worker-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] kpi_worker-app.service: Job kpi_worker-app.service/start failed with result 'dependency'.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] Dependency failed for Podman  mongo-app.service.
2025-01-14T23:46:13+03:00 [1:kobotoolbox7:systemd] mongo-app.service: Job mongo-app.service/start failed with result 'dependency'.

dnutan · January 14, 2025, 9:23pm

num_locks=2048

Number of locks available for containers, pods, and volumes. Each created container, pod, or volume consumes one lock. Locks are recycled and can be reused after the associated container, pod, or volume is removed. The default number available is 2048. If this is changed, a lock renumbering must be performed, using the podman system renumber command.

You can remove unused volumes:

runagent -m kobotoolbox7 podman volume ls |wc -l
runagent -m kobotoolbox7 podman volume rm --all # (possible to use [podman volume rm](https://docs.podman.io/en/v4.4/markdown/podman-volume-rm.1.html), that by default removes unused volumes

maybe some steps can be ran directly outside the container user env… (without runagent…). I have very little experience with it:

podman volume ls |wc -l
podman volume prune

Or increase num_locks in podman’s containers.conf file (on nethserver it appears to be at /usr/share/containers.containers.conf), might come with some performance penalty.

kemboielvis22 · January 14, 2025, 9:25pm

i have tried to increase to 4096 but it says out of range

mrmarkuz · January 14, 2025, 9:27pm

Here’s how to remove unused volumes: (Solution 1)

You may need to also run the podman volume rm -a or podman volume prune commands in the container user environments. (runagent -m app1)

kemboielvis22 · January 14, 2025, 9:34pm

This says

WARNING! This will remove all volumes not used by at least one container. The following volumes will be removed:
No dangling volumes found

mrmarkuz · January 14, 2025, 9:49pm

Did you try removing volumes in rootless container user environments?

cd /home; for d in * ; do runagent -m "$d" podman volume rm -a; done

kemboielvis22 · January 14, 2025, 10:10pm

am still having issues

kemboielvis22 · January 14, 2025, 10:13pm

Is there a way to increase num_locks without the out of range error

dnutan · January 14, 2025, 10:21pm

did you run podman system renumber after the change?

kemboielvis22 · January 14, 2025, 10:24pm

yeah i get the error below

Error: failed to open 4096 locks in /libpod_lock: numerical result out of range

dnutan · January 14, 2025, 10:25pm

Not the same error word per word but probably gives a good hint:

github.com/containers/podman

Error: failed to open 16384 locks in /libpod_rootless_lock_1000: numerical result out of range

opened 04:46PM - 11 Oct 22 UTC

baxeno

kind/bug

**Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)** … /kind bug **Description** I have been doing some performance testing where I needed _many_ containers, and I think this is a bug and/or misleading error message in Podman. Default limit in `engine.num_locks` is `2048` and each container use a lock. When starting container 2049 with default configuration the following error is displayed: ``` Error: error allocating lock for new container: allocation failed; exceeded num_locks (2048) ``` The issue happens when trying to increase the lock limit. **Steps to reproduce the issue:** 1. Start one or more container(s) ``` #!/bin/bash COUNT=2500 for i in $(seq "$COUNT"); do podman run --name "client-${i}" -d ubi9/ubi-init sleep 2 done ``` 3. Stop all container(s) 4. Increase `engine.num_locks` to `16384` in `~/.config/containers/containers.conf` ``` [engine] num_locks = 16384 ``` 6. Run `podman system renumber` or `podman system df` **Describe the results you received:** Received the following error: ``` Error: failed to open 16384 locks in /libpod_rootless_lock_1000: numerical result out of range ``` **Describe the results you expected:** I would expect `podman system renumber` to remove the `libpod_rootless_lock_1000` file which is too small and create a new one with the correct size. It took me a little time to figure out that Podman was referring to a file in `/dev/shm`, so maybe add full path to the rootless lock file in error message. Manually workaround: ``` rm /dev/shm/libpod_rootless_lock_1000 ``` A more generic workaround: ``` rm "/dev/shm/libpod_rootless_lock_$(id -u)" ``` **Additional information you deem important (e.g. issue happens only occasionally):** **Output of `podman version`:** ``` Client: Podman Engine Version: 4.2.1 API Version: 4.2.1 Go Version: go1.18.5 Built: Wed Sep 7 21:58:19 2022 OS/Arch: linux/amd64 ``` **Output of `podman info`:** ``` host: arch: amd64 buildahVersion: 1.27.0 cgroupControllers: - cpu - io - memory - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.1.4-2.fc36.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.4, commit: ' cpuUtilization: idlePercent: 91.26 systemPercent: 3.33 userPercent: 5.42 cpus: 8 distribution: distribution: fedora variant: workstation version: "36" eventLogger: journald hostname: linux-bth idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.19.13-200.fc36.x86_64 linkmode: dynamic logDriver: journald memFree: 1153249280 memTotal: 16574967808 networkBackend: cni ociRuntime: name: crun package: crun-1.6-2.fc36.x86_64 path: /usr/bin/crun version: |- crun version 1.6 commit: 18cf2efbb8feb2b2f20e316520e0fd0b6c41ef4d spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64 version: |- slirp4netns version 1.2.0-beta.0 commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64 libslirp: 4.6.1 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.3 swapFree: 16936067072 swapTotal: 16940785664 uptime: 29h 43m 31.00s (Approximately 1.21 days) plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan - ipvlan volume: - local registries: search: - registry.fedoraproject.org - registry.access.redhat.com - docker.io - quay.io store: configFile: /home/bth/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: {} graphRoot: /home/bth/.local/share/containers/storage graphRootAllocated: 245118922752 graphRootUsed: 198633914368 graphStatus: Backing Filesystem: extfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 12 runRoot: /run/user/1000/containers volumePath: /home/bth/.local/share/containers/storage/volumes version: APIVersion: 4.2.1 Built: 1662580699 BuiltTime: Wed Sep 7 21:58:19 2022 GitCommit: "" GoVersion: go1.18.5 Os: linux OsArch: linux/amd64 Version: 4.2.1 ``` **Package info (e.g. output of `rpm -q podman` or `apt list podman` or `brew info podman`):** ``` podman-4.2.1-2.fc36.x86_64 ``` **Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)** Yes, `v4.2.1` is the latest release at the moment and there is no information in the troubleshooting guide. **Additional environment details (AWS, VirtualBox, physical, etc.):** Fedora 36 on physical Intel.

Received the following error:
Error: failed to open 16384 locks in /libpod_rootless_lock_1000: numerical result out of range
Describe the results you expected:

I would expect podman system renumber to remove the libpod_rootless_lock_1000 file which is too small and create a new one with the correct size.

It took me a little time to figure out that Podman was referring to a file in /dev/shm, so maybe add full path to the rootless lock file in error message.

Manually workaround:
rm /dev/shm/libpod_rootless_lock_1000
A more generic workaround:
rm "/dev/shm/libpod_rootless_lock_$(id -u)"

	// DefaultSHMLockPath is the default path for SHM locks
	DefaultSHMLockPath = /libpod_lock
	// DefaultRootlessSHMLockPath is the default path for rootless SHM locks
	DefaultRootlessSHMLockPath = /libpod_rootless_lock

kemboielvis22 · January 14, 2025, 10:34pm

$ podman system renumber 
Error: failed to open 4096 locks in /libpod_lock: numerical result out of range

Resolved using

$ sudo rm /dev/shm/libpod_lock
$ sudo systemctl restart podman

kemboielvis22 · January 14, 2025, 10:40pm

Thank you have been able to resolve