Summary
All rootless modules on one of our NS8 nodes keep their images with the Podman storage driver vfs. Because vfs has no copy-on-write, every image layer is a full copy. On ext4 (no reflinks) this is real disk usage: ~94 GiB, about 9 % of the root filesystem. Nothing in the core scripts we searched sets the driver, so we assume Podman picked it on first use. We would like to know whether this is intended, whether core can limit the growth, and whether you see problems with the cleanup we plan.
Environment (measured 2026-11-04)
NS8 core 3.24.0, single node (VM), Debian 12.15, kernel 6.1.0-54-amd64
No /etc/containers/storage.conf, no /usr/share/containers/storage.conf, no ~/.config/containers/storage.conf in the module homes
runagent -m <module> podman info --format '{{.Store.GraphDriverName}}' prints vfs (checked: traefik1, nextcloud1, wordpress1), without any warning. All 13 rootless module homes have storage/vfs/dir. Rootful podman (crowdsec1, core containers) uses overlay.
…/traefik1/.local/share/containers/storage/vfs was last modified 2024-07-20 08:05. The dpkg list file of fuse-overlayfs is dated 2024-07-20 08:04, so it was already installed when the first vfs storage was created.
A grep for vfs|storage.conf|STORAGE_DRIVER over /var/lib/nethserver/node, /var/lib/nethserver/cluster, /usr/local/sbin, /usr/local/bin, /usr/local/agent and /etc/nethserver (excluding the Python env) finds only a comment in node/bin/refresh-container-metrics (line 112, mentioning vfs-containers). Nothing in these core paths sets the storage driver. The code inside the module images was not checked.
Observations
vfs/dir per module: wordpress1 18 G, roundcubemail1 14 G, metrics1 12 G, samba1 11 G, nextcloud1 11 G, dokuwiki1 9.8 G, piler1 9.7 G, collabora1 5.4 G, loki1 1.6 G, mail1 684 M, traefik1 583 M, ldapproxy1 221 M, imapsync1 157 M (sum ≈ 94 GiB). du of the whole filesystem matches df, so nothing is double counted.
The sizes shown by podman images add up to far less: wordpress1 ≈ 1.5 GB vs 18 G on disk, nextcloud1 ≈ 2.1 GB vs 11 G.
Leftovers after module updates: one dangling image in nextcloud1 (403 MB) and one in wordpress1 (333 MB). nextcloud1 also still has ghcr.io/nethserver/nextcloud:1.1.7 next to 1.7.8.
Volumes live in the same storage root (e.g. …/nextcloud1/.local/share/containers/storage/volumes, the data volume is ~290 GB), so podman system reset is not an option for us.
Impact
This was not the cause of our recent disk alert (that was Nextcloud trash, already fixed). It is a permanent overhead that grows with every module and every image version, and on ext4 it is not shared between modules or layers.
What I plan (nothing beyond this for now)
Per module, one at a time and with low I/O priority: check with podman ps -a that no container uses the dangling image, save it with podman save (rollback via podman load), remove it with podman rmi <id> (no -f), and compare du before and after.
No podman system reset, no podman image prune -a, no podman system prune, no storage-driver change on this production node.
Possibly test overlay on a throw-away node later, only after your feedback.
Questions
Is vfs for rootless modules intentional? If yes, what is the reason (compatibility, user namespaces, stability, performance)? If not, why did Podman end up with vfs although fuse-overlayfs was installed?
Can core prevent the growth, e.g. by shipping a storage.conf that selects overlay for new modules, or by documenting the cost? Are there known problems with native overlay for rootless modules (Podman 4.3.1, kernel 6.1)?
Is there a supported way to move an existing module from vfs to overlay while keeping its volumes?
Should update-module remove superseded images (dangling ones and old module image tags)? Is there a reason to keep them (rollback, retries, restore-module)?
Do you see any concern with our cleanup under “What we plan”?
Related threads (Debian 13 upgrade, similar symptoms):
Hi @capote, thanks for the detailed report. I used AI to help draft this answer, to keep my thoughts and findings together.
vfs is not intentional. Core doesn’t set the storage driver anywhere. Podman 4.3.1 on Debian 12 picks vfs for rootless users by default, even with fuse-overlayfs installed. The driver is recorded in each module’s Podman DB, so it stays vfs for the lifetime of that module instance. A storage.conf alone won’t convert existing modules, and there’s no automatic in-place conversion of the layer store.
Debian 13 (Trixie) uses overlay by default, for both rootful and rootless modules, so the problem goes away with new modules on the new OS. With the existing tools, I’d suggest this path:
Upgrade the node to Trixie first. On Debian 12, new modules would be created with vfs again. Debian 12 is also no longer supported by NS8 from milestone 8.10 on, so the upgrade is needed anyway. If you have more nodes in the cluster, keep the same distro version on all of them.
Rebuild the modules with clone or restore. A module created by clone-module or by restoring a backup is a new instance, so it gets a fresh graphroot using the new default. Check it with runagent -m <module> podman info --format '{{.Store.GraphDriverName}}'.
Do it one module at a time. Verify the new instance, then remove the old one. Start with the small modules.
Keep in mind: make sure you have a verified backup first, you need enough free space for the old and new instance at the same time (relevant for your 290 GB Nextcloud volume), and modules you don’t rebuild stay on vfs after the OS upgrade.
An in-place graphroot conversion might also be possible as an alternative to clone/restore. We have never tested it, but we have some ideas, and we’d be glad to discuss this approach with you further if you’re interested. Please don’t try anything on your production node in the meantime.
Leftover images are a separate topic. update-module is expected to remove images that are no longer used, so a dangling image or an old tag like nextcloud:1.1.7 next to 1.7.8 after an update is unexpected, and we’d treat it as a bug. We’ve fixed the cleanup procedure several times, so details would help:
The module versions before and after the update, and the core version at the time of the update
The update-module output or the related journal lines
podman images and podman ps -a for the affected module (nextcloud1, wordpress1)
Whether the old tag was still in use by a container or left over from a failed or retried update
Your cleanup plan (save, then podman rmi without -f, no reset or prune) is conservative and fine as a stopgap.
I was afraid this was going to give me a headache.
My problem is that this is the only production system at a small company—it’s located several hundred kilometers away and has been running steadily for 629 days.
It’s still running on PVE version 8.3.3 and hasn’t been touched because there’s no IT staff on site.
I can administer it remotely, but no one has dared to upgrade the operating systems—which would require a reboot—since there’s no one on site to at least perform a manual reboot if something freezes.
And it’s not getting any better as the days go by.
I’m going to look for a way to organize the upgrades first. After that, I’d like to tackle this issue together with you all.
I have to admit, I haven’t really understood what this VFS architecture is supposed to achieve. In any case, I haven’t dared to tackle this issue, even with AI support.
I’ll check in again when I’m ready.
Please do the update and reboot first, as soon as you can. With 629 days of uptime and the recent critical kernel vulnerability (CVE-2026-98432 thread), that is far more urgent than anything about disk usage.
About VFS, what could be the point? It’s the simplest storage driver Podman has. Each layer is a plain full copy in a directory, with no overlay mount and no dependency on kernel or filesystem features, so it works almost anywhere. That’s why Podman uses it as a fallback. On Debian 12 it was also the default for rootless users, which is how your modules ended up with it. It wasn’t a design choice of NS8. The only price is disk space, since nothing is shared between layers and images.
So there’s no emergency with VFS: it works fine, it just uses more disk. Please don’t touch it for now. Get the Debian 12 updated first, and we’ll look at the Trixie upgrade and its storage later.