F2FS-fs (loop0) inconsistent node block on Proxmox and VM

NethSecurity
,
1

Hello,

I am experiencing the following issue, which has been occurring for some time now. The only current workaround is to recreate the VM from scratch.

[   25.416039] F2FS-fs (loop0): inconsistent node block, nid:3234, node_footer[nid:6862,ino:6862,ofs:0,cpver:1927202292601017288,blkaddr:7488]
...
[   55.056365] F2FS-fs (loop0): inconsistent node block, nid:3233, node_footer[nid:6858,ino:6858,ofs:0,cpver:1927202292601017288,blkaddr:7485]

Here is the disk space usage information (df -h):

:~# df -h
Filesystem          Size  Used Avail Use% Mounted on
/dev/root            56M   56M     0 100% /rom
tmpfs               3.9G   19M  3.9G   1% /tmp
/dev/loop0          243M  164M   80M  68% /overlay
overlayfs:/overlay  243M  164M   80M  68% /
/dev/sda1            16M  6.9M  9.2M  43% /boot
tmpfs               512K     0  512K   0% /dev

This issue only occurs on the secondary (backup) router. The primary router does not experience this problem at all.

The VM has never been forcefully terminated; it is always shut down properly via the poweroff command.

I am running this VM on Proxmox. Here is the configuration file:

agent: 1,fstrim_cloned_disks=1
bios: ovmf
boot: order=sata0;scsi0
cores: 8
cpu: host,flags=+aes
machine: q35,viommu=intel
memory: 8192
meta: creation-qemu=10.0.2,ctime=1757445954
name: backup-router
net0: virtio=BC:24:11:8E:A7:27,bridge=vmbr0,mtu=9000,queues=8
numa: 0
onboot: 1
ostype: l26
sata0: none,media=cdrom
scsi0: NVMe:998/vm-998-disk-0.qcow2,discard=on,iothread=1,size=331644416,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=85a77ed3-c154-4d27-a5a5-f4bb46ccdfad
sockets: 1
tablet: 0
vmgenid: a8319a2e-af21-4206-9cf1-d3a0f7d7b7db
vmstatestorage: NVMe

I have previously tested this on an HDD array managed by a hardware RAID controller, and currently, it runs on a ZFS pool consisting of 4 disks in a RAID10 configuration, but the issue persists.

I can provide additional logs or information if needed.

Does anyone have any idea how to resolve this filesystem corruption?

Best regards,

A.H.

2

How, step by step, did you recreate the VM?

Are you hosting other VMs on the same Proxmox system without issue?

I’d reflash the system: Updates → Update with images file, using the latest image downloaded from nethsecurity.org.

3

Hello,

Are there any other VMs on the server: YES, about 10-15 VMs, only this one has a problem.

I currently have 3 HVs with proxmox. As Primary and Secondary routers they are made in the same way:
Creating a VM. I do this:

  1. wget ‘https://updates.nethsecurity.nethserver.org/stable/8.7.2/targets/x86/64/nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img.gz’
  2. gunzip nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img.gz
  3. mv nethsecurity-8.7.1-x86-64-generic-squashfs-combined-efi.img vm-998-disk-0.raw

After that I start the server.
When the file system on the Secondary router breaks via the web:
System > update > Update with image file > upload a new one to reflash the VM.

Then I install the packages I need.

Sometimes the disk has problems while it’s working. Sometimes after stopping the VM with shutdown and starting it up there are problems again.
I can’t say what exactly breaks the file system. As far as I understand, F2FS is a bit sensitive to asynchronous writing.

I’ve tried it on a ZFS array. I’ve tried it on an SSD array with a raid controller.
I’ve also tried restoring from backup. And it only works on the Secondary router.

What I’m planning in the next few days is to download the img file again and create a new VM. Using the disk with these parameters:
cache=writethrough - as I’m not currently using cache on the qcow2 disk.
I will remove this option too - iothread=1

After the last update there is a recurring problem with pve-qemu-kvm (still under investigation).

It is possible that I am doing something wrong and this is causing problems on the disk.

Regards,
A.H.

4

I guess there’s a typo as you’re downloading the 8.7.2 image but moving the 8.7.1 one.

Did you already try to import the disk instead of moving it as explained in the docs?

qm importdisk 998 nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img local-lvm
5

@mrmarkuz I did as you suggested.

root@minihv:~# cd /mnt/storage/images/
root@minihv:/mnt/storage/images# wget 'https://updates.nethsecurity.nethserver.org/stable/8.7.2/targets/x86/64/nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img.gz'
--2026-05-21 20:44:51-- https://updates.nethsecurity.nethserver.org/stable/8.7.2/targets/x86/64/nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img.gz
Resolving updates.nethsecurity.nethserver.org (updates.nethsecurity.nethserver.org)... 104.18.42.227, 172.64.145.29, 2a06:98c1:3105::6812:2ae3, ...
Connecting to updates.nethsecurity.nethserver.org (updates.nethsecurity.nethserver.org)|104.18.42.227|:443... connected.
HTTP request sent, waiting for response... 200 OK
Length: 65000172 (62M) [application/gzip]
Saving to: 'nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img.gz'

nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.im 100%[========================================================================================================================>] 61.99M 91.4MB/s in 0.7

2026-05-21 20:44:52 (91.4 MB/s) - 'nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img.gz' saved [65000172/65000172]

root@minihv:/mnt/storage/images# gunzip nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img.gz

gzip: nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img.gz: decompression OK, trailing garbage ignored
root@minihv:/mnt/storage/images# qm importdisk 998 nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img storage
importing disk 'nethsecurity-8.7.2-x86-64-generic-squashfs-combined-efi.img' to VM 998 ...
Formatting '/mnt/storage/images/998/vm-998-disk-0.raw', fmt=raw size=331644416 preallocation=off
transferred 0.0 B of 316.3 MiB (0.00%)
...
transferred 316.3 MiB of 316.3 MiB (100.00%)
unused0: successfully imported disk 'storage:998/vm-998-disk-0.raw'
root@minihv:/mnt/storage/images# qm set 998 --scsi0 storage:998/vm-998-disk-0.raw
update VM 998: -scsi0 storage:998/vm-998-disk-0.raw
root@minihv:/mnt/storage/images# qm set 998 --boot order=scsi0
update VM 998: -boot order=scsi0
root@minihv:/mnt/storage/images#

Unfortunately, I can’t reproduce what’s causing the disk crash.

Looking for a solution to another problem, I can assume that discard=1 might be causing an inconsistent node block, but I’m not sure.

I rebuilt the secondary router, and it seems to be working fine for now, by removing discard and configuring cache=writethrough.

As a reminder, to avoid confusion:
The parameters ssd=1, discard=1, are set manually in promox, not by the commands specified in the documentation. I do this to protect SSD/NVMe disks from wear.
Since I only enable the parameter ssd=1

Thanks for the suggestion, I’ll monitor and share feedback.

1 Like