I run NS8 in a KVM vm, since the release of NS8 (having migrated from NS7).
Yesterday it started complaining that backups couldn’t complete. I didn’t realize it was a storage issue, since it supposedly backs up in a remote share with no space issues.
The truth is that NS8 own virtual disk grew full.
So I stopped the VM, and in my host (UNRAID) console, I did a
qemu-img resize my-whatever.qcow2 +24G
(it was 40GB before)
qemu-img info, showed me that indeed the virtual disk grew to 64GB, although I expected that NS8 wouldn’t “know” yet, as I probably need to grow the partition itself?
Thing is, I booted the VM, it booted ok with seemingly no errors in the console, but I DO NOT see cluster admin any more! It says “cannot create task cluster status” or something like that.
I don’t know if it is related with the resize or the vm has an issue that there was no space before the reboot and now it can’t even boot properly. In the console I am at the login.
What do I do??? How do I properly grow the partition (if that is the issue) or how do I make it work anyway?
…maybe the server should alert admin in a more dynamic way that something is wrong?
I mean, backup failing (initially just one of the backup sources, the rest ok), doesn’t really alert you that the whole server is in a spiral of death!
IN ANY CASE, to the problem at hand… I followed a similar path to that thread that solved the issue back then, but it did NOT solve anything in this case!
My system is still dead.
I did “growpart /dev/vda 5” (the partition that seems to have been full).
I rebooted the vm… initially for a few seconds I DID get a cluster admin screen, I went on to see the disks (I don’t remember where, as I don’t have a working cluster admin working now) and I happened to see “no disk” where the three partitions should be enumerated!
After that, cluster admin went down again!!!
I did lsblk in the console and I DO see vda5 now has the new size (62.9G from 39.somethingG).
But… no go!
What happened?
…also not that vda5 seems to show two mountpoints (this was also before I resized anything) and I don’t know if this is correct:
/var/lib/containers/storage/overlay
and
/
(which / is the only I expected to see)
OK I resolved everything.
I probably need to document everything for people to know what to do (and seemingly… my future self too )…
If you start seeing the server act weird, it is a good idea to go to nodes and check the node if it has an issue. Check disks and possibly / is full!
If it is, you need to stop the vm (assuming NS8 is on a vm - act accordingly) and on the host grow the virtual disk. (qemu-img resize /whatever.qcow2 +SizeInGYouWantToGrow)
This only grows the disk not the partitions inside.
You reboot so that the vm acknowledges the resize (probably).
On NS8 console, you do “lsblk” to see which mount point, is that needs resize. Usually (in Rocky Linux pre-made image), it is vda5. There you can also see that the “disk” (vda) already shows the new size.
So if (as expected) the partition to grow is vda5 you do: “growpart /dev/vda 5” (with the space).
This grows the partition, but you still HAVEN’T resized the filesystem inside it (!). In NS8 case, it is xfs. So next setp is “sudo xfs_growfs -d /”… and you should be set.
System should be working now, but I would reboot once more anyway.
)…