So I installed an NS8 (not my first), for a friend SOHO, using the pre-made image (Rocky), just for mail services, nothing else (yet). Set a single email, connected it to a few mail clients and told them, they can move PROGRESSIVELY older mail (for archiving) to that account.
They did that for a day or so, then they reported me, that it doesn’t seem to sync with the server (their mail clients).
So I went to the web interface only to find it was NOT available!
Then I rebooted NS8 (it is a VM) AND NOW IT DOES NOT BOOT NS8, but only Linux (the console prompt says “Rocky Linux 9.5 (Blue Onyx)” but NOT the cluster admin URLs!!!
HELP! What can I do!? I need to at least access the mail that was already transfered to their server!
What happened? Could it be overwhelmed by some sudden “attack” of mailbox transfers? (their emails are more than 100GB total… they may have tried to move folders NOT progressively after some point)
How could this destroy NS8 itself???
EDIT:
Note that the server’s qcow image has grown to 37GB and lives inside an array that has plenty of more space to grow.
Is it possible it cannot grow more for some reason and is full or something?
EDIT #2:
Even network is not up. I cannot ping the server or the server ping anything else.
I used NS7 for years and NS8 almost since the beginning. I have stable NS8 installs, I never expected a server would crash in a day and NOT boot NS8 at all…
At this point I am looking even at just using this qcow as source image to setup a fresh NS8 and get (somehow) the mailbox from the “old” one (where “old” means a few days ago…). I will try anything you can propose! Just help please…
Usually you should boot the fresh rocky image and setup the network, then go to cluster-admin and setup NS8 like create a cluster. If you reboot before you setup NS8 you get a broken network/cluster-admin etc. as you explained it.
But that never happened when the NS8 was already working.
Maybe you can get some helpful logs using journalctl
I did find the mailbox in the directory you mentioned. It does have content. Is there a specific file I should look for in that tree, that “should” be big enough (they probably migrated 10+GB) to verify I have it?
(EDIT: There seem to be some .dot folders which seem to be the mailbox folders… They have gargled text, which I assume are how mail server translates… Greek folders.)
As I said, this was not my first NS8, or second or third (and my first NS8 itself was installed maybe 10 times as I started from beta). This one was properly set to network, was reboot at least 2-3 times already and worked fine. This is why I told them to start migrating mail. As I said, this is not just broken cluster-admin… I doubt any NS8 services are running at all. Among other services!
NMTUI itself doesn’t even work! it says “Could not contact NetworkManager: Could not connect: Connection refused.”!!!
So if mailbox survived, how I import it to a fresh NS8? (assuming I attach this qcow as a second disk… as I don’t see any other way)
journalctl has 17k lines and plenty of errors. But since I can just see the console from a web-based shell of the VM (so not “get” capabilities AFAIK)… I don’t know how to retrieve it…
…to be honest I am hoping NS team themselves are to be triggered by how a fresh working NS8 with a single “addon” (mail and webtop- with webtop not even used yet)., just “crashed” fully… when the only actual interaction users had, was to migrate mail folders to the single mailbox using their mail clients (one folder at a time)…
du -h /home/mail1/.local/share/containers/storage/volumes/dovecot-data/_data
Check if containers are running:
podman ps
Check if the api is working:
api-cli run cluster/get-name
Check network services:
ss -tulpn
Install mail app on new NS8
Get the right UID and GID on the new NS8:
root@contabo:~# ls -l /home/mail1/.local/share/containers/storage/volumes/dovecot-data/_data/
total 36
drwx------ 4 427779 427780 4096 May 12 17:47 admin
Put the mail directories and files to the same location on the new NS8
For migrating the mailbox, it is my last resort. We will revisit this IF I go that way.
Cannot dump the log because… see #2… no space! I guess this is the issue! Is this resolvable and if yes, will it allow NS8 to recover?
(but I guess that second part we need to see later)
My host is UNRAID (i.e. KVM). As I said I used the pre-built qcow image, so I didn’t define a size somewhere and the file did grow from it’s initial size (what was it… 2-3GB?) to around 37GB… maybe this is the size defined in the partition table? Can I edit this somehow?
Seems to work.
I didn’t connect any clients (everybody is asleep now in this timezone), but webtop (a good way to test it first time in that install) did connect to the important mailbox and I can see the contents. That’s all that matters.
Thank you very much for your help Markus, appreciated.
…and I hope the system can now handle a single 150GB mailbox…
(yes… don’t ask)
I wonder if NS8 itself could auto set some triggers (free space check) to notify sdmin before catastrophy.
The very least, someone should update documentation to say about the default partition sizes of the pre-built images, as a warning and link to the resize procedure.
