[Solved] Can't access services after complete update and full backup

server-manager
v7

(Robert ) #1

NethServer Version: 7
Module:
First I want to thank you for providing such a great all-in-one-solution for mail, nextcloud, shared folders and so on. It’s important to me to know where my data is. That’s why I decided to deal with Nethserver as base system for a small family home server.
Last night I did an complete update of the base system with “yum update” followed by “yum upgrade”. Everything seems to work as supposed to. I could login to webmail and nextcloud for example.
This morning I recognized that I was not able to connect to the mailserver, because username and password would not match.
I tried to access the Server-Manager/Dashboard via https://hostname:980 as I did it before a lot of times successfully, but today the page is empty so that I can’t log in to the server-manager.
Every night there is an incremental backup to an external usb-storage. At Sunday there is also an complete backup to the same device.
What can I do now?
Thank you very much.


[solved] Data-Backup to USB-disk needs the same space at local file-system for "security-copy"
(Federico Ballarini) #2

First thing.
Can you access via SSH?
If yes, try to launch commands:

systemctl restart httpd-admin
systemctl restart httpd

and post the output if there are errors.

If there isn’t any error try to access to server manager via IP or via dns name.


(Robert ) #3

I could launch both commands without any errors,
but after that I could not access to the server manager nor with ip nor with dns name.


(Federico Ballarini) #4

Have you already tried to reboot the server?


(Robert ) #5

yes, i tried
sudo reboot
from the command line.
Maybe I should try it again after both commands you gave me worked without any errors?


(Federico Ballarini) #6

Wait… we can analyze the problem before…

Check log with this command
less /var/log/httpd-admin/error_log
and
less /var/log/messages

if there is something interesting.

Then check also if your drive is full with df -h command


(Robert ) #7

oh, this seems to be the right way!
the first command gives for example
[Mon Feb 18 15:37:30.972215 2019] [log_config:warn] [pid 1053] (28)No space left on device: [client 192.168.251.40:
the second command gives for example
Feb 18 16:00:01 nethserver crond: postdrop: warning: uid=48: No space left on device
and the third command finally shows, that there is no more space on the root device:
/dev/mapper/centos_nethserver-root 48G 48G 0 100% /


(Federico Ballarini) #8

Try to clean something of your data into /var/lib/nethserver if you can.
You can also swap with swapoff -a && swapon -a


(Robert ) #9

I’m not sure if I understand the plan.

  1. Should I try to copy some files from the root device to /var/lib/nethserver?
  2. Or should I try to copy some files from /var/lib/nethserver to another device for example?
    The swapoff and swapon-comands are new to me.

(Federico Ballarini) #10

You have to free space from your server.
You need to remove or move some files from your server to another device or delete something you don’t need. Your files are stored or into /root or into /var/lib/nethserver/

If you use webtop cloud or nextcloud you have to enter into their folders and remove or download and save into your PC some files to free space.

If there are only emails on your server you have to cleanup mail accounts.


(Robert ) #11

I have shared folders, the data is stored on a second hdd (sdc1) in the server. It is not necessary to delete something there, isn’t it?
Then I have Nextcloud files and I must confess, that I don’t know where those files are stored.
Then there is the Root-partition. Im not sure what I could delete here, because here are the system files from nethserver, right?
Also there is a home-partition on the same device as root which seems to have enough space:

    Filesystem                          Size  Used Avail Use% Mounted on
    /dev/mapper/centos_nethserver-root   48G   48G     0 100% /
    devtmpfs                            1.6G     0  1.6G   0% /dev
    tmpfs                               1.6G     0  1.6G   0% /dev/shm
    tmpfs                               1.6G  9.1M  1.6G   1% /run
    tmpfs                               1.6G     0  1.6G   0% /sys/fs/cgroup
    /dev/sda1                           459G  300G  136G  69% /mnt/backup/nethserver
    /dev/sdc1                           2.7T   60G  2.5T   3% /var/lib/nethserver/ibay
    /dev/sdb1                           976M  156M  754M  18% /boot
    /dev/mapper/centos_nethserver-home   48G   53M   46G   1% /home
    tmpfs                               326M     0  326M   0% /run/user/0

(Federico Ballarini) #12

You need to free space from this partition. For nextcloud and webtop if you can access via web you can delete files from it.


(Robert ) #13

These are the directories in the root-folder:
bin dev home lib64 media opt razor-agent.log run srv tmp var
boot etc lib lost+found mnt proc root sbin sys usr
Which one can I delete or clean without any problems? I’m really not sure which files are needed by the system an which I could delete.


(Federico Ballarini) #14

You can find your files into /var/lib/nethserver/ibay if you have shared folders.
Check also into /root if you have something you can clean.
/var/tmp folder can be cleaned without any problem.


(Robert ) #15

Thank you for your patience!

Now I cleaned the /var/tmp folder.
In /root I have those files:

 .bash_logout   .cshrc              original-ks.cfg  .spamassassin
..               .bash_profile  .gnupg              .pki             .tcshrc
anaconda-ks.cfg  .bashrc        kickstart-post.log  .razor
.bash_history    .cache         .my.cnf             .rnd

I guess I am not allowed to delete them all, right?

In /var/lib/nethserver/ibay I have my shared folders on another partition, that is mounted to that point. So if I delete something there it won’t affect the root-partition, right?

Is there something else that I can delete?


(Federico Ballarini) #16

These are all small files. Don’t clean them.

If they are correctly mounted to the other disk you have not to clean them.

Enter in Nextcloud and Webtop and check if there are files here.

Then try also to launch command df -h and check empty space. If it’s enough try to connect to server manager.


(Robert ) #17

Can’t enter nextcloud from the webinterface!
I don’t use the webtop-module.
It’s not enough space at this moment. Connection to server-manager ist not possible.

If there aren’t any other files that I can delete from the root-partition, is there a way to get back to the state of the system before the update?


(Federico Ballarini) #18

Nextcloud files are stored here: try to check /var/lib/nethserver/nextcloud/user

Note that username it’s shown like this d7f37f86-c02e-1035-87dc-838d1385dc2d


(Federico Ballarini) #19

Hi @tscheepy
Any news? Can you give us a feedback?


(Robert ) #20

Hi federico.ballarini,
fortunately I am able to access the server-manager, nextcloud, webmail and anything seems to work as before. That’s why I’m very happy at the moment, tkank you!
Meanwhile I could determine why there was no space at the root-partition and this is related with the daily backup, which creates an folder
/mnt/backup-backup-data/nethserver/
at the root-partition to store the duplicity-full.*.*difftar.gz files there, although that’s not the folder where the backup-files are supposed to be stored as I set to in the backup-module in server-manager:
/mnt/backup/nethserver/
where my external drive is mounted to. This configuration worked as supposed to till the above mentioned complete update of the base system. That’s why I will open a new thread.
Summary: All problems related to the nethserver services like server-manager, nextcloud, webmail, … could be solved by ensuring that there is enough space at the root-partition. Therefore it was very important to check the free space at the root-partition, as mentioned in the 10th post above. So this thread could be closed as fixed.