Hard drive filled up, now web page, sogo and nextcloud won't sign in

#1

My hard drive filled up and I had to ssh in to delete a ton of files. Most were logs and nextcloud version history files.

After getting some space back and rebooting, nextcloud and SOGo start up but our usernames and passwords dont work. It just spins for a long time. The admin page on port 980 doesn’t seem to be starting up at all.

A symptom that I didn’t realize at the time was the reason I deleted version and trash files in nextcloud the rough and dirty way. I was trying to use the occ command to do it, but it would get part of the way into in and it would only see an admin user and would stop about halfway through saying it lost communication to the ldap server.

So the auth is severely pooched on my install. How can I recover?

#2

Which account-provider do you use (AD/DC - nsdc; openLDAP - slapd)?
Check if the related services and httpd-admin service are running:

systemctl status sssd slapd nsdc httpd-admin -l

…and start them if they aren’t.

1 Like
#3

OK… you got me one stop towards fixing it. I ran the command… slapd isn’t there, ad/samba is running, sssd not found. httpd-admin was throwing an error and I dug around and found the log file and it was complaining because I had gotten a little too frisky when I was emergency deleting log files and accidentally deleted the whole httpd-admin folder in /var/log. I didn’t check closely enough when I tried to delete all the http-. and thought I double checked I wouldn’t get any folders with that command… but I missed that one. I recreated that and started the service and now I have access to httpd-admin again.

BUT as I was logging in to the web admin, a warning flashed by saying generic account provider error. I went to the accounts provider area and it is set to samba/ldap but I need some help diagnosing that one further.

#4

Got the error to flash by again, and got a copy paste on it before it disappeared… Account provider generic error: SSSD exit code 1

I went back and re-ran your command again and looked closer. Here is the whole output of the slapd sssd section. I don’t know whether this is telling me it isn’t there because it’s not what I have it configured to use or not there because it isn’t running and should be. On the other hand… here is the output of the entire command.

● sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-28 21:04:55 EST; 14h ago
 Main PID: 1318 (sssd)
   CGroup: /system.slice/sssd.service
           ├─1318 /usr/sbin/sssd -i --logger=files
           ├─1360 /usr/libexec/sssd/sssd_be --domain cgetut.com --uid 0 --gid 0 --logger=files
           ├─1640 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
           └─1641 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --logger=files

Nov 28 21:04:54 obscuredhost.com systemd[1]: Starting System Security Services Daemon...
Nov 28 21:04:54 obscuredhost.com sssd[1318]: Starting up
Nov 28 21:04:54 obscuredhost.com sssd[be[cgetut.com]][1360]: Starting up
Nov 28 21:04:55 obscuredhost.com sssd[pam][1641]: Starting up
Nov 28 21:04:55 obscuredhost.com sssd[nss][1640]: Starting up
Nov 28 21:04:55 obscuredhost.com systemd[1]: Started System Security Services Daemon.
Nov 28 21:05:07 obscuredhost.com sssd[be[cgetut.com]][1360]: Backend is offline
Unit slapd.service could not be found.

● nsdc.service - NethServer Domain Controller container
   Loaded: loaded (/usr/lib/systemd/system/nsdc.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-28 21:04:54 EST; 14h ago
     Docs: man:systemd-nspawn(1)
 Main PID: 1256 (systemd-nspawn)
   Status: "Container running."
   CGroup: /machine.slice/nsdc.service
           ├─1256 /usr/bin/systemd-nspawn --quiet --keep-unit --boot --network-bridge=br0 --machine=nsdc --capability=CAP_SYS_TIME
           ├─1270 /usr/lib/systemd/systemd
           └─system.slice
             ├─samba.service
             │ ├─1719 /usr/sbin/samba -i --debug-stderr
             │ ├─2077 /usr/sbin/samba -i --debug-stderr
             │ ├─2078 /usr/sbin/samba -i --debug-stderr
             │ ├─2079 /usr/sbin/samba -i --debug-stderr
             │ ├─2080 /usr/sbin/samba -i --debug-stderr
             │ ├─2081 /usr/sbin/samba -i --debug-stderr
             │ ├─2084 /usr/sbin/samba -i --debug-stderr
             │ ├─2085 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
             │ ├─2087 /usr/sbin/samba -i --debug-stderr
             │ ├─2094 /usr/sbin/samba -i --debug-stderr
             │ ├─2099 /usr/sbin/samba -i --debug-stderr
             │ ├─2103 /usr/sbin/samba -i --debug-stderr
             │ ├─2108 /usr/sbin/samba -i --debug-stderr
             │ ├─2116 /usr/sbin/samba -i --debug-stderr
             │ ├─2119 /usr/sbin/samba -i --debug-stderr
             │ ├─2129 /usr/sbin/samba -i --debug-stderr
             │ ├─2131 /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
             │ ├─2147 /usr/sbin/samba -i --debug-stderr
             │ ├─2348 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
             │ ├─2349 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
             │ └─2362 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
             ├─console-getty.service
             │ └─1672 /sbin/agetty --noclear --keep-baud console 115200,38400,9600 vt220
             ├─systemd-logind.service
             │ └─1662 /usr/lib/systemd/systemd-logind
             ├─dbus.service
             │ └─1622 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
             ├─ntpd.service
             │ └─1651 /usr/sbin/ntpd -u ntp:ntp -g
             └─systemd-journald.service
               └─1454 /usr/lib/systemd/systemd-journald

Nov 28 21:04:55 obscuredhost.com systemd-nspawn[1256]: [  OK  ] Started Network Service.
Nov 28 21:04:55 obscuredhost.com systemd-nspawn[1256]: [  OK  ] Reached target Network.
Nov 28 21:04:55 obscuredhost.com systemd-nspawn[1256]: [  OK  ] Started Samba domain controller daemon.
Nov 28 21:04:55 obscuredhost.com systemd-nspawn[1256]: [  OK  ] Started Login Service.
Nov 28 21:04:55 obscuredhost.com systemd-nspawn[1256]: [  OK  ] Reached target Multi-User System.
Nov 28 21:04:55 obscuredhost.com systemd-nspawn[1256]: [  OK  ] Reached target Graphical Interface.
Nov 28 21:04:55 obscuredhost.com systemd-nspawn[1256]: Starting Update UTMP about System Runlevel Changes...
Nov 28 21:04:55 obscuredhost.com systemd-nspawn[1256]: [  OK  ] Started Update UTMP about System Runlevel Changes.
Nov 28 21:04:56 obscuredhost.com systemd-nspawn[1256]: CentOS Linux 7 (Core)
Nov 28 21:04:56 obscuredhost.com systemd-nspawn[1256]: Kernel 3.10.0-957.27.2.el7.x86_64 on an x86_64

● httpd-admin.service - Server Manager UI httpd instance
   Loaded: loaded (/usr/lib/systemd/system/httpd-admin.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2019-11-28 21:04:55 EST; 14h ago
     Docs: https://github.com/NethServer/nethserver-httpd-admin
 Main PID: 1374 (code=exited, status=1/FAILURE)

Nov 28 21:04:54 obscuredhost.com systemd[1]: Started Server Manager UI httpd instance.
Nov 28 21:04:55 obscuredhost.com httpd[1374]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/httpd-admin/' for main error log
Nov 28 21:04:55 obscuredhost.com httpd[1374]: AH00014: Configuration check failed
Nov 28 21:04:55 obscuredhost.com systemd[1]: httpd-admin.service: main process exited, code=exited, status=1/FAILURE
Nov 28 21:04:55 obscuredhost.com systemd[1]: Unit httpd-admin.service entered failed state.
Nov 28 21:04:55 obscuredhost.com systemd[1]: httpd-admin.service failed.
Nov 29 04:00:36 obscuredhost.com systemd[1]: Server Manager UI httpd instance is not active.

How should I proceed?

#5

sssd log is one that I cleared and deleted for space. But the system is not starting far enough that anything gets put back in to the newly created sssd.log. All the other sssd related logs all show something similar to “The data provider returned an error” or “Data Provider offline”.

systemctl start sssd just returns me to the prompt without an error.

#6

You can forget about slapd, it is for OpenLDAP account provider.
From the command outcome sssd and nsdc are running, and for httpd-admin you already found a fix.

To fix the account provider error there are others than will be able to provide better help, but you can try:

signal-event nethserver-sssd-update
signal-event nethserver-dc-update
#7

I ran those and it said command not found. I did a search and it found signal-event in /usr/sbin/e-smith
but it still didn’t find the program when I copy and pasted your commands in. Since the command I found is signal-event I tried both commands without the dash before nethserver. Just a space there. I got no error, just returned to the command prompt. Nothing changed though. Still can’t sign in to nextcloud or sogo.

#8

The troubleshooting steps from this thread can be useful:

#9

Here is the output of both commands in that. If you need more let me know. The 2nd command output was huge, but I tried to copy all the way back to the last time I rebooted, but not before tha

{
   "BindDN" : "ldapservice@AD.Cobscuredhost.COM",
   "LdapURI" : "ldaps://nsdc-webmail.ad.cobscuredhost.com",
   "DiscoverDcType" : "ldapuri",
   "StartTls" : "",
   "port" : 636,
   "host" : "nsdc-webmail.ad.cobscuredhost.com",
   "isAD" : "1",
   "isLdap" : "",
   "UserDN" : "dc=ad,dc=cobscuredhost,dc=com",
   "GroupDN" : "dc=ad,dc=cobscuredhost,dc=com",
   "BindPassword" : "djoli_LA2W",
   "BaseDN" : "dc=ad,dc=cobscuredhost,dc=com",
   "LdapUriDn" : "ldap:///dc%3Dad%2Cdc%3Dcobscuredhost%2Cdc%3Dcom"
}

Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd-networkd[23]: Enumeration completed
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd-networkd[23]: host0           : gained carrier
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd-networkd[23]: host0           : link configured
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started Network Service.
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Reached target Network.
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started Samba domain controller daemon.
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started Login Service.
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Reached target Multi-User System.
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Reached target Graphical Interface.
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Starting Update UTMP about System Runlevel Changes...
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd-logind[24]: New seat seat0.
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started Update UTMP about System Runlevel Changes.
Nov 28 21:04:55 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Startup finished in 1.182s.
Nov 28 21:04:56 nsdc-webmail.ad.cobscuredhost.com samba[26]: samba version 4.8.12 started.
Nov 28 21:04:56 nsdc-webmail.ad.cobscuredhost.com samba[26]: Copyright Andrew Tridgell and the Samba Team 1992-2018
Nov 28 21:04:56 nsdc-webmail.ad.cobscuredhost.com samba[26]: binary_smbd_main: samba: using 'standard' process model
Nov 28 21:04:57 nsdc-webmail.ad.cobscuredhost.com winbindd[43]: [2019/11/28 21:04:57.984110,  0] ../source3/winbindd/winbindd_cache.c:3160(initialize_winbindd_cache)
Nov 28 21:04:57 nsdc-webmail.ad.cobscuredhost.com winbindd[43]:   initialize_winbindd_cache: clearing cache and re-creating with version number 2
Nov 28 21:04:58 nsdc-webmail.ad.cobscuredhost.com winbindd[43]: [2019/11/28 21:04:58.586833,  0] ../lib/util/become_daemon.c:136(daemon_ready)
Nov 28 21:04:58 nsdc-webmail.ad.cobscuredhost.com winbindd[43]:   daemon_ready: daemon 'winbindd' finished starting up and ready to serve connections
Nov 28 21:04:58 nsdc-webmail.ad.cobscuredhost.com smbd[34]: [2019/11/28 21:04:58.805222,  0] ../lib/util/become_daemon.c:136(daemon_ready)
Nov 28 21:04:58 nsdc-webmail.ad.cobscuredhost.com smbd[34]:   daemon_ready: daemon 'smbd' finished starting up and ready to serve connections
Nov 28 21:20:01 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Starting Cleanup of Temporary Directories...
Nov 28 21:20:01 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started Cleanup of Temporary Directories.
Nov 29 03:59:53 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Created slice system-nsdc\x2drun.slice.
Nov 29 03:59:53 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started nsdc-run worker process.
Nov 29 15:10:10 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started nsdc-run worker process.
Nov 29 15:19:54 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Failed to duplicate autofs fd: Bad file descriptor
Nov 29 15:27:06 nsdc-webmail.ad.cobscuredhost.com dbus[21]: [system] Reloaded configuration
Nov 29 15:27:06 nsdc-webmail.ad.cobscuredhost.com dbus[21]: [system] Reloaded configuration
Nov 29 15:27:07 nsdc-webmail.ad.cobscuredhost.com dbus[21]: [system] Reloaded configuration
Nov 29 15:27:07 nsdc-webmail.ad.cobscuredhost.com dbus[21]: [system] Reloaded configuration
Nov 29 15:27:07 nsdc-webmail.ad.cobscuredhost.com dbus[21]: [system] Reloaded configuration
Nov 29 15:27:07 nsdc-webmail.ad.cobscuredhost.com dbus[21]: [system] Reloaded configuration
Nov 29 15:27:07 nsdc-webmail.ad.cobscuredhost.com dbus[21]: [system] Reloaded configuration
Nov 29 15:27:07 nsdc-webmail.ad.cobscuredhost.com dbus[21]: [system] Reloaded configuration
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Stopping Samba domain controller daemon...
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 42 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com winbindd[43]: [2019/11/29 15:27:16.312289,  0] ../source3/winbindd/winbindd.c:239(winbindd_sig_term_handler)
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 30 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com winbindd[43]:   Got sig[15] terminate (is_parent=1)
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 28 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 31 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 36 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 44 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 39 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 33 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 40 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 29 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 38 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 35 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[26]: sigterm_signal_handler: Exiting pid 37 on SIGTERM
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Stopped Samba domain controller daemon.
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started Samba domain controller daemon.
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Reloading.
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[1976]: samba version 4.9.13 started.
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[1976]: Copyright Andrew Tridgell and the Samba Team 1992-2018
Nov 29 15:27:16 nsdc-webmail.ad.cobscuredhost.com samba[1976]: binary_smbd_main: samba: using 'standard' process model
Nov 29 15:27:17 nsdc-webmail.ad.cobscuredhost.com winbindd[2004]: [2019/11/29 15:27:17.072100,  0] ../source3/winbindd/winbindd_cache.c:3161(initialize_winbindd_cache)
Nov 29 15:27:17 nsdc-webmail.ad.cobscuredhost.com winbindd[2004]:   initialize_winbindd_cache: clearing cache and re-creating with version number 2
Nov 29 15:27:17 nsdc-webmail.ad.cobscuredhost.com winbindd[2004]: [2019/11/29 15:27:17.090139,  0] ../lib/util/become_daemon.c:136(daemon_ready)
Nov 29 15:27:17 nsdc-webmail.ad.cobscuredhost.com winbindd[2004]:   daemon_ready: daemon 'winbindd' finished starting up and ready to serve connections
Nov 29 15:27:17 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started nsdc-run worker process.
Nov 29 15:27:17 nsdc-webmail.ad.cobscuredhost.com smbd[1994]: [2019/11/29 15:27:17.300840,  0] ../lib/util/become_daemon.c:136(daemon_ready)
Nov 29 15:27:17 nsdc-webmail.ad.cobscuredhost.com smbd[1994]:   daemon_ready: daemon 'smbd' finished starting up and ready to serve connections
Nov 29 15:27:17 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Failed to duplicate autofs fd: Bad file descriptor
Nov 29 21:20:01 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Starting Cleanup of Temporary Directories...
Nov 29 21:20:01 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started Cleanup of Temporary Directories.
Nov 30 05:36:41 nsdc-webmail.ad.cobscuredhost.com systemd[1]: Started nsdc-run worker process.
#10

Don’t see anything strange, only that error but don’t know if it is important.

Did you check other troubleshooting steps from the other thread?

#11

Thanks for showing me how to get the diag outputs. It seems my two issues were only remotely related. Long story short, I had added a server on 192.168.1.6 which was the same IP address as the sssd config. It didn’t show up until I rebooted the server the first time after the hard drive filled up.

So the original problem of the full drive really was the culprit behind the admin page failing, but the authentication problem was a result of another issue that didn’t show up until the reboot of the first problem. Once I got the IP conflict fixed everything started working again.

Thanks for all the help and the commands that allowed me to see the root cause! I had really forgotten that Netserver had 2 IP associated with it.

1 Like
#12

Glad you got it working, thanks for sharing the solution with us (reassigned the solution mark to your post as it better describes the situation).
Did you find out why the drive filled up (so it wont bite again in the near future)?