SSSD start failed after update

Vladimir_Pobedinskiy · May 22, 2020, 7:22am

SSSD stopped working after patching system from 7.7 to 7.8**
SSSD service fails to start after upgrading sssd packages

systemctl status sssd

● sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Пт 2020-05-22 16:49:31 +10; 12min ago
  Process: 19154 ExecStart=/usr/sbin/sssd -i ${DEBUG_LOGGER} (code=exited, status=4)
 Main PID: 19154 (code=exited, status=4)
systemd[1]: Starting System Security Services Daemon...
sssd[19154]: ldb: module version mismatch in ../modules/paged_results.c : ldb_version=1.5.4 module_version=1.4.2
sssd[19154]: ldb: failed to initialise module /usr/lib64/ldb/modules/ldb/paged_results.so : Unavailable
sssd[19154]: SSSD couldn't load the configuration database [5]: Input/output error.
systemd[1]: sssd.service: main process exited, code=exited, status=4/NOPERMISSION
systemd[1]: Failed to start System Security Services Daemon.
systemd[1]: Unit sssd.service entered failed state.
systemd[1]: sssd.service failed.

/etc/sssd/sssd.conf

[sssd]
domains = ***.ru
config_file_version = 2
services = nss, pam
default_domain_suffix = ***.ru

[domain/***.ru]
use_fully_qualified_names = True
id_provider = ldap
ldap_uri = ldap://127.0.0.1
ldap_search_base = dc=directory,dc=nh
ldap_user_search_base = ou=People,dc=directory,dc=nh
ldap_group_search_base = ou=Groups,dc=directory,dc=nh
ldap_tls_reqcert = never
ldap_pwd_policy = shadow
cache_credentials = True
default_shell = /usr/libexec/openssh/sftp-server


[nss]
filter_users = ldapservice

reconnection_retries = 10

How to fix this problem? Help please!

Andy_Wismer · May 22, 2020, 12:21pm

@Vladimir_Pobedinskiy

Hi & Welcome to the NethServer community!

If you have a problem, it would help, if you give a bit more details about your system and what you’re running it on. Are backups available?

From your post above, all I can see is RHEL 7.7 to RHEL 7.8…

Now, I don’t assume you’re talking about RHEL (Red Hat Enterprise Linux) - you would probably be at the Red-Hat Forum…

So I assume you mean the recent update from NethServer 7.7x to NethServer 7.8.
How did you do the update, by hand or using one of the available web interfaces (Old Dashboard or newer Cockpit)?

My 2 cents
Andy

Vladimir_Pobedinskiy · May 22, 2020, 12:39pm

Sorry. I search solution in internet and found same problem in red hat page and c&p from there (https://access.redhat.com/solutions/4992301).
Our mail server has been reboot while updating. Boot files has been corrupted and I reinstall it.
When system is loading it now is
core verion is 3.10.0-1127.8.2.el7.x86_64
OS: NethServer release 7.8.2003 (final)
I am update from web interface (cockpit).
Backups not present (we have 2TB of mail files).
Now I have 24 hour to restore access to mail.

Can I dismiss from using LDAP?
Can I setup LDAP server on another machine and add same users to it and configure failed server to use another LDAP?
Can I reinstall system without lost mail?

Andy_Wismer · May 22, 2020, 12:47pm

Hi Vladimir

From your profile you’re close to the more well known Vladivostok - and VERY far from where I am, in Switzerland…

24h restore “should” be possible! (Heads up motivation!)

First question: Where are the 2 TB Mails now?
On the reinstalled server?

Is this a native installation, or are you using virtualization in some form? (Proxmox, Hyper-V, VMWare, XEN…)

Andy

Vladimir_Pobedinskiy · May 22, 2020, 12:50pm

We have 2TB mail on failed server.
It native (dedicated computer in office).

Andy_Wismer · May 22, 2020, 12:53pm

OK, so the mails are on your older, now not booting server…
But the files / folders are still there, just the server won’t boot now…

You now have a newly installed “rescue” server standing more or less next to the old server, in the same network.

Your NethServer is NOT your firewall, that is another box.

Are these assumptions correct?

Vladimir_Pobedinskiy · May 22, 2020, 1:02pm

It running. Work fine but I cannot autorize on it by mail client (web or standalone) - sssd service corruped.
I’m reinstall it but it not work.

Other info in first post.
I have accces to console to execute commands. I have access to files on this server. But I have not access to mail using mail client.

Andy_Wismer · May 22, 2020, 1:06pm

Ah, OK.

You reinstalled NethServer OVER the not working server, is this correct?

Your Mails are on that very server, but you can’t access them AND your LDAP Account Provider is not working.

PS: As this is an english forum, it would greatly help if you could log in to your NethServer in english, just to make screenshots anyone can read…
If it were just verbaltim cryllic text, I could copy / paste into Google translate…
But that doesn’t work with images!

dnutan · May 22, 2020, 1:08pm

Have you checked if there are unfinished yum transactions?

Andy_Wismer · May 22, 2020, 1:11pm

@dnutan

There ARE version mismatches, like here:

So I assume the server had been halfway updated, then an unexpected reboot…
Database updated, library not…

My 2 cents
Andy

Vladimir_Pobedinskiy · May 22, 2020, 1:12pm

It work over 6 month. Today it rebooted while updating. I recover boot files (on Boot partion) but SSSD service corrupted. I reinstall this service but no results.

I finish all transations using YUM (yum-complete-transaction). System is Up-to-date.

[root@**** ~]# yum-complete-transaction
Loaded modules: fastestmirror, langpacks, nethserver_events
Loading mirror speeds from cached hostfile
 * ce-base: mirror.mi.incx.net
 * ce-extras: mirror.mi.incx.net
 * ce-sclo-rh: mirror.mi.incx.net
 * ce-sclo-sclo: mirror.mi.incx.net
 * ce-updates: mirror.mi.incx.net
 * epel: ftp-stud.hs-esslingen.de
 * nethforge: nethserver.interlin.nl
 * nethserver-base: nethserver.interlin.nl
 * nethserver-updates: nethserver.interlin.nl
No unfinished transactions left.

Andy_Wismer · May 22, 2020, 1:15pm

Maybe removing the LDAP Account Provider, and then reinstalling that would work (ie set the db and files right).
File shares are “public” when using LDAP as a provider. (seems so from what I see above…)

Andy

Vladimir_Pobedinskiy · May 22, 2020, 1:19pm

What could be the losses resulting from this operation? Is it safe for mail files?

Andy_Wismer · May 22, 2020, 1:23pm

Files are actually kept, same goes if you were using AD.

Mail folders are NOT touched! At least not with this option removing your Account Provider and reinstalling it.

Still, as you have NO backup, it might be a good idea to hook up a spare disk to that server and copy the whole mail folder over, just as a point in time backup (!).

Rule One in IT:
Better a backup too many, than one too little (Meaning NO backup!)…

Andy

Vladimir_Pobedinskiy · May 22, 2020, 1:35pm

yum reinstall sssd-common

Loading moduled: changelog, fastestmirror, langpacks, nethserver_events
Loading mirror speeds from cached hostfile
 * ce-base: repo.uk.bigstepcloud.com
 * ce-extras: repo.uk.bigstepcloud.com
 * ce-sclo-rh: repo.uk.bigstepcloud.com
 * ce-sclo-sclo: repo.uk.bigstepcloud.com
 * ce-updates: repo.uk.bigstepcloud.com
 * epel: epel.mirrors.arminco.com
 * nethforge: nethserver.interlin.nl
 * nethserver-base: nethserver.interlin.nl
 * nethserver-updates: nethserver.interlin.nl
Resolution deps
--> Checing scenario
---> Package sssd-common.x86_64 0:1.16.4-21.el7_7.3 marked for update
--> Handle dependence: sssd-common = 1.16.4-21.el7_7.3 from package: sssd-ipa-1.16.4-21.el7_7.3.x86_64
--> Handle dependence: sssd-common = 1.16.4-21.el7_7.3 from package:: sssd-proxy-1.16.4-21.el7_7.3.x86_64
--> Handle dependence: sssd-common = 1.16.4-21.el7_7.3 from package:: sssd-ldap-1.16.4-21.el7_7.3.x86_64
--> Handle dependence: sssd-common = 1.16.4-21.el7_7.3 from package:: sssd-common-pac-1.16.4-21.el7_7.3.x86_64
--> Handle dependence: sssd-common = 1.16.4-21.el7_7.3 from package:: sssd-krb5-common-1.16.4-21.el7_7.3.x86_64
--> Handle dependence: sssd-common = 1.16.4-21.el7_7.3 from package:: sssd-ad-1.16.4-21.el7_7.3.x86_64
--> Handle dependence: sssd-common = 1.16.4-21.el7_7.3 from package:: sssd-krb5-1.16.4-21.el7_7.3.x86_64
---> Package sssd-common.x86_64 0:1.16.4-37.el7_8.3 marked as update
---> Package sssd-common.x86_64 0:1.16.4-37.el7_8.3 marked for removing
--> Check deps completed
Error: Package: sssd-common-pac-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
            Need: sssd-common = 1.16.4-21.el7_7.3
            Remove: sssd-common-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
                sssd-common = 1.16.4-21.el7_7.3
            Update: sssd-common-1.16.4-37.el7_8.3.x86_64 (ce-updates)
                sssd-common = 1.16.4-37.el7_8.3
            Available: sssd-common-1.16.4-37.el7.x86_64 (ce-base)
                sssd-common = 1.16.4-37.el7
Error: Package: sssd-ldap-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
            Need: sssd-common = 1.16.4-21.el7_7.3
            Removing: sssd-common-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
                sssd-common = 1.16.4-21.el7_7.3
            Update: sssd-common-1.16.4-37.el7_8.3.x86_64 (ce-updates)
                sssd-common = 1.16.4-37.el7_8.3
            Available: sssd-common-1.16.4-37.el7.x86_64 (ce-base)
                sssd-common = 1.16.4-37.el7
Error: Package: sssd-ipa-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
            Need: sssd-common = 1.16.4-21.el7_7.3
            Removing: sssd-common-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
                sssd-common = 1.16.4-21.el7_7.3
            Update: sssd-common-1.16.4-37.el7_8.3.x86_64 (ce-updates)
                sssd-common = 1.16.4-37.el7_8.3
            Available: sssd-common-1.16.4-37.el7.x86_64 (ce-base)
                sssd-common = 1.16.4-37.el7
Error: Package: sssd-krb5-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
            Need: sssd-common = 1.16.4-21.el7_7.3
            Removing: sssd-common-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
                sssd-common = 1.16.4-21.el7_7.3
            Update: sssd-common-1.16.4-37.el7_8.3.x86_64 (ce-updates)
                sssd-common = 1.16.4-37.el7_8.3
            Available: sssd-common-1.16.4-37.el7.x86_64 (ce-base)
                sssd-common = 1.16.4-37.el7
Error: Package: sssd-proxy-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
            Need: sssd-common = 1.16.4-21.el7_7.3
            Removing: sssd-common-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
                sssd-common = 1.16.4-21.el7_7.3
            Update: sssd-common-1.16.4-37.el7_8.3.x86_64 (ce-updates)
                sssd-common = 1.16.4-37.el7_8.3
            Available: sssd-common-1.16.4-37.el7.x86_64 (ce-base)
                sssd-common = 1.16.4-37.el7
Error: Package: sssd-ad-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
            Need: sssd-common = 1.16.4-21.el7_7.3
            Removing: sssd-common-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
                sssd-common = 1.16.4-21.el7_7.3
            Update: sssd-common-1.16.4-37.el7_8.3.x86_64 (ce-updates)
                sssd-common = 1.16.4-37.el7_8.3
            Available: sssd-common-1.16.4-37.el7.x86_64 (ce-base)
                sssd-common = 1.16.4-37.el7
Error: Package: sssd-krb5-common-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
            Need: sssd-common = 1.16.4-21.el7_7.3
            Removing: sssd-common-1.16.4-21.el7_7.3.x86_64 (@ce-updates)
                sssd-common = 1.16.4-21.el7_7.3
            Update: sssd-common-1.16.4-37.el7_8.3.x86_64 (ce-updates)
                sssd-common = 1.16.4-37.el7_8.3
            Available: sssd-common-1.16.4-37.el7.x86_64 (ce-base)
                sssd-common = 1.16.4-37.el7
you can try --skip-broken to avoid this problem

How to fix it?

Andy_Wismer · May 22, 2020, 1:45pm

@mrmarkuz

Hi Markus

Maybe you’ve got a tip here, I recall reading you battle with version conflicts recently…

Thx
Andy

Vladimir_Pobedinskiy · May 22, 2020, 2:11pm

Now I need concrete instructions.

Andy_Wismer · May 22, 2020, 2:18pm

@giacomo

Any tips here?

Thx

giacomo · May 22, 2020, 2:22pm

You could try to reinstall only the corrupted packages, if any.

First you need to update everything, I do not know why you have problems, probably some broken transaction.
Blind shot:

yum --disablerepo=* --enablerepo=nethserver-updates,nethserver-base,base,updates update

I just fixed a machine with the same problem:

rm -f /usr/lib64/ldb/modules/ldb/paged_results.so
systemctl restart sssd

Andy_Wismer · May 22, 2020, 2:31pm

@giacomo

This guy’s server rebooted (no reason given) during upgrade 7.7x to 7.8…

Sh*t happens!