System updates fail

NethServer Version: 7.6.1810

For some time now updates have failed. I have not worried about it too much since otherwise everything seems to be fine.
The result is the same whether I try ‘update all’ or individual updates. It goes through init > resolve RPM dependencies > download and then throws an error: check API for more info: system packages /update.
The log entry:

pkgactionTue Sep 03 2019 09:56:24 GMT-0700 (Pacific Daylight Time)

[YumTestTransactionError] Test Transaction Errors: 
package glib2-2.56.1-4.el7_6.x86_64 is already installed 
package iproute-4.11.0-14.el7_6.2.x86_64 is already installed 
package openssl-libs-1:1.0.2k-16.el7_6.1.x86_64 is already installed 
package gnutls-3.3.29-9.el7_6.x86_64 is already installed 
package nethserver-subscription-3.4.1-1.ns7.noarch is already installed 
package libsss_autofs-1.16.2-13.el7_6.8.x86_64 is already installed 
package pango-1.42.4-2.el7_6.x86_64 is already installed 
package libsss_nss_idmap-1.16.2-13.el7_6.8.x86_64 is already installed 
package python2-future-0.16.0-15.20181019gitbee0f3b.el7.noarch is already installed 
package nss-sysinit-3.36.0-7.1.el7_6.x86_64 is already installed 
package python-zope-component-1:4.1.0-5.el7.noarch is already installed 
package shadow-utils-2:4.1.5.1-25.el7_6.1.x86_64 is already installed 
package duplicity-0.7.19-1.el7.x86_64 is already installed 
package nethserver-dnsmasq-1.7.0-1.ns7.noarch is already installed 
package python-2.7.5-80.el7_6.x86_64 is already installed 
package mokutil-15-2.el7.centos.x86_64 is already installed 
package nethserver-base-3.7.3-1.ns7.noarch is already installed 
package rspamd-1.9.1-1.x86_64 is already installed 
package freetype-2.8-12.el7_6.1.x86_64 is already installed 
package libsss_idmap-1.16.2-13.el7_6.8.x86_64 is already installed 
package rsync-3.1.2-6.el7_6.1.x86_64 is already installed 
package nss-util-3.36.0-1.1.el7_6.x86_64 is already installed 
package nss-pem-1.0.3-5.el7_6.1.x86_64 is already installed 
package glibc-2.17-260.el7_6.6.x86_64 is already installed 
package sope49-xml-4.9-20190311_2200.ns7.x86_64 is already installed 
package python2-mock-1.0.1-10.el7.noarch is already installed 
package libsss_certmap-1.16.2-13.el7_6.8.x86_64 is already installed 
package libipa_hbac-1.16.2-13.el7_6.8.x86_64 is already installed 
package krb5-libs-1.15.1-37.el7_6.x86_64 is already installed 
package nss-3.36.0-7.1.el7_6.x86_64 is already installed 
package openldap-2.4.44-21.el7_6.x86_64 is already installed 
package sope49-gdl1-contentstore-4.0.7-1.ns7.x86_64 is already installed 
package perl-4:5.16.3-294.el7_6.x86_64 is already installed 
package libuuid-2.23.2-59.el7_6.1.x86_64 is already installed 
package clamav-unofficial-sigs-5.6.2-7.el7.noarch is already installed 
package util-linux-2.23.2-59.el7_6.1.x86_64 is already installed 
package libmount-2.23.2-59.el7_6.1.x86_64 is already installed 
package webtop5-webdav-0.1.2-1.ns7.noarch is already installed 
package efivar-libs-36-11.el7_6.1.x86_64 is already installed 
package python-libs-2.7.5-80.el7_6.x86_64 is already installed 
package dbus-1:1.10.24-13.el7_6.x86_64 is already installed 
package sope49-gdl1-4.9-20190311_2200.ns7.x86_64 is already installed 
package glibc-common-2.17-260.el7_6.6.x86_64 is already installed 
package openldap-clients-2.4.44-21.el7_6.x86_64 is already installed 
package sope49-ldap-4.9-20190311_2200.ns7.x86_64 is already installed 
package NetworkManager-libnm-1:1.12.0-10.el7_6.x86_64 is already installed 
package tomcat-jsp-2.2-api-0:7.0.76-9.el7_6.noarch is already installed 
package perl-libs-4:5.16.3-294.el7_6.x86_64 is already installed 
package mesa-libgbm-18.0.5-4.el7_6.x86_64 is already installed 
package sope49-appserver-4.9-20190311_2200.ns7.x86_64 is already installed 
package openssl-1:1.0.2k-16.el7_6.1.x86_64 is already installed 
package policycoreutils-2.5-29.el7_6.1.x86_64 is already installed 
package pulledpork-0.7.3-5.ns7.noarch is already installed 
package dbus-libs-1:1.10.24-13.el7_6.x86_64 is already installed 
package libsmartcols-2.23.2-59.el7_6.1.x86_64 is already installed 
package nethserver-lib-2.2.11-1.ns7.noarch is already installed 
package openvpn-2.4.7-1.el7.x86_64 is already installed 
package libkadm5-1.15.1-37.el7_6.x86_64 is already installed 
package libgcc-4.8.5-36.el7_6.2.x86_64 is already installed 
package NetworkManager-1:1.12.0-10.el7_6.x86_64 is already installed 
package webtop5-1.2.15-1.ns7.noarch is already installed
PRIORITY 3
SYSLOG_FACILITY 1
SYSLOG_IDENTIFIER pkgaction
SYSLOG_PID 13969
_AUDIT_LOGINUID 0
_AUDIT_SESSION 12771
_BOOT_ID 8bf8360ecf314681a51ee53376ad7533
_CAP_EFFECTIVE 1fffffffff
_CMDLINE /usr/bin/python /usr/libexec/nethserver/pkgaction --json --update *
_COMM pkgaction
_EXE /usr/bin/python2.7
_GID 0
_HOSTNAME juliet.deltaecho.lan
_MACHINE_ID 5b5afa945f734a45b20f7bd43c9b8082
_PID 13969
_SOURCE_REALTIME_TIMESTAMP 1567529784418359
_SYSTEMD_CGROUP /user.slice/user-0.slice/session-12771.scope
_SYSTEMD_OWNER_UID 0
_SYSTEMD_SESSION 12771
_SYSTEMD_SLICE user-0.slice
_SYSTEMD_UNIT session-12771.scope
_TRANSPORT syslog
_UID 0
__CURSOR s=3e75db0f2b9b4e26a3a68c3f91bcb540;i=3755b;b=8bf8360ecf314681a51ee53376ad7533;m=a0d4998c68;t=591a8f7f9a3d4;x=1d1f0282dfcc7a0
__MONOTONIC_TIMESTAMP 690761600104
__REALTIME_TIMESTAMP 1567529784419284

Please, run a full backup before proceed.
Would you please try to connect to SSH and run this commands

yum clean all

and

yum update

posting the results?

yum clean all finishes with no comments.
yum update fails at the transaction check with the following error, and I am totally lost::
–> Finished Dependency Resolution
–> Running transaction check
—> Package kernel.x86_64 0:3.10.0-862.6.3.el7 will be erased
–> Finished Dependency Resolution
Error: Multilib version problems found. This often means that the root
cause is something else and multilib version checking is just
pointing out that there is a problem. Eg.:

     1. You have an upgrade for libgcc which is missing some
        dependency that another package requires. Yum is trying to
        solve this by installing an older version of libgcc of the
        different architecture. If you exclude the bad architecture
        yum will tell you what the root cause is (which package
        requires what). You can try redoing the upgrade with
        --exclude libgcc.otherarch ... this should give you an error
        message showing the root cause of the problem.

     2. You have multiple architectures of libgcc installed, but
        yum can only see an upgrade for one of those architectures.
        If you don't want/need both architectures anymore then you
        can remove the one with the missing update and everything
        will work.

     3. You have duplicate versions of libgcc installed already.
        You can use "yum check" to get yum show these errors.

   ...you can also use --setopt=protected_multilib=false to remove
   this checking, however this is almost never the correct thing to
   do as something else is very likely to go wrong (often causing
   much more problems).

   Protected multilib versions: libgcc-4.8.5-36.el7_6.2.i686 != libgcc-4.8.5-36.el7.x86_64

Error: Protected multilib versions: glibc-2.17-260.el7_6.6.i686 != glibc-2.17-260.el7.x86_64
Error: Protected multilib versions: libstdc+±4.8.5-36.el7_6.2.i686 != libstdc+±4.8.5-36.el7.x86_64
[root@juliet ~]#

So this lead to the question: why server has installed libgcc, glibc and libstdc i686 instead of x86_64?
What software did you installed on the nethserver more than modules?
Maybe yum deplist libgcc and so on could help to find it.

Consider also to use tag “code” (ctrl+shift+c) into posts for improve reading of output.

1 Like

I have only nethserver modules installed.

[root@juliet ~]# yum deplist libgcc
Loaded plugins: auto-update-debuginfo, changelog, fastestmirror,
: nethserver_events
Loading mirror speeds from cached hostfile

Ok, i posted the “wrong” command.
I would like to know which package used the libraries as dependancy, but i searched the opposite (which dependencies the libraries has).
I am not skilled or experienced enough to know if it’s safe to remove libgcc, glibc and libstdc with i686 (even i am not sure i know how to do that…)

@support_team
Can somebody help here?

@vbn You can check which packages require libgcc with following command:

rpm -q --whatrequires libgcc

2 Likes

[root@juliet ~]# rpm -q --whatrequires libgcc
no package requires libgcc

Does that mean that it is safe to remove libgcc? And if so how? (I wonder how it got in there in the first place).

Me too, especially the 32 bit version (libgcc.i686 4.8.5-36.el7_6.2)

I think it is save to say you can remove the package, you may remove the 32bit version first.

yum remove libgcc.i686

than

yum clean all
yum update
1 Like

Removal of libgcc will remove also glibc and libstdc?

You probably right, lets take one step at a time.
In the end we may have to remove nss-softokn-freebl-3.36.0-5.el7_5.i686 too…

EDIT

@vbn on second tough you may want to remove all i686 (=32 bit packages)
yum remove *.i686

just out off curiosity - or if you are not sure if it is oke ,then press N(o) and post output - which packages are / will be removed by this?

[root@juliet ~]# yum remove *.i686
Loaded plugins: auto-update-debuginfo, changelog, fastestmirror,
              : nethserver_events
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).
--> Running transaction check
---> Package audit-libs.i686 0:2.8.4-4.el7 will be erased
---> Package bzip2-libs.i686 0:1.0.6-13.el7 will be erased
---> Package cracklib.i686 0:2.9.0-11.el7 will be erased
---> Package elfutils-libelf.i686 0:0.172-2.el7 will be erased
---> Package elfutils-libs.i686 0:0.172-2.el7 will be erased
---> Package glibc.i686 0:2.17-260.el7 will be erased
---> Package libattr.i686 0:2.4.46-13.el7 will be erased
---> Package libcap.i686 0:2.22-9.el7 will be erased
---> Package libcap-ng.i686 0:0.7.5-4.el7 will be erased
---> Package libdb.i686 0:5.3.21-24.el7 will be erased
---> Package libgcc.i686 0:4.8.5-36.el7 will be erased
---> Package libgcrypt.i686 0:1.5.3-14.el7 will be erased
---> Package libgpg-error.i686 0:1.12-3.el7 will be erased
---> Package libselinux.i686 0:2.5-14.1.el7 will be erased
---> Package libsepol.i686 0:2.5-10.el7 will be erased
---> Package libstdc++.i686 0:4.8.5-36.el7 will be erased
---> Package lz4.i686 0:1.7.5-2.el7 will be erased
---> Package nss-softokn-freebl.i686 0:3.36.0-5.el7_5 will be erased
---> Package pam.i686 0:1.1.8-22.el7 will be erased
---> Package pcre.i686 0:8.32-17.el7 will be erased
---> Package rdma-core.i686 0:17.2-3.el7 will be erased
---> Package systemd-libs.i686 0:219-62.el7 will be erased
---> Package xz-libs.i686 0:5.2.2-1.el7 will be erased
---> Package zlib.i686 0:1.2.7-18.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                  Arch       Version               Repository      Size
================================================================================
Removing:
 audit-libs               i686       2.8.4-4.el7           @base          244 k
 bzip2-libs               i686       1.0.6-13.el7          @base           76 k
 cracklib                 i686       2.9.0-11.el7          @base          186 k
 elfutils-libelf          i686       0.172-2.el7           @base          899 k
 elfutils-libs            i686       0.172-2.el7           @base          757 k
 glibc                    i686       2.17-260.el7          @base           15 M
 libattr                  i686       2.4.46-13.el7         @base           19 k
 libcap                   i686       2.22-9.el7            @base          100 k
 libcap-ng                i686       0.7.5-4.el7           @base           44 k
 libdb                    i686       5.3.21-24.el7         @base          1.8 M
 libgcc                   i686       4.8.5-36.el7          @base          190 k
 libgcrypt                i686       1.5.3-14.el7          @base          608 k
 libgpg-error             i686       1.12-3.el7            @base          337 k
 libselinux               i686       2.5-14.1.el7          @base          206 k
 libsepol                 i686       2.5-10.el7            @base          690 k
 libstdc++                i686       4.8.5-36.el7          @base          997 k
 lz4                      i686       1.7.5-2.el7           @base          356 k
 nss-softokn-freebl       i686       3.36.0-5.el7_5        @updates       499 k
 pam                      i686       1.1.8-22.el7          @base          2.4 M
 pcre                     i686       8.32-17.el7           @base          1.4 M
 rdma-core                i686       17.2-3.el7            @base          101 k
 systemd-libs             i686       219-62.el7            @base          1.3 M
 xz-libs                  i686       5.2.2-1.el7           @base          245 k
 zlib                     i686       1.2.7-18.el7          @base          180 k

Transaction Summary
================================================================================
Remove  24 Packages

Installed size: 28 M

Would removing any of these get me into trouble?

this is very strange, you prettymuch have a 32 bit tool-chain for building (compiling) 32 applications…

phoe,
I say No… 99.9% sure… not 100%

sure we can solve the problems if they would happen , I do not know how critical your server is. ie can we work on problems if they would pop-up

lets check if the 64 bit counterparts are installed,

some critical ones (ouput of my setup):

 # rpm -q pam
pam-1.1.8-22.el7.x86_64

# rpm -q systemd-libs
systemd-libs-219-62.el7_6.9.x86_64

Yaay - I now have a fully updated system and no apparent issues. Both pam and systemd-libs are present at listed versions.
I want to thank you guys for helping me out. Great community. And who knows I might have learned a thing or two from this.
Thanks again.

1 Like

Thanks for the update @vbn. What you did to unlock the status?

I removed all the .i686 files, ran yum clean all and then yum update. After those 330 updates there was one new update, which I cleared in the same manner. BTW I have current backups so I was not too worried about removing the 32 bit files, now I just hope they stay away.
Thanks again.

3 Likes

Nice to here, that it works for you, you you mark the topic as solved please?