To much kernels installed, but config gives a limit of 5

[dnutan (Marc)]

@davidep, should we investigate pkgaction and the python API for yum?

[davidep (Davide Principi)]

Yes please!

It seems there are some clues leading to the UI. I looked at the python code of pkgaction. The way the configuration is initialized could differ from the yum cli command.

[dnutan (Marc)]

An observation and an idea/theory, hoping to bring some clues:

Observation
When using yum, if a new kernel is available and the number of installed kernels matches the number of kernels to keep, an old kernel is removed:

=========================================================================================================
 Package                           Arch       Version                       Repository              Size
=========================================================================================================
Installing:
 kernel                            x86_64     3.10.0-693.17.1.el7           updates                 43 M

Updating:
 dhclient                          x86_64     12:4.2.5-58.el7.centos.1      updates                282 k
 kernel-tools                      x86_64     3.10.0-693.17.1.el7           updates                5.1 M
 kernel-tools-libs                 x86_64     3.10.0-693.17.1.el7           updates                5.1 M
 kmod                              x86_64     20-15.el7_4.7                 updates                121 k
 kmod-libs                         x86_64     20-15.el7_4.7                 updates                 50 k

Removing:
 kernel                            x86_64     3.10.0-514.21.1.el7           @updates               148 M

Idea
Maybe pkgaction update (or UI) only looks which packages can be updated but it’s unaware of associated removal actions.

[stephdl (Stéphane de Labrusse)]

I cannot reproduce, but I only use the command line…the software panel is not for me

@m.traeumner can you reproduce the bug yet, do you see the kernel number increased over 5 ?
If yes I would be curious if all the kernel are visible in grub ?

For what I read it could be a full nethserver problem, I did not read (much) issues like this…fun.

If this issue could not be solved by a correct fix, maybe we could have a workaround, after the yum transaction, we could check the number of kernel and set it to 5 if it is not the case.

Sure a /boot filled by a huge number of kernel is annoying.

[mark_nl (Mark Verlinde)]

Dug in this a little bit (interest is to ship arm images with a small boot partion )

yum history shows a update from the command line does something different as from the server-manager

command line:

# yum history
Loaded plugins: changelog, fastestmirror, nethserver_events
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    16 |  root <root>             | 2018-08-15 12:32 | I, U, E        |    7

# yum history info 16
Loaded plugins: changelog, fastestmirror, nethserver_events
Config time: 0.022
Yum version: 3.4.3
rpmdb time: 0.000
Transaction ID : 16
Begin time     : Wed Aug 15 12:32:30 2018
Begin rpmdb    : 618:30c6fc96227748c9b609e33795c783bcddfa8ac0
End time       :            12:33:19 2018 (49 seconds)
End rpmdb      : 617:606dc5a76105ca61d555abb449759a9f47a569cc
User           : root <root>
Return-Code    : Success
Command Line   : update
Transaction performed with:
    Installed     rpm-4.11.3-32.el7.x86_64                        @anaconda
    Installed     yum-3.4.3-158.el7.centos.noarch                 @anaconda
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64         @anaconda
    Installed     yum-plugin-fastestmirror-1.1.31-46.el7_5.noarch @updates
Packages Altered:
    Updated centos-release-7-5.1804.1.el7.centos.x86_64  @updates
    Update                 7-5.1804.4.el7.centos.x86_64  @updates
    Erase   kernel-3.10.0-862.el7.x86_64                 @anaconda
    Erase   kernel-3.10.0-862.3.2.el7.x86_64             @updates
    Install kernel-3.10.0-862.11.6.el7.x86_64            @updates
    Updated kernel-tools-3.10.0-862.9.1.el7.x86_64       @updates
    Update               3.10.0-862.11.6.el7.x86_64      @updates
    Updated kernel-tools-libs-3.10.0-862.9.1.el7.x86_64  @updates
    Update                    3.10.0-862.11.6.el7.x86_64 @updates
    Updated python-perf-3.10.0-862.9.1.el7.x86_64        @updates
    Update              3.10.0-862.11.6.el7.x86_64       @updates

from server-manger (GUI):

# yum history
Loaded plugins: changelog, fastestmirror, nethserver_events
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    16 | System <unset>           | 2018-08-15 12:46 | I, U           |    5

# yum history info 16
Loaded plugins: changelog, fastestmirror, nethserver_events
Config time: 0.021
Yum version: 3.4.3
rpmdb time: 0.000
Transaction ID : 16
Begin time     : Wed Aug 15 12:46:50 2018
Begin rpmdb    : 618:30c6fc96227748c9b609e33795c783bcddfa8ac0
End time       :            12:47:39 2018 (49 seconds)
End rpmdb      : 619:d7680163c8cfda4360537725551cdbfa5d3b63ba
User           : System <unset>
Return-Code    : Success
Transaction performed with:
    Installed     rpm-4.11.3-32.el7.x86_64                        @anaconda
    Installed     yum-3.4.3-158.el7.centos.noarch                 @anaconda
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64         @anaconda
    Installed     yum-plugin-fastestmirror-1.1.31-46.el7_5.noarch @updates
Packages Altered:
    Updated centos-release-7-5.1804.1.el7.centos.x86_64  @updates
    Update                 7-5.1804.4.el7.centos.x86_64  @updates
    Install kernel-3.10.0-862.11.6.el7.x86_64            @updates
    Updated kernel-tools-3.10.0-862.9.1.el7.x86_64       @updates
    Update               3.10.0-862.11.6.el7.x86_64      @updates
    Updated kernel-tools-libs-3.10.0-862.9.1.el7.x86_64  @updates
    Update                    3.10.0-862.11.6.el7.x86_64 @updates
    Updated python-perf-3.10.0-862.9.1.el7.x86_64        @updates
    Update              3.10.0-862.11.6.el7.x86_64       @updates

@dnutan

Thoughts and a recap:

• yum and yum-cron are able to remove old kernels.
• server-manger GUI does not remove old kernels due to either pkgaction update not implementing a “packages to remove” action, or some quirks in yum’s python-API version.

Best guess is the yum python-API does something different here, but cant find why/where.

A workaround can be to install the yum-utils and incorporate package-cleanup --oldkernels --count=x somehow in pkgaction update. ideally read x from yum.conf

[mark_nl (Mark Verlinde)]

Its a duplicate topic, continuing there

Again with the hoarding of kernels

I not a software developer most definitely not a python developer nor a yum specialist; so the next can be not accurate or even completely wrong, still i think it valuable to try to describe my findings. Yum update from the command line and yum update from the server-manger (GUI) are completely different beasts. The yum update from the command line populates a transaction-object which includes the whole dependency resolve result. It does this thought cli.YumBaseCli() by building a transactio…

[stephdl (Stéphane de Labrusse)]

Confirmed @giacomo @davidep I can reproduce the bug on one of my server

Normally I do only yum update from cli, this time I did the update by the software-center and I can confirm that my kernel number was 5 before and 6 after the update with the server-manager

[root@NS7 ~]# rpm -q kernel
kernel-3.10.0-693.11.1.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
kernel-3.10.0-693.21.1.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
[root@NS7 ~]# 
[root@NS7 ~]#  Update with the software-center
[root@NS7 ~]# rpm -q kernel
kernel-3.10.0-693.11.1.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
kernel-3.10.0-693.21.1.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
kernel-3.10.0-862.11.6.el7.x86_64

[flatspin (Ralf Jeckel)]

Also can confirm:

before update via softwarecenter 5 kernels installed:

 rpm -q kernel
kernel-3.10.0-862.el7.x86_64
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64

after update via softwarecenter 6 kernels installed:

rpm -q kernel
kernel-3.10.0-862.el7.x86_64
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
kernel-3.10.0-862.11.6.el7.x86_64

When updating via CLI:

oldest kernel is erased.

rpm -q kernel
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
kernel-3.10.0-862.11.6.el7.x86_64

PS: this works ATM only on non-subscription machines, because kernel 3.10.0-862.11.6 isn’t yet in sb-repo.

[m.traeumner (Michael Träumner)]

Sorry for late response (holiday):

@stephdl

@m.traeumner can you reproduce the bug yet

Yes I can.

[root@groupware ~]# rpm -q kernel
kernel-3.10.0-693.21.1.el7.x86_64
kernel-3.10.0-693.5.2.el7.x86_64
kernel-3.10.0-693.11.1.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.11.6.el7.x86_64

@mark_nl

Its a duplicate topic, continuing there

It looks like we continue here, I set a link to this thread at the other one.

[giacomo (Giacomo Sanchietti)]

Ok guys, hold tight: we may have a fix!

Could someone try the RPM from this PR?

pkgaction: honor installonly_limit

Just execute:

yum --enablerepo=nethserver-testing update nethserver-base

Then, go the UI and apply the updates: the oldest kernels should be automatically removed at the end of the process.

EDIT: I’m pretty sure this is a good fix, I just merged the patch. The package is available from testing repo.

[dnutan (Marc)]

It is working.
Task progress shows removal of kernel.
Removal was not logged on yum.log or /var/log/messages.

Details

Before:

# rpm -q kernel
kernel-3.10.0-862.el7.x86_64
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64

# yum -q --assumeno update kernel
=========================================================================================================
 Package             Arch                Version                            Repository              Size
=========================================================================================================
Installing:
 kernel              x86_64              3.10.0-862.11.6.el7                updates                 46 M
Removing:
 kernel              x86_64              3.10.0-862.el7                     @anaconda               62 M

Transaction Summary
=========================================================================================================
Install  1 Package
Remove   1 Package

Exiting on user command

After updating through Software Center:

# rpm -q kernel
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
kernel-3.10.0-862.11.6.el7.x86_64

[giacomo (Giacomo Sanchietti)]

@dnutan

Removal was not logged on yum.log or /var/log/messages.

I already noticed it but I don’t think we need to invoke same special API since all other actions are already logged.

[dnutan (Marc)]

All good. Checked old logs and kernels removed by yum updates (installonly_limit trigger) weren’t logged neither. So it is same behaviour on both sides.

[stephdl (Stéphane de Labrusse)]

same for me, kernel are set to two after the software center update

[giacomo (Giacomo Sanchietti)]

Thank you for the testin!

Closed and released.

/cc @EddieA @fasttech @flatspin

[MrE (Enrique D)]

I have this:

~]# cat /etc/yum.conf|grep limit
installonly_limit=5
~]# ll /boot | grep -c vmli
8

I have 30% used space in boot, so I’m not worry yet:

/dev/sda1 1014M 303M 712M 30% /boot

I need to wait a little more, instead of do this, right?:

~]# yum install yum-utils
and clean up
~]# package-cleanup --oldkernels --count=5

[mark_nl (Mark Verlinde)]

@MrE

~]# yum install yum-utils
and clean up
~]# package-cleanup --oldkernels --count=5

This works right away, but

@MrE

I have 30% used space in boot, so I’m not worry yet:

You can also wait for the next kernel update

[m.traeumner (Michael Träumner)]

@giacomo

Could someone try the RPM from this PR?

Tested also, it works great.

Before:

[root@GroupwareBackup ~]# rpm -q kernel
kernel-3.10.0-693.21.1.el7.x86_64
kernel-3.10.0-693.5.2.el7.x86_64
kernel-3.10.0-693.2.2.el7.x86_64
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64

After

[root@GroupwareBackup ~]# rpm -q kernel
kernel-3.10.0-693.21.1.el7.x86_64
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.11.6.el7.x86_64

[flatspin (Ralf Jeckel)]

Seems to work properly.

Before update via Softwarecenter:

rpm -q kernel
kernel-3.10.0-862.el7.x86_64
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64

after:

rpm -q kernel
kernel-3.10.0-862.3.2.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.6.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
kernel-3.10.0-862.11.6.el7.x86_64

Oldest kernel was removed correctly.