To much kernels installed, but config gives a limit of 5

NethServer Version: 7.4.1708
Module: yum kernel

Hi,
I’ve found an already discussed problem. I’ve to much kernels installed and can’t do an update because the boot-filesystem has not enough space.

Transaction check error:
installing package kernel-3.10.0-693.17.1.el7.x86_64 needs 14MB on the /boot filesystem

The discussion is at:

yum list installed kernel

gives me

kernel.x86_64                     3.10.0-327.el7                           @anaconda
kernel.x86_64                     3.10.0-327.36.1.el7                      @updates
kernel.x86_64                     3.10.0-327.36.2.el7                      @updates
kernel.x86_64                     3.10.0-327.36.3.el7                      @updates
kernel.x86_64                     3.10.0-514.2.2.el7                       @updates
kernel.x86_64                     3.10.0-514.6.1.el7                       @updates
kernel.x86_64                     3.10.0-514.10.2.el7                      @updates
kernel.x86_64                     3.10.0-514.16.1.el7                      @updates
kernel.x86_64                     3.10.0-514.21.1.el7                      @updates
kernel.x86_64                     3.10.0-514.21.2.el7                      @updates
kernel.x86_64                     3.10.0-514.26.1.el7                      @updates
kernel.x86_64                     3.10.0-514.26.2.el7                      @updates
kernel.x86_64                     3.10.0-693.2.2.el7                       @updates
kernel.x86_64                     3.10.0-693.5.2.el7                       @updates
kernel.x86_64                     3.10.0-693.11.1.el7                      @updates

I’ve cleaned it with

package-cleanup --oldkernels --count=2

and it works, but the limit at the config is set to 5 and I’ve 15 installed.

# ================= DO NOT MODIFY THIS FILE =================
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at NethServer official site: https://www.nethserver.org
#
#
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=https://github.com/NethServer/dev
distroverpkg=centos-release
#
# 20proxy - disabled
#

#
# 30groups use compat group mode
#
group_command=compat

#  This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
#  It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d

Has somebody an idea?

I never reproduced the problem. I checked tens of systems and all of them have 5 kernels installed.
Did you always apply updates using the server manager software center button?
The problem could come from the interface command. What do you think @davidep?

1 Like

I did my updates always at the softwarecenter, but this time it didn’t worked, gives me the “clear yum cache” error. After clearing it I tried again, but no success. So I tried it at the commandline and got the “needs 14MB on the /boot filesystem” error.

You didn’t find any difference?

Question: Is installonlypkgs option enabled even if not present on yum.conf ? (I guess it is)

installonly_limit Number of packages listed in installonlypkgs to keep installed at the same time. Note that as of version 3.2.24, yum will now look in the yumdb for a installonly attribute on installed packages. If that attribute is “keep”, then they will never be removed.

installonlypkgs List of package provides that should only ever be installed, never updated. Kernels in particular fall into this category. Defaults to kernel, kernel-bigmem, kernel-enterprise, kernel-smp, kernel-modules, kernel-debug, kernel-unsupported, kernel-source, kernel-devel, kernel-PAE, kernel-PAE-debug.

Note that because these are provides, and not just package names, kernel-devel will also apply to kernel-debug-devel, etc.

1 Like

Quoting myself:

I’ve discovered the problem: that lone server had kernel-debug installed, doubling the count of kernels. So it really has 5 kernels installed.
We’re back to start: I can’t reproduce the problem, all servers have 5 kernels installed.

1 Like
[root@groupware ~]# rpm -qa | grep kernel
kernel-3.10.0-514.16.1.el7.x86_64
kernel-3.10.0-514.6.1.el7.x86_64
kernel-3.10.0-327.36.2.el7.x86_64
kernel-3.10.0-514.21.2.el7.x86_64
kernel-3.10.0-327.el7.x86_64
kernel-tools-libs-3.10.0-693.11.1.el7.x86_64
kernel-3.10.0-514.10.2.el7.x86_64
kernel-3.10.0-514.21.1.el7.x86_64
kernel-headers-3.10.0-693.11.1.el7.x86_64
kernel-tools-3.10.0-693.11.1.el7.x86_64
kernel-3.10.0-693.2.2.el7.x86_64
kernel-3.10.0-327.36.3.el7.x86_64
kernel-3.10.0-693.5.2.el7.x86_64
kernel-3.10.0-327.36.1.el7.x86_64
kernel-3.10.0-514.26.2.el7.x86_64
kernel-3.10.0-514.2.2.el7.x86_64
kernel-3.10.0-514.26.1.el7.x86_64
kernel-3.10.0-693.11.1.el7.x86_64

I think the most are really different kernels. Only the ones with tools and headers are not different ones.

Do these commands match anything?

find /var/lib/yum/yumdb/ | grep installonly
grep -R installonly /var/lib/yum/yumdb/
1 Like

Thanks for answer Davide,
like I wrote at my first post I have done

package-cleanup --oldkernels --count=2

It also deletes the only one which was not an update

kernel.x86_64                     3.10.0-327.el7                           @anaconda

So I had only 2 kernels left, 3 after the last update.

Your commands gives out nothing, but I think after deleting the old ones it is not meaningful.
Perhaps somebody else has more than 5 kernels and can help with the output.

1 Like

This is from a server installed long ago (~1y), and kernel limit seems to be working as expected. All updates were done through command line (not from server-manager).

Installed Packages
kernel.x86_64             3.10.0-514.21.1.el7             @updates
kernel.x86_64             3.10.0-514.26.2.el7             @updates
kernel.x86_64             3.10.0-693.5.2.el7              @updates
kernel.x86_64             3.10.0-693.11.1.el7             @updates
kernel.x86_64             3.10.0-693.11.6.el7             @updates

@davidep commands return empty.

2 Likes

I too update always from CLI (yum update), never from server manager.
Is that a potentially hazardous way of updating?
(no problems with too many kernels either)

1 Like

Software Center uses yum for updating. I update via Software Center and yum directly and don’t have more than 5 kernels.

@Jim What is about you, how many kernels have you at this time.
I ask you, because you are the other one who had such problems.

Nope:

[root@Nethserver ~]# find /var/lib/yum/yumdb/ | grep installonly
[root@Nethserver ~]# grep -R installonly /var/lib/yum/yumdb/
[root@Nethserver ~]# yum list installed kernel
Loaded plugins: changelog, fastestmirror, nethserver_events
Loading mirror speeds from cached hostfile
 * base: repo1.dal.innoscale.net
 * epel: linux.mirrors.es.net
 * extras: mirror.hmc.edu
 * nethforge: mirror.nordest.systems
 * nethserver-base: mirror.nordest.systems
 * nethserver-updates: mirror.nordest.systems
 * updates: centos.mirror.ndchost.com
Installed Packages
kernel.x86_64                   3.10.0-514.el7                         @anaconda
kernel.x86_64                   3.10.0-514.6.2.el7                     @updates
kernel.x86_64                   3.10.0-514.10.2.el7                    @updates
kernel.x86_64                   3.10.0-514.16.1.el7                    @updates
kernel.x86_64                   3.10.0-514.21.1.el7                    @updates
kernel.x86_64                   3.10.0-514.21.2.el7                    @updates
kernel.x86_64                   3.10.0-514.26.1.el7                    @updates
kernel.x86_64                   3.10.0-514.26.2.el7                    @updates
kernel.x86_64                   3.10.0-693.2.2.el7                     @updates
kernel.x86_64                   3.10.0-693.5.2.el7                     @updates
kernel.x86_64                   3.10.0-693.11.1.el7                    @updates
kernel.x86_64                   3.10.0-693.11.6.el7                    @updates
[root@Nethserver ~]#

All updates were done via the UI.

Cheers.

1 Like

@davidep, should we investigate pkgaction and the python API for yum?

2 Likes

Yes please!

It seems there are some clues leading to the UI. I looked at the python code of pkgaction. The way the configuration is initialized could differ from the yum cli command.

1 Like

An observation and an idea/theory, hoping to bring some clues:

Observation
When using yum, if a new kernel is available and the number of installed kernels matches the number of kernels to keep, an old kernel is removed:

=========================================================================================================
 Package                           Arch       Version                       Repository              Size
=========================================================================================================
Installing:
 kernel                            x86_64     3.10.0-693.17.1.el7           updates                 43 M

Updating:
 dhclient                          x86_64     12:4.2.5-58.el7.centos.1      updates                282 k
 kernel-tools                      x86_64     3.10.0-693.17.1.el7           updates                5.1 M
 kernel-tools-libs                 x86_64     3.10.0-693.17.1.el7           updates                5.1 M
 kmod                              x86_64     20-15.el7_4.7                 updates                121 k
 kmod-libs                         x86_64     20-15.el7_4.7                 updates                 50 k

Removing:
 kernel                            x86_64     3.10.0-514.21.1.el7           @updates               148 M

Idea
Maybe pkgaction update (or UI) only looks which packages can be updated but it’s unaware of associated removal actions.

3 Likes

I cannot reproduce, but I only use the command line…the software panel is not for me :slight_smile:

@m.traeumner can you reproduce the bug yet, do you see the kernel number increased over 5 ?
If yes I would be curious if all the kernel are visible in grub ?

For what I read it could be a full nethserver problem, I did not read (much) issues like this…fun.

If this issue could not be solved by a correct fix, maybe we could have a workaround, after the yum transaction, we could check the number of kernel and set it to 5 if it is not the case.

Sure a /boot filled by a huge number of kernel is annoying.

Dug in this a little bit (interest is to ship arm images with a small boot partion :grinning:)

yum history shows a update from the command line does something different as from the server-manager

command line:

# yum history
Loaded plugins: changelog, fastestmirror, nethserver_events
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    16 |  root <root>             | 2018-08-15 12:32 | I, U, E        |    7

# yum history info 16
Loaded plugins: changelog, fastestmirror, nethserver_events
Config time: 0.022
Yum version: 3.4.3
rpmdb time: 0.000
Transaction ID : 16
Begin time     : Wed Aug 15 12:32:30 2018
Begin rpmdb    : 618:30c6fc96227748c9b609e33795c783bcddfa8ac0
End time       :            12:33:19 2018 (49 seconds)
End rpmdb      : 617:606dc5a76105ca61d555abb449759a9f47a569cc
User           : root <root>
Return-Code    : Success
Command Line   : update
Transaction performed with:
    Installed     rpm-4.11.3-32.el7.x86_64                        @anaconda
    Installed     yum-3.4.3-158.el7.centos.noarch                 @anaconda
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64         @anaconda
    Installed     yum-plugin-fastestmirror-1.1.31-46.el7_5.noarch @updates
Packages Altered:
    Updated centos-release-7-5.1804.1.el7.centos.x86_64  @updates
    Update                 7-5.1804.4.el7.centos.x86_64  @updates
    Erase   kernel-3.10.0-862.el7.x86_64                 @anaconda
    Erase   kernel-3.10.0-862.3.2.el7.x86_64             @updates
    Install kernel-3.10.0-862.11.6.el7.x86_64            @updates
    Updated kernel-tools-3.10.0-862.9.1.el7.x86_64       @updates
    Update               3.10.0-862.11.6.el7.x86_64      @updates
    Updated kernel-tools-libs-3.10.0-862.9.1.el7.x86_64  @updates
    Update                    3.10.0-862.11.6.el7.x86_64 @updates
    Updated python-perf-3.10.0-862.9.1.el7.x86_64        @updates
    Update              3.10.0-862.11.6.el7.x86_64       @updates

from server-manger (GUI):

# yum history
Loaded plugins: changelog, fastestmirror, nethserver_events
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    16 | System <unset>           | 2018-08-15 12:46 | I, U           |    5

# yum history info 16
Loaded plugins: changelog, fastestmirror, nethserver_events
Config time: 0.021
Yum version: 3.4.3
rpmdb time: 0.000
Transaction ID : 16
Begin time     : Wed Aug 15 12:46:50 2018
Begin rpmdb    : 618:30c6fc96227748c9b609e33795c783bcddfa8ac0
End time       :            12:47:39 2018 (49 seconds)
End rpmdb      : 619:d7680163c8cfda4360537725551cdbfa5d3b63ba
User           : System <unset>
Return-Code    : Success
Transaction performed with:
    Installed     rpm-4.11.3-32.el7.x86_64                        @anaconda
    Installed     yum-3.4.3-158.el7.centos.noarch                 @anaconda
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64         @anaconda
    Installed     yum-plugin-fastestmirror-1.1.31-46.el7_5.noarch @updates
Packages Altered:
    Updated centos-release-7-5.1804.1.el7.centos.x86_64  @updates
    Update                 7-5.1804.4.el7.centos.x86_64  @updates
    Install kernel-3.10.0-862.11.6.el7.x86_64            @updates
    Updated kernel-tools-3.10.0-862.9.1.el7.x86_64       @updates
    Update               3.10.0-862.11.6.el7.x86_64      @updates
    Updated kernel-tools-libs-3.10.0-862.9.1.el7.x86_64  @updates
    Update                    3.10.0-862.11.6.el7.x86_64 @updates
    Updated python-perf-3.10.0-862.9.1.el7.x86_64        @updates
    Update              3.10.0-862.11.6.el7.x86_64       @updates

Best guess is the yum python-API does something different here, but cant find why/where.

A workaround can be to install the yum-utils and incorporate package-cleanup --oldkernels --count=x somehow in pkgaction update. ideally read x from yum.conf

2 Likes

Its a duplicate topic, continuing there :grinning:

Confirmed @giacomo @davidep I can reproduce the bug on one of my server

Normally I do only yum update from cli, this time I did the update by the software-center and I can confirm that my kernel number was 5 before and 6 after the update with the server-manager

[root@NS7 ~]# rpm -q kernel
kernel-3.10.0-693.11.1.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
kernel-3.10.0-693.21.1.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
[root@NS7 ~]# 
[root@NS7 ~]#  Update with the software-center
[root@NS7 ~]# rpm -q kernel
kernel-3.10.0-693.11.1.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
kernel-3.10.0-693.21.1.el7.x86_64
kernel-3.10.0-862.3.3.el7.x86_64
kernel-3.10.0-862.9.1.el7.x86_64
kernel-3.10.0-862.11.6.el7.x86_64
1 Like