Ultimate ns7 software updates origin policy

davidep · January 9, 2019, 4:13pm

I’d like to remove the “Unlocked” software updates origin. Any NethServer instance should be subject to just one repository policy/configuration!

We discussed a lot of about the software origin policy feature and the need for more updates stability when 7.5 was released ¹. Then we found a bug during the release of 7.6 that (partially) invalidated that effort ².

I was working on SCLo repositories version lock when I started to think about cutting down the use cases.

why removing “unlocked” ? Because the maintenance of the two options is complex and bug/error prone. Instead of stability, we got bugs! Because it simplifies the configuration (“do not make me think…”)
why keeping “unlocked” ? Because it relies on the public CentOS repository infrastructure, so one can be sure to always access the best CentOS mirror available out there.

So to go safely with the “unlocked” removal we have to rely on our list of monitored centos mirrors, extend it and keep it up-to-date ³. I think this is a sustainable effort for our community!

What do you think?

davidep · January 10, 2019, 3:27pm

So no reply = everybody agrees?

Another unsuccessful use case of the current “locked” setup: today I found this message for root on one of my VMs with the locked software origin:

Failed to check for updates with the following error message: 
Failed to build transaction: nethserver-subscription-3.3.1-1.ns7.noarch requires jq

Overnight yum-cron has EPEL disabled. As the new nethserver-subscription package requires a package from EPEL yum-cron fails. The error is reproducible also from the Software Center.

The solution here could be:

always enable EPEL (like we did until 7.5), OR
host dependencies from EPEL on nethserver-updates (like we did on ns6)

Please give me some feedback because it’s not sustainable from my point of view!

davidep · January 10, 2019, 3:35pm

dnutan · January 11, 2019, 1:33am

Cannot think much about it right now, no clear answer but will try to comment on it…
First and foremost, updates breaking things can affect every OS and software application.

Locked updates shall prevent things like getting caught by surprise by changes breaking backward compatibility between RHEL 7 versions, or alike, in cases where the sysadmin couldn’t test the updates beforehand.

By locking updates to minor version release:

no automatic OS “upgrade” is done
sysadmin has to be notified of availability of new version
upgrade requires manual intervention

Some user cases:
Apart of missing security updates and patched vulnerabilities, what happens when a server is more than one version behind the current release? Will the notification show the latest available version and will it be able to upgrade to it in an easy way?

@fausp faced the same problem.

From past comments on linked threads, won’t work/sustain unless the process is fully automated.

If EPEL isn’t enabled, we might get failed (dependencies) updates. If enabled, some functionality might be broken by an update. On missing dependencies, an option could be a separate update process with EPEL enabled only for the affected package(s), but if feasible it introduces unwanted complexity (check-update; if failed find cause; if missing dependencies, get affected package and check-update with EPEL enabled).

Would it help to collect pros and cons or a list of issues updates can bring?

m.traeumner · January 11, 2019, 9:39am

I think this a very important point. I think a lot of admins are very glad about not doing everything manualy.

What do you think about an information and a checkbox for an upgrade to a newer version by installing updates like this:

Attention!!! A new release version is out (recommended or not recommended), would you like to install it also? Yes No

davidep · January 11, 2019, 11:23am

Thank you very much for your help!

Yes this is the goal of the current “locked” origin. By removing the “unlocked” option we’d get for the CentOS project repositories

less configuration complexity
less code complexity
more stability during manual minor upgrades (i.e. 7.5 -> 7.6)
more confident automated (yum-cron) updates

…however we have still the other upstream project, EPEL which doesn’t follow the same minor upgrade lifecycle. Here the choice is more difficult:

if we disable updates from official EPEL, and host the required subset of its packages in nethserver-updates, we need to regularly pull their updates, to avoid the same limitations we had in ns6 (never-updated dependencies)
if we keep the official EPEL enabled, we still have some update risks ¹. We could mitigate them with a better QA process that checks out epel-testing regularly.
Maybe I’m wrong, but I feel that EPEL updates are less critical than distro minor upgrades

Yes, and for this reason we could start by checking epel-testing regularly. It becomes a bit of effort for our teams and the whole community …but are we here to help each other?

If this little improvement is still not enough, we can implement the required automation and host EPEL packages in our repos.

We have to understand that for each automated task there is someone who actually is doing his job (paid or not).

I think we already have a minor distro upgrade button in the Software Center page when “locked” origin is set.

https://fedoraproject.org/wiki/EPEL#Can_I_rely_on_these_packages.3F

pike · January 12, 2019, 1:59am

In my suggestion the “locked” setting had to be the default, following the concept that…

I download 7.5.
I update 7.5
If i want to upgrade to whatever i need to activate the upgrade procedure.

Anyway, locked setting failed. At least twice, as far as i can remember (7.5 and 7.6).
This should say “ok, that was just a joke”?

Also: remove unlocked updates could be nice, but biggest question now is: will remove the chance to live upgrade the system?

(cosmetic question: do the word “final” keep having sense into NethServer version?)

stephdl · January 12, 2019, 7:58am

Only my thought here, we want stability, for my concerns I update my servers a really long time after a minor upgrade of centos, my servers are still not upgraded to 7.6. But at the end, some Sysadmins wants the last updates all the time, and in some case we will break their systems if they upgrade a minor version.

Epel is considered also like a system breaker too, even if the risks are lower, for example in SME Server a script check the rpm version from EPEL, integrated in the repository and download to testing the new version of EPEL. Developers check from time to time the new versions and test them, before to release to stable.

Of course after said that, new versions are not deployed soon, and @davidep has stated, automatic or manual tasks are still done by a human in the last steps…

From my experience of NethServer, the risks come from centos minor version upgrade, from EPEL, we will have maybe one service broken, and many time to fix it, so I would like to tend to a locked system, with epel enabled (it saves my developer’s life)

giacomo · January 15, 2019, 1:14pm

The system will be upgraded using the UI, just like is happening now

The “final” is more related to the ISO than to the OS itself.

flatspin · January 15, 2019, 5:20pm

This is confusing! In my understanding it is: Alpha1, Alpha2, Beta1, Beta2, RC1, RC2, Final and this is OS-related only, because it is =>

@davidep
Locked or unlocked that is here the question. (sorry, coudn’t withstand that )
Locked gave also problems like unlocked. So where is the advantage of locked?
Today I removed locked to updated a VM to 7.6, because the machine didn’t show any available modules, installed modules or updates in softwarecenter. After update all was o.k. again.

As a example I’d like to remember Proxmox. You can have stable subscription repos or the community repo, with untested stuff, which can cause problems. That’s it. Pay or have the risc. Fair deal for me.
That’s why I have a subscrition on Proxmox and on NS for my productionsystem.

To have subscription, locked and unlocked is a 3-way-strategy which is a lot more complex to handle.
and IMO complexity is the mother or trouble.

Just my thoughts.

giacomo · January 17, 2019, 8:26am

Nailed it!

That’s exactly the same word I said in private to @davidep

But bear in mind that providing a stable product for free is the best way to gather feedback, help and grow the community . So the proposal is to remove the lock/unlock feature to simplify the development but also make sure that all machines access the correct repositories.

We (as developer) will save time (and bugs) removing the feature, and reinvest some of the saved time to maintain the mirrorlist part and make sure the machines correctly access the repos.

michelandre · January 17, 2019, 2:44pm

“Pay or have the risc”

I don’t think that this is in line with the GPL and OpenSource but rather with “Extortion and Protection Rackets”.

A friend who just wishes to help!

michelandre · January 17, 2019, 3:20pm

m.traeumner Michael Träumner NethServer Ambassador

I think a lot of admins are very glad about not doing everything manualy.

Those newbie admins might be surprised…
Ref: New Centos 7.6 update.

Due to a bug in the yum-cron/NsReleaseLock feature those who enabled the automated updates found a bad surprise today: their systems were updated to CentOS 7.6 by the daily cron job.

That is the reason every OS recommends to verify updates in a test environment before applying them.

stephdl Stéphane de Labrusse Dev Team

Epel is considered also like a system breaker too, even if the risks are lower, for example in SME Server a script check the rpm version from EPEL, integrated in the repository and download to testing the new version of EPEL. Developers check from time to time the new versions and test them, before to release to stable.

I am pretty sure SME has much less developers than NethServer…

Michel-André

pike · January 17, 2019, 3:56pm

Why?
Using Sid puts you on the bleeding edge. Or using Gentoo.

Debian, Centos, Slackware and OpenSUSE are quite known for their stability and reliability, but that’s the goal of the project, which can’t be good for any.
I want the lock policy, just for avoid that security updates are missing on my boxes but without breaking services or manual update.

Reviewing and delay updates costs time and people, subscription helps both development and update review. I won’t use it, but i won’t blame Nethesis for choose this strategy.

flatspin · January 18, 2019, 10:36am

Hard words my friend and IMO completely wrong. An opensource project needs to have a fundamental and reliable business model to survive. A lot of OSS-projects have such business models, most of them are selling the support and/or stable repos. Again I want to mention proxmox. It is licensed under AGPL and it’s business modell is exactly what I said above (subscription repo + support). So no violation with GPL or FOSS and defenitely no extorsion.
FOSS/Opensource doesn’t mean that it’s forbidden to earn money with it or that it has to be free of charge. @medworthy has given some well-founded comments on that here

But I admit that I choose my words in a provocantively way.

PS: And I apologize that I have forgotten to give a like to this article of medworthy until today. But now it’s done

davidep · January 24, 2019, 10:29am

Thank you all for contributing to this discussion!

What I want to do now is

close the issue SCLo repositories version lock as “wontfix”
implement and release the ultimate software updates policy during 7.6

The new policy will overwrite existing settings. Despite current admin’s choice about locked/unlocked SOP the update will turn the system behavior to “ultimate”.

What is “ultimate”?

Receives updates from CentOS 7.6.1810 mirrors, always available from Software Center
When CentOS 7.7.XXXX will be out nothing changes until a system upgrade is started from the Software Center
EPEL 7 updates are always available from the Software Center page, directly from EPEL mirrors.
We must run the QA process with epel-testing enabled, to find problems early with EPEL 7 updates

Note: the “ultimate” policy will not apply to NethServer Subscriptions plans and Enterprise versions.

Edit: tracked here

davidep · February 18, 2019, 4:10pm

Changes to mirrorlist.nethserver.org have been deployed. It must continue to work seamlessly with the current clients configuration.
Packages are ready for QA and are available from the testing repository /cc @quality_team

davidep · February 22, 2019, 3:14pm

The new policy has been released. This new #feature brings a more predictable behavior of system updates.

Released RPMs are:

nethserver-release-7-13.ns7.noarch.rpm
nethserver-base-3.7.0-1.ns7.noarch.rpm
nethserver-subscription-3.4.0-1.ns7.noarch.rpm
nethserver-subscription-ui-3.4.0-1.ns7.noarch.rpm

Recap:

The software origin policy radio button was removed from UI
The NethServer.repo YUM config file is now a template: as such it overwrites any local configuration
The YUM configuration forcibly points to version 7.6.1810 for both NethServer and CentOS repositories
EPEL repository is always accessed as generic version “7”
When NS 7.7 is released the system upgrade must be started manually, from the Software Center page: automated/cron tasks were removed
The YUM configuration split between update and install use case was removed

The above changes does not affect Subscription and Enterprise versions.

davidep · April 12, 2019, 7:23am

I’m running a daily cronjob to see what is coming up from epel-testing. An advanced AI decides when it’s worth notifying the admin of newly added packages

Create an executable script /etc/cron.daily/10epel_testing_monitor with the following contents:

#!/bin/bash
epel_cache=/var/lib/misc/epel_testing_monitor.cache
epel_testing=$(/usr/bin/yum --setopt=exit_on_lock=1 -q -d 0 -e 0 -y --enablerepo=epel-testing check-update)
if [[ -n $epel_testing ]] && [[ "${epel_testing}" != $(<$epel_cache) ]]; then
    cat - >$epel_cache <<<"$epel_testing"
    echo -e "Upcoming releases from epel-testing:"
    echo "$epel_testing"
fi

The script runs on a real-world production server! It does not change any installed package, just send an email to root when it finds a new package release available from epel-testing. That means I have to check it!

flatspin · April 12, 2019, 7:38am

Like it !