We discussed a lot of about the software origin policy feature and the need for more updates stability when 7.5 was released ¹. Then we found a bug during the release of 7.6 that (partially) invalidated that effort ².
why removing “unlocked” ? Because the maintenance of the two options is complex and bug/error prone. Instead of stability, we got bugs! Because it simplifies the configuration (“do not make me think…”)
why keeping “unlocked” ? Because it relies on the public CentOS repository infrastructure, so one can be sure to always access the best CentOS mirror available out there.
So to go safely with the “unlocked” removal we have to rely on our list of monitored centos mirrors, extend it and keep it up-to-date ³. I think this is a sustainable effort for our community!
Another unsuccessful use case of the current “locked” setup: today I found this message for root on one of my VMs with the locked software origin:
Failed to check for updates with the following error message:
Failed to build transaction: nethserver-subscription-3.3.1-1.ns7.noarch requires jq
Overnight yum-cron has EPEL disabled. As the new nethserver-subscription package requires a package from EPEL yum-cron fails. The error is reproducible also from the Software Center.
The solution here could be:
always enable EPEL (like we did until 7.5), OR
host dependencies from EPEL on nethserver-updates (like we did on ns6)
Please give me some feedback because it’s not sustainable from my point of view!
Cannot think much about it right now, no clear answer but will try to comment on it…
First and foremost, updates breaking things can affect every OS and software application.
sysadmin has to be notified of availability of new version
upgrade requires manual intervention
Some user cases:
Apart of missing security updates and patched vulnerabilities, what happens when a server is more than one version behind the current release? Will the notification show the latest available version and will it be able to upgrade to it in an easy way?
From past comments on linked threads, won’t work/sustain unless the process is fully automated.
If EPEL isn’t enabled, we might get failed (dependencies) updates. If enabled, some functionality might be broken by an update. On missing dependencies, an option could be a separate update process with EPEL enabled only for the affected package(s), but if feasible it introduces unwanted complexity (check-update; if failed find cause; if missing dependencies, get affected package and check-update with EPEL enabled).
Would it help to collect pros and cons or a list of issues updates can bring?
Yes this is the goal of the current “locked” origin. By removing the “unlocked” option we’d get for the CentOS project repositories
less configuration complexity
less code complexity
more stability during manual minor upgrades (i.e. 7.5 → 7.6)
more confident automated (yum-cron) updates
…however we have still the other upstream project, EPEL which doesn’t follow the same minor upgrade lifecycle. Here the choice is more difficult:
if we disable updates from official EPEL, and host the required subset of its packages in nethserver-updates, we need to regularly pull their updates, to avoid the same limitations we had in ns6 (never-updated dependencies)
if we keep the official EPEL enabled, we still have some update risks ¹. We could mitigate them with a better QA process that checks out epel-testing regularly.
Maybe I’m wrong, but I feel that EPEL updates are less critical than distro minor upgrades
Yes, and for this reason we could start by checking epel-testing regularly. It becomes a bit of effort for our teams and the whole community …but are we here to help each other?
If this little improvement is still not enough, we can implement the required automation and host EPEL packages in our repos.
We have to understand that for each automated task there is someone who actually is doing his job (paid or not).
I think we already have a minor distro upgrade button in the Software Center page when “locked” origin is set.
Only my thought here, we want stability, for my concerns I update my servers a really long time after a minor upgrade of centos, my servers are still not upgraded to 7.6. But at the end, some Sysadmins wants the last updates all the time, and in some case we will break their systems if they upgrade a minor version.
Epel is considered also like a system breaker too, even if the risks are lower, for example in SME Server a script check the rpm version from EPEL, integrated in the repository and download to testing the new version of EPEL. Developers check from time to time the new versions and test them, before to release to stable.
Of course after said that, new versions are not deployed soon, and @davidep has stated, automatic or manual tasks are still done by a human in the last steps…
From my experience of NethServer, the risks come from centos minor version upgrade, from EPEL, we will have maybe one service broken, and many time to fix it, so I would like to tend to a locked system, with epel enabled (it saves my developer’s life)
This is confusing! In my understanding it is: Alpha1, Alpha2, Beta1, Beta2, RC1, RC2, Final and this is OS-related only, because it is =>
@davidep
Locked or unlocked that is here the question. (sorry, coudn’t withstand that )
Locked gave also problems like unlocked. So where is the advantage of locked?
Today I removed locked to updated a VM to 7.6, because the machine didn’t show any available modules, installed modules or updates in softwarecenter. After update all was o.k. again.
As a example I’d like to remember Proxmox. You can have stable subscription repos or the community repo, with untested stuff, which can cause problems. That’s it. Pay or have the risc. Fair deal for me.
That’s why I have a subscrition on Proxmox and on NS for my productionsystem.
To have subscription, locked and unlocked is a 3-way-strategy which is a lot more complex to handle.
and IMO complexity is the mother or trouble.
That’s exactly the same word I said in private to @davidep
But bear in mind that providing a stable product for free is the best way to gather feedback, help and grow the community . So the proposal is to remove the lock/unlock feature to simplify the development but also make sure that all machines access the correct repositories.
We (as developer) will save time (and bugs) removing the feature, and reinvest some of the saved time to maintain the mirrorlist part and make sure the machines correctly access the repos.
Due to a bug in the yum-cron/NsReleaseLock feature those who enabled the automated updates found a bad surprise today: their systems were updated to CentOS 7.6 by the daily cron job.
That is the reason every OS recommends to verify updates in a test environment before applying them.
stephdl Stéphane de Labrusse Dev Team
Epel is considered also like a system breaker too, even if the risks are lower, for example in SME Server a script check the rpm version from EPEL, integrated in the repository and download to testing the new version of EPEL. Developers check from time to time the new versions and test them, before to release to stable.
I am pretty sure SME has much less developers than NethServer…
Why?
Using Sid puts you on the bleeding edge. Or using Gentoo.
Debian, Centos, Slackware and OpenSUSE are quite known for their stability and reliability, but that’s the goal of the project, which can’t be good for any.
I want the lock policy, just for avoid that security updates are missing on my boxes but without breaking services or manual update.
Reviewing and delay updates costs time and people, subscription helps both development and update review. I won’t use it, but i won’t blame Nethesis for choose this strategy.
Hard words my friend and IMO completely wrong. An opensource project needs to have a fundamental and reliable business model to survive. A lot of OSS-projects have such business models, most of them are selling the support and/or stable repos. Again I want to mention proxmox. It is licensed under AGPL and it’s business modell is exactly what I said above (subscription repo + support). So no violation with GPL or FOSS and defenitely no extorsion.
FOSS/Opensource doesn’t mean that it’s forbidden to earn money with it or that it has to be free of charge. @medworthy has given some well-founded comments on that here
But I admit that I choose my words in a provocantively way.
PS: And I apologize that I have forgotten to give a like to this article of medworthy until today. But now it’s done
implement and release the ultimate software updates policy during 7.6
The new policy will overwrite existing settings. Despite current admin’s choice about locked/unlocked SOP the update will turn the system behavior to “ultimate”.
What is “ultimate”?
Receives updates from CentOS 7.6.1810 mirrors, always available from Software Center
When CentOS 7.7.XXXX will be out nothing changes until a system upgrade is started from the Software Center
EPEL 7 updates are always available from the Software Center page, directly from EPEL mirrors.
We must run the QA process with epel-testing enabled, to find problems early with EPEL 7 updates
Note: the “ultimate” policy will not apply to NethServer Subscriptions plans and Enterprise versions.
I’m running a daily cronjob to see what is coming up from epel-testing. An advanced AI decides when it’s worth notifying the admin of newly added packages
Create an executable script /etc/cron.daily/10epel_testing_monitor with the following contents:
#!/bin/bash
epel_cache=/var/lib/misc/epel_testing_monitor.cache
epel_testing=$(/usr/bin/yum --setopt=exit_on_lock=1 -q -d 0 -e 0 -y --enablerepo=epel-testing check-update)
if [[ -n $epel_testing ]] && [[ "${epel_testing}" != $(<$epel_cache) ]]; then
cat - >$epel_cache <<<"$epel_testing"
echo -e "Upcoming releases from epel-testing:"
echo "$epel_testing"
fi
The script runs on a real-world production server! It does not change any installed package, just send an email to root when it finds a new package release available from epel-testing. That means I have to check it!
always enable EPEL (like we did until 7.5), OR
host dependencies from EPEL on nethserver-updates (like we did on ns6)
)
