Recently IpFire project annouced a datacenter migration for servers hosting the project.
(if you don’t know IPFire, is a firewall-security centric distribution available in bare metal and virtual environment for managing firewall and connections, with some lacking features but a few interesting addons)
https://blog.ipfire.org/post/public-service-announcement-we-are-moving-our-servers
And… things gone wrong. Well… happens.
https://blog.ipfire.org/post/an-update-on-our-data-center-migration
This part was quite interesting to me…
Part Five - Re-install all the things
How do you repair this? Some machines had a couple of blocks broken which could be repaired with a simple filesystem check. Some others were okay but showed old data. Some others were just scrambled egg.
It would have been dangerous to continue like this. We would find compromised data months or years later and of course we cannot afford that. So I made the decision to re-install all the machines that were broken and restore from the off-site backup.
We also were based on CentOS 7. Something that used to work very reliably for us. Up to that point where it become older and older and older. Right now, you have to add a bunch of third-party repositories to have a recent version of Postfix, Apache, and what not. This became high maintenance and every time updates were installed, something else broke. This is not really what I expected from an “enterprise” distribution and so some time ago we decided that we need to look around for something else. We did some trials with Debian Buster earlier this year which is a totally different experience. Things are more sane, solid and just straight-forward. This is more like what we need.
Their virtual infrastructure was oVirt and GlusterFS based, so a “total RedHat arrangement” from Virtualizer, FileSystem services, guest OS.
Current virtual infastructure seems ProxMox based
And… this lead to some questions to me.
- Did the project starts with CentOS 7.0?
- Did the project followed the updates of the software releases without upgrades?
- The “totally different experience” obtained by Debian Buster was mostly due to the far more recent release of the “new Debian guy”? CentOS 7 was released during July of 2014, Buster was full freezed during March of this year and released in June.
- Were the project aware of all the changes due to five years of Hardware and product upgrades? Sometimes run updates and upgrades on production system could generate a lot of fear and delays…
During upgrades from 7.3 to 7.6 several packages lead to a bit of mess on the NethServer experience. Also there are a bit of “new interesting stuff” like CockPit that are bringing a lot of code and debug for dev team, also for catch-up (and debug) updates by interesting addons like WebTop, MatterMost, NextCloud and not-so-less-interesting SoGo.
Now questions for everyone who want to say something:
- Had you some similar experiences with CentOS 7?
- Was oVirt “a bad idea” from the beginning?
- Is oVirt a “interesting option” in small virtual environments?
- Were people of IpFire only… unprepared for the tasks, in your opinion?
- Maybe using third part repositores was a not so good idea into a public-server environment?
And now for @dev_team:
- Is RedHat/CentOS project giving you a smooth enough path for developing NethServer?
- Have you felt some kind of “what the jazz are they doing” sometimes?
- CentOS 8 seems a viable path for “next” NethServer?