A Testimonial to a large NS7 to NS8 Migration

Andy_Wismer · November 24, 2024, 6:54pm

Hi all!

Two weeks ago I had to migrate a clients NS7 setup including the Proxmox Hypervisor to newer hardware - and at the same time migrating from NS7 to NS8. Firewalling was done all the time using a Unifi UDM-Pro. (BTW, an excellent choice for firewalls IMHO!).

Background:

The old NS7 was an emergency new installation of a failed earlier setup of NS7, installed on Proxmox by someone I did not know. This person was not available anymore.
The Proxmox was VERY badly setup.

VMs allocated two Passthru Real SATA Disks - and mirrored by NS7, instead of using a mirrored disk on Proxmox. More than twice the ressources used. Each disk was individually backuped with Proxmox, using more than twice the needed space and time.

The same mistake was made with the main ERP system, running on Windows Server, also mirrored by VM OS…

Not enough BAD mistakes…

The main system Disks was a RAID0 - with all VMs having a very small OS disk placed there. And one disk was damaged beyond repair… (VMs there running, but no backup possible of that partition! Always full crash!).

The local Backup Disks were at least mirrored by Proxmox.

The worst thing:

PBS installed on top of Proxmox - using the defective system disk!

Both Backup Systems (PBS and VZDumpBackup) rely on the same set of System Disks, which are defect beyond repair.

My repair (actually new installation on existing hardware and disks!) 2 years ago was tough, but the system has been runing solid since.

Proxmox Hypervisor

The hardware for the earlier Proxmox was essentially a “Gamer-PC”. While initially good enough looking CPU and RAM, the IO was really bad, the whole system was kinda frozen for any backups.

New Proxmox:

This is NOT new hardware, but so called refurbished systems, often used by large companies for 3 years (Tax reasons) then replaced and sold.

This box has completly new disk systems, a RAIDed SAS 2 TB system for Proxmox System and VMs.

A ZFS in RAID10 using 4x 20 TB Enterprise Class spinners, for mass storage.

Normal Load for 10 VMs: ca 1 % !!!
Load during Backups: 9% !!!

IOdelay during Backups 1-2%…

An issue is Swap, even with 128 GB RAM, 80% used, swap sometimes still overfills.
Next set of SSDs will include more swap.

The Migration preprations:

The NS7 had three major uses:

AD, File Server and Nextcloud, all in all about 1 TB of data.

As usual, I prepped a suitable VM for NS8, here, as is my standard, a Debian 12. This was installed according to my earlier HowTo of Debian 12 on Proxmox
( A Debian 12 based new Install of NS8, running on a low powered Proxmox 8.22 (Odroid H3+) )
For this use case, NS8 was allocated 24 of 40 CPU cores, and 16 GB RAM.
The Debian VM also has a 16 GB swap partition.

DNS for the new NS8 and old NS7 were checked and adapted as needed.
IPs are all static, local internal DNS used.

The actual Migration:

Starting up the Migration Assistant on NS7 went without issues.

Migrating Nextcloud went surprisingly smooth:

then

Nice!

Now for the AD & File-Server:

To cut a long story short, this migration did not present any issues during migration.

Post Migration:

DNS corrections in DNS and in AD DNS were quick to resolve using RSAT Tools on an admin-PC.
To connect a NAS, requiring the old NetBIOS name of the NS7 AD was easily solved, as the old NethServer GUI (AKA Server-Manager, using Port 980 still showed this information, screenshotted BEFORE the migration.)

AD was working extremly well, not surprising.

But still missing caveats, and this after Milestone 3 is BAD (!):

Still no WSDD, using the correct Name will allow connections, just the AD name is very random.

The actual Domain Controller Name is created randomly, as it seems, and is NOT displayed anywhere in NS8 GUI.

MS RSAT tools or LAM, if installed, will show this randomly created (and never displayed) name.
But WHY NOT in the GUI during creation, And why is the admin not asked if a name as DC1 would be preferred?

OK, this is a migration from NS7, but even NS7 never showed this name. NS7 accepted its Hostname for the AD, which NS8 will not!

As this name is required for Windows Network Drives (IP is not usable for MS-Office documents…) - it MUST be shown. NethServer 8 should NOT be a system only for Cracks!

This is the randomly created “Hostname” for the DC:
NSDC-KGAN-938A1

As it’s not shown anywhere, without knowing this, connecting correctly is only working via IP.

→ The use of randomly created Hostnames (Which actually even require DNS entries!) is per se pure BullS*it!
For something as the primary AD and FileServer, this is a real NO-GO !!!

I really do not understand why dozens of unneeded / unfinished Applications are being pushed, but the one really important thing most SME users need - AD and File Server - is treated so poorly…

Another MAJOR Issue I do not understand is why Apps like RoundCube have a one click working LE SSL, but critical Stuff like the actual Mailserver does not have this (yet).

Neither does AD, but it didn’t have that in NS7 either, yet this was easily added in permanently in NS7 with a few lines for e-smith. Not so in NS8.

But still, I’m at least happy I can report a flawless migration from a “large” setup - without any issues.
So far, client’s very happy.

Post installation issues like using an OpenMediaVault VM as second File-Storage, including joining that VM box to NS8 AD did pose some challenges (Triggering eg. research on the Hostname of the AD issue…).

The old Proxmox PVE is now equipped with new disks (finally!) and is running as a somewhat overpowered Proxmox Backup Server PBS. This is working excellently.

A full backup of all 10 VMs (After a reboot from Proxmox PVE, when incrementals are not usable) takes around 4 hours.
Incremental backup of all 10 VMs takes only about 5 Minutes!

My Conclusions for the Devs:

Note:
I’m aware that here, in this case, the random hostname for the file server is probably a residual issue from NS7, which also never showed that name, but in NS7 it was nt needed, as NS7’s hostname would work. This is not the case for NS8, as NS8 is a cluster manager, the File Server is a Container, and has it’s own Hostname.
Yet even in a new Install on NS8 with AD - the hostname is given, eg DC1 - without any information or showing that name! Not Good at all!

→ Networks should be planned, not by chance!
Nethesis is an Italian Company, not something like the German Fereral Railways…
(According to management, trains there run by chance…)

Give Samba / File Server the needed treatment.
WSDD is almost a must here, especially due to the random File Server Host Name!
The Random Hostname is an absolute No-Go! This must be visible and settable.
Include (soon) the option for a second file server (on a second node, for example).
Also soon include AD replication (on a second node), but I expect this will take longer…
Include LE & SSL support for both AD and Mail servers!

I will include a HowTo to add in AD support to a current OpenMediaVault NAS - working for hardware and VMs (both in use and tested!). This is to join any AD, but specifically a NS8 based AD. OpenMediaVault does not include AD connectivity or support out of the box.
Sometime the next 2 weeks…

My 2 cents
Andy

yummiweb · November 27, 2024, 1:07pm

Dear Andy,

Thank you for this detailed description - and warning!

After all my tests (new installation but no migration) I would not have expected that. It would have caused me considerable problems.

I agree with your comments to almost 100% and wonder that your observations have apparently noticed anyone else? Or has nobody reported that yet? Or have they all just moved away disappointed?

And why is that not documented? Because it is identical “bad” like the NethServer 7? And why are eliminated functions actually not clearly documented, e.g. in a table with comparison to the previous Nethserver 7. With the reference “no longer, it does not come again” or “is still missing, is still being developed”.

Imagine the migration is carried out by people with limited knowledge - those people who have also waited the previous Nethserver 7 and as is also intended in the operating concept. If it crashes and these people request external help in their need - migration to Neth8 quickly becomes a migration to something else (or in the cloud). These people would be lost forever.

I admire the work and energy that the DEV team and everyone else put into the new development, just like I already admire the confidence of the team in advance of keeping the scheduled schedule.

Unfortunately, I have to say that your experience - even if it contains positive aspects - scares me a little. What will there be if someone comes with a more complex setup? The possibilities of Nethserver 7 were diverse and those of the Neth8 are too. Has anyone done that in all their complexity?

You tend to transfer the many positive experiences with Nethserver 7 PR advance to the new version - but why? Is that justified? I trust the developers, but who knows my posts knows that I think certain decisions are very limiting.

For my part, I have prepared the migration (s) in such a way that I have moved almost all services into independent machines (no Neth8). What remains are the AD, the files server, the mail server and sogo. So basically everything that should remain in the case of user management - and at one point. For me, the question of the square and other integrated drives remained. From the “rest” (actually the essentials "I hoped that there are no guns.

Even if it is not so pleasant, but I consider my own section to be necessary for “Problems with Migration to Neth8”. Or have I only overlooked them?

mrmarkuz · November 27, 2024, 11:18pm

First, thanks for the detailed feedback. In general I agree that there are some things missing like second file server or wsdd.

I want to clarify some of the points:

Maybe it’s a bug but there should be DNS records for the old hostname.

From GitHub - NethServer/ns8-samba: NS8 Samba configuration

If file-server role is migrated, the host name is retained as a NetBIOS alias and a DNS CNAME record, pointing to the DC host name.

You can define it when creating the User domain.
As said after migration the old name should be usable.

The Mailserver cert is created when saving the mailserver app settings in the web UI so it’s the same as with Roundcube.

It’s already there, see also User domains — NS8 documentation
But without SysVol Replication as it’s not supported by Samba AFAIK.

@dnutan provided a nice comparison:

Andy_Wismer · November 28, 2024, 2:53am

Which (The old NetBIOS Hostname for the DC) was never shown in NS7, neither in the old Server-Manager AKA NethGUI nor in the newer Cockpit. Cockpit never showed the old NetBIOS AD name either…

As a long time user, I was always under the assumption it was the same as the NethServer7.

Yes, and NS8 is running samba DNS - if LAM works out of the box, fine. It did in my case, but I had all needed DNS entries already on the Unifi UDM (Firewall, DNS and DHCP, all very usable!).
And I already had a Win10 PC ready with RSAT, already in the old NS7 as AD member, that worked too.
But where will a newbie see any of this?
They will look in AD settings, nothing is visible there…

In Roundcube I still see the option to add or remove LE / SSL. And also an option to set the ssl dns name actually used.

But I do not see ANY options for the actual Mail-Server!

AFAIK, Roundcube can run on a different node as Mailserver, so if Mail IS using the same LE/SSL as Roundcube, this can pose issues if a APP gets migrated…

Again, where does a newbie see or is informed that mail uses the same LE/SSL as Roundcube?
And what about situations where NO Webmail is required or wanted?
Mail does not get a LE/SSL cert?

For large systems, running on a Proxmox Cluster, the lack of File-Server options can make backups extremly long, as the file-volume can not be split up on hosts.

For the moment, I’m forced to use an OpenMediaVault as an AD member, just so I can split up the huge File-Server…
Not really ideal, but workable.

Sorry for being the devils advocate, but that is also the purpose of such a Testemonial, to find out what’s missing, lacking - both from an experts view and from a newbie’s viewpoint.
I do try to include options, what is set, but also what does the system inform the user…

My 2 glowing pieces of coal
Andy

mrmarkuz · November 28, 2024, 9:25am

The NETBIOS Name is shown in the Fileserver application in NS7:

So next to the NetBIOS name there should also be a CNAME for the old NS7 hostname pointing to the Samba hostname. So one should be able to just use the old NS7 hostname but I didn’t test, just found it in the docs.

As far as that is possible it should be the same but as a long time user you know that Samba was a nsdc container in NS7 and now is a podman container in NS8 and NS8 has a completely new design so the assumption of NS7 and NS8 being the same is wrong.

You are talking about a migration from NS7 which usually isn’t done by a newbie. For a new installation the newbie can define everything so he should know.
But also a newbie should be able to install LAM (not needed for common AD work) from Software Center or RSAT tools on a Windows client which is the easiest way to manage Samba/AD.

For sure the AD information in the cluster-admin could be improved to for example show the hostname.

Maybe I didn’t understand correctly, do you have a proposal what should be changed to be good for newbies?

The mailserver really needs a cert nowadays so a LE cert is automatically obtained which is possible in NS8 because working www and DNS is mandatory. So it should just work for the newbie. The hostname for Mail is set in the settings of Mail (like in Roundcube):

Sorry, I didn’t formulate well. It’s not the same cert. Roundcube and Mail use different LE certs, this is an improvement in NS8.
I meant that the way to click save in the settings to get a LE cert is the same for both.

I use xigmanas as AD member, works out of the box without going to CLI for joining AD.

Absolutely no problem, it’s always a good way to discuss things from different view points.

Andy_Wismer · November 28, 2024, 9:47am

Yes, but only if installed AFTER Cockpit onwards. NS7 setup with NethGUI were stuck with the then NS7 Hostname as NetBIOS Name. And as you recall, changing the AD wasn’t an option then.

Most of my clients were set up during NethGUI times (!), only a few later.

Well, I DO see a lot of users trying to migrate their NS7 to NS8 on the Forum - and a lot do NOT have basic understanding of Windows Networking (If it can be called net-working! )
You are forgetting that NS7 has a big Home-User base or whatever one would call that.
Non professionals, enthusiasts, fans, gamers and they come with the whole bandwidth of Know-How or Know-Not…

And none of them like the fact that NS7 is EOL, most would like to migrate or have tried.

We need to make things easier on this segment of our users IMHO…

Just recently, Capote and Stephdl Tried a whole day or two to NO success! (LAM)
(It’s on the forum)

But mail still does not have this option:

And - I do have a client with working mail, but SSL certs for clients like Outlook / TB is NOT working since the beginning…

It IS working for Webmail, Nextcloud and others!
But NOT for IMAP / SMTP (587, 465) We do not use POP3.

So how would the mail be working with SSL for clients (Internal & External)?

My 2 cents
Andy

mrmarkuz · November 28, 2024, 10:13am

I know but I couldn’t reproduce and your LAM seems to work too. If one user has one problem with one app it doesn’t mean automatically that the app doesn’t work at all.

Yes, because it’s not needed to not use a cert in Mail. A cert is mandatory so it’s obtained automatically so why should there be an option?

I can’t reproduce.
This seems to be just an issue on your server because if that won’t work people would complain in the forum.

Thunderbird just works here using SSL/TLS (even sending via SMTP), even the autoconfig works (mailing also works for K9 or kmail and IIRC I tested Outlook too):

I think we need to look deeper into this issue. Did you already open a support thread?
Maybe it helps to remove the mail http route (it needs to be done manually on CLI in traefik) and save the settings in the Mail app to recreate it?

Here’s a thread about renewing the dovecot cert, it should be similar for postfix (SMTP).

As long as the LE cert is valid and clients use the correct mail hostname and the DNS returns a right IP (no matter if internal or external) it should just work.
As soon as the LE cert is obtained it is valid for the hostname (no matter which IP).

EDIT:

No, the NetBIOS name is shown in NethGUI too and IIRC it was possible to set it at creation of the Account Provider in NS7 but the default was ad.NETBIOSNAME.tld.

…to be continued…

Andy_Wismer · November 28, 2024, 10:54am

Then users actually SEE it’s done - and as an option to change / renew.

There are still plenty of users in the forum trying to set up mail with fake domains and no idea about http / DNS challenges for LE…

My 2 cents
Andy

michelandre · November 28, 2024, 2:29pm

Salut All,

This is how I am migrating from NethServer-7.9.

As recommended by @capote and @Andy_Wismer, I installed an UCG Ultra for the firewall ($149 USD); it looks like a small cigar box, quite small.

With the DNS server inside the UCG, DNS records are pointing all the domain names to the NethServer IP address.
This way, the EOL NethServer can run for quite longer and still be protected and secure.

I installed a Proxmox VE server, a Debian-12 minimum server and finally ISPconfig running under the Debian server.

In ISPconfig, I created 2 domains for testing.

The first domain WordPress web site I migrated was micronator-101.com:

Domain:
Inside ISPconfig, I created a new site for this domain.
Certificate;
I created a Let’s Encrypt cert using acme.sh
WordPress:
I scp the root directory of the domain from NethServer to the ISPconfig newly created web site.
I ajusted the owner:group.
I ajusted the DB access from: define('DB_HOST', '127.0.0.1:3312'); to ``define(‘DB_HOST’, ‘localhost’);`
I ajusted the path of the Wordfence application config file.
DB:
I scp the DB dump from NethServer to the Debian /tmp directory.
I created a new DB on the Debian host with the same user:password:priviledges as the original DB and imported the BD from /tmp.
DNS:
In the UCG Ultra, I changed the DNS records to point to the Debian IP.
E-mail:
Since NethServer and ISPconfig both use postfix with MAILDIR, after creating a user in ISPconfig, i just scp the mail directory of the user from NethServer to ISPconfig.
I deleted the dovecot.index and voilà!
No problem with Webmail or Thunderbird.

From the internet, the access to micronator-101.com is to the one running on ISPconfig.

Both domains are running at the same time, just in case…
On the LAN LOCAL, you can access the one choosen by the main Nameserver setting of the station or make an entry in the hosts file of the station…

It was even easier to “clone” the DokuWiki web site as it doesn’t use a DB.

When NethSecurity will be more mature and stable, I might try it…

In a few days, all the documentations will be on my web site.

Just a sure and simple way to migrate,

Michel-André

Madhatter · November 29, 2024, 11:02am

Hi All,

I’m the the client of which the above migration has been done.

A couple of years ago I started a post on this very forum because we had serious issues with our Proxmox system after moving it to a new location. Andy reached out and offered to help out.

I won’t go in any details, I’m no expert in the matter. But Andy managed to recover all data and turn the system around in a stable setup. Since then Andy has been our go to person to keep the system reliable and provide advice on what upgrades can be done to improve. The system had been running solid for some time, but as the company grew realized the system was at its limit. As described in the original post, the system was really never setup as a professional machine.

So I got to work sourcing the hardware, and Andy got to work preparing the migration. Once the hardware was installed it really only took about a day to migrate, and users experienced almost no inconvenience while migrating.

Of course we did not only migrate NS7 to NS8, but also the hardware, migrate all vm’s to new Proxmox, networking via Unifi, DNS and proxy optimizations, PBS, …

All in all great experience, and for sure I can recomend Andy to others. We’ve had IT companies come over in the past trying to sell us hardware, services and maintenance contracts that would cost us a small fortune. I’m glad we went the other way, and have a high performance system and high performing support!

Best regards,

Bart Yzewijn

yummiweb · November 29, 2024, 2:41pm

Dear Markus

[quote=“mrmarkuz, post:3, topic:24942”]
@dnutan provided a nice comparison:

Thank you very much, that is a very nice overview, but I would have liked it to be accessible directly without intensive forum searches or forum queries - e.g. in various Nethserver 8 announcements or the milestone descriptions.

Is this overview kept up to date? I thought that email connectors were now included in NETH8. I was not aware that there was no longer an SMB print server, for example, will it stay that way? Unfortunately, that would change a lot…

I think both - being up to date and being easy to find - of such an overview are extremely important, so that no false ideas about NETH8 are created based on the Nethserver7 experience. And also so that you have an up-to-date status of what is still to come and what you have to say goodbye to.

For new users, the following always applies: read up on the functions, what is not there may never be there. However, switchers (and testers) were always informed that a lot of things are still in progress. But what exactly is “still” in the works, what has been “discarded” and what is possibly still “up for discussion” (in which one could get involved)?

Andy_Wismer · November 29, 2024, 2:50pm

For those of us familiar with a Hypervisor like Proxmox and VMs / Containers, spinning up eg a Debian LXC, connect it to AD and installing CUPS is a matter of minutes.

And yes, @dnutan has put in a lot of effort and kept most, if not all, very up to date, AFAIK.

My 2 cents
Andy

danb35 · November 29, 2024, 3:04pm

To play the devil’s advocate, even granting that it’s as easy as you say to do this, the more^[1] that has to be farmed out to another system, the less valuable Nethserver is. The counterpoint to that, of course, is that there are lots of other things that NS8 does that NS7 doesn’t, so it’s a bit of a balancing act.

and particularly the more that Nethserver used to do, that I use, and that it doesn’t do any more ↩︎

yummiweb · November 29, 2024, 3:04pm

The mail server hostname is included in the email settings, but there is no option to choose whether it should be a self-signed certificate or one generated via LE.

And even under “Settings > TLS certificates” it is not clear what type of certificate it is. There are certificates listed there, but it is not clear who the issuer is and other details such as the validity period etc. are not visible.

The user can only guess because he may still remember where he ticked the box for “Use LE certificate” and “retrieved” is written behind the certificate, but such an estimation method is very impractical for various reasons.

For some services the certificate can still be verified via a browser, but this becomes extremely impractical with “Mail”.

And yes, Mail would also like a trustworthy certificate for IMAP and POP, at least I am very sure of that with Apple Mail.

In my personal case, this “lack of overview” is extremely impractical because my NETH8 is not the first in the chain of LE Acme clients. Before that comes an “OpenWRT” Acme, whose NGINX forwards the corresponding certificate requests to the respective servers in the backend (e.g. NETH8 and web server). For the installation of the NETH8, I sent all requests directly to this, but since then the “OpenWRT Acme” has been doing it. Whether these are reliably forwarded is no longer so easy to check under NETH8. Under Nethserver7 (that’s where I first handled it this way) I can check whether the certificate was issued successfully and, if necessary, request a new certificate to be issued manually - via CLI and GUI. Under NETH8, an update is not offered in the GUI and I don’t know of it under CLI.

yummiweb · November 29, 2024, 3:09pm

To play the devil’s advocate, even granting that it’s as easy as you say to do this, the more that has to be farmed out to another system, the less valuable Nethserver is. The counterpoint to that, of course, is that there are lots of other things that NS8 does that NS7 doesn’t, so it’s a bit of a balancing act.

However, NETH8 was not only communicated as a completely independent new development, but also explicitly as an upgrade, as a continuation for Nethserver 7 users - which is what the migration tool should also contribute to.

Andy_Wismer · November 29, 2024, 3:09pm

I fully agree with this.

My suggestion is also more intended as motivation to not stop the main project of migration by a relatively small obstacle. In other words: a Stopgap or Workaround.

Once experience with containers / docker is available, it should (yes, still conjunctive 2 form) be possible to make one’s own CUPs container…

As with any balancing act: you never know if it works for you, until you try it out!
But never forget that that also entails a falling flat on the face, if worse comes to worse.
Backups are still your friend!

My 2 pieces of glowing coal
Andy

yummiweb · November 29, 2024, 3:18pm

There are alternatives for everything, for individual components or for the entire Nethserver (no offense intended). But if we preferred those, we wouldn’t be here, would we?

And if a print server option can be easily implemented via a standalone VM or an OS container, shouldn’t that be the perfect task for a “NETH 8 container”?

mrmarkuz · November 29, 2024, 3:22pm

There’s already a feature request for CUPS:

Andy_Wismer · November 29, 2024, 3:23pm

I also fully agree with this.

But it’s not yet available in green nor in any other color or version…
Until then, it’s either twiddling thumbs or spinning your own.

The main part is the well working AD for me. But that’s what I mainly use NS8 for.
File-Server could be better (WSDD…) but it works. NextCloud? Also works rock solid.

Those here long enough also remember, NS7 wasn’t as polished when it came out either. But it quickly became the polished workhorse a lot of us loved. Let NS8 mature, like any good wine.

My 2 cents
Andy

alefattorini · November 29, 2024, 3:34pm

Thanks Bart for your post.
Andy is one of our mainstays, our Swiss watch of support
I’m happy to read your words on Andy and NethServer.