NS7 to NS8 - Less then satisfactory

NethServer Version: NS8
Module: your_module
I hope this thread does not come as grumbling and griping about nethserver. I recently tried to migrate from NS7 to NS8 and wanted to share the negative experience I had. Perhaps it will resonate with others with similar experience or maybe something in my environment was the cause.

I have been using NS7 for a long time as my primary email server. I never deployed anything else in the NS7 along side just email and webtop. WebTop served quite well over the years. The activesync although was not perfect, it worked well for the most part to keep email, calendar and contacts in sync on many of our users android devices.

We have 23 email domains and few 100s email boxes. Email authentication was done using Active Directory. All worked as expected. I decided to give NS8 a try since I no longer wanted to remain on centos. We have moved away from centos into Debian for all linux machines. Deploying new NS8 on top of Debian was the logical choice.

Installation went fine. But soon after final reboot I have noticed high CPU and RAM usage on the NS8. I have virtual environment so NS8 was a virtual machine same as NS7. I duplicated same resources from NS7 to NS8. After I installed mail and webtop, the resource usage usage went up much higher. I had to double the core and memory, still it was using more resources than the fully loaded NS7 with all email domain and email boxes.

Instead of full migration I only migrated one email domain using imapsync. Inclusion of imapsync was a great addition as it made email migration a breeze. Webtop worked without a problem however I could not make the activesync work. I was hoping it would work out of the box. But the sync simply did not work.

I uploaded a 3rd party Sectigo SSL but IMAP and SMTP did not want to use it for some reason. I tried to use signal-event command as I did in the past for NS7, only then I realized NS8 is running in container form inside Debian. That is probably the reason of so high resource requirement. Suffice to say I could not find any directory for NS8 like it was in NS7. Probably containerization was the cause. I tried to find documentation on this but nothing surfaced regarding the new architecture of NS8.

Attemot to create a cluster also proved fruitless. The 2nd NS8 instance simply wont talk to the master node. The cluster admin dashboard was very buggy to say the least. It constantly keep showing me either blank page or gray lines which seemed like placeholders. Repeated refresh of the page did not help much. It loaded completely sporadically when it saw fit.

While trying to figure out what was the best route to implement multiple domain, I installed 2 extra instances of mail app then fund out there are no way to uninstall them through GUI or console. Neth wiki had no mention of how to uninstall app.
I do apologize if it is there and I just missed it.

Although I could connect mobile devices using IMAP and there was no issue for incoming email, outgoing emails had issues. Mobile simply could not send out emails. They would sit in outbox forever.

I could not look through Logs as the log page was very unstable in loading. When it did load, there was not much info.

I use an external Smarthost to relay emails from email server. But the smarthost option is completely non existent and there was no way for me to add it to postfix via console.

As I mentioned earlier, I use Neth purely for stable Email platform. I do not use all other extra bells and whistle. I simply need an email server with activesync that just works. I understand neth8 is still ways off implementing all the features that ns7 had, but I expected the minimal features that are included in the current release of ns8 to work without major issue.

I did move away from Nethserver all together as I cannot take risk parts of Email features not working for so many email boxes. But I am still going to keep an Eye on NS8 and will have a test deployment to check regularly the maturity of NS8.

Is it just me who had all these issues or had it been a common experience?

5 Likes

I think It’s not just your problem…
https://community.nethserver.org/search?q=ns8%20%23bug

1 Like

Which one? Proxmox?

It needs more ressources because of containers but not that extreme. In this case there may be another issue if you just installed webtop and mail…
Do you use fast disks like SSD or NVME?

You could turn on debug log in Webtop app advanced settings to maybe get more information why it doesn’t sync.

There’s a migration tool to migrate from NS7 to NS8, see NethServer 7 migration — NS8 documentation

Documentation is available following the links on the main site: https://www.nethserver.org/

Info NS8: NethServer 8, application platform for Hybrid Cloud – NethServer
Dev manual: Home | NS8 dev manual
Admin manual: NethServer 8 administrator manual — NS8 documentation

Do you have 2 nodes in the LAN? Working DNS is mandatory, the nodes are talking over wireguard VPN.

You can only have 1 working mail server per node because it needs to use the default mail ports, see Mail server — NS8 documentation

To uninstall, you need to choose the installed apps in Software Center and click on instances:

You can uninstall by clicking on the dots menu:

There seems to be another issue, usually the logs page is fast enough (I use it a lot for testing apps)

You could check logs on CLI using journalctl

It’s already in testing repository and should be available soon.
Mail customization is documented here: GitHub - NethServer/ns8-mail: NS8 Mail module with SMTP, IMAP, Spam/Virus filter
Here is a custom config for using a smarthost.

I’m sorry to read that but IMO the only way to make a system stable is to raise issues (as you did, thank you) and fix them.
The container concept really has it’s advantages but it’s a new world to learn.

9 Likes

I’m sorry, but reading this, I can’t bite my tongue. I hope it is taken as constructive criticism.
Totally agree with you. But…

4 Likes

Thank you @mrmarkuz as always for stepping in to provide quick reply.

Yes, my hypervisor is Proxmox. For fair comparison, both ns7 and ns8 had been on same storage. I use Ceph which is currently little over 900TB. Both VMs had identical configuration as far as the resources goes. So storage probably wasnt the cause.

I stepped into the migration with expectation of ns8 being slightly faster out of the box than existing ns7. Over the years I have installed several apps such as Mattermost, nextcloud etc and not to mention load of emails. So naturally it is little on the heavy side.

I could certainly increase the log level, but just loading the complete page was a pain. Most of the time it never finished loading. As is the case with most of the GUI pages.

I was aware of the migration tool. But I did not want to commit full migration before trying out the ns8. Wanted a clean slate, thus planned to deploy parts by parts. After experiencing continuous issue with the GUI pages, any plan there might have been for full migration was put on hold.

Yes there is working DNS in the enviroment, both primary and secondary. No wireguard VPN port was blocked. Both VMs were on the same subnet.

You know, I do not recall seeing that Uninstall option for the mail instance. I am not going to look at it again. If it is indeed there, I have absolutely no clue how I have missed it when I was specifically looking for that option! Big apologies is in order!

I moved away for no other reason but the Containerisation of ns8. I am sure there is very good reason for Neth team to go that route and in coming days I hope to get to know of the reason. But i think it should have been left simple, purely my opinion. I am comparing initial out of the box experience before any configuration. I believe the container approach is major contributor such experience. Perhaps my need to have some control attributed to the decision to move away as well.

I totally understand a product in active development. If I remember correctly, ns7 initially had quite a few issue as well. But they were just inconvenience. I dont remember the basic GUI had any issue nor issue with resource consumption.

I now have a functioning core email server leveraging Postfix, dovecot, SoGo, 100% functioning activesync taking slightly less resources than even NS7 did.
As I mentioned in the original post, we cannot have issue with our email infrastructure. That’s why I never actively used any other available app that came with NS7. We have dedicated nextcloud on vanilla debian, even though putting them under one vm one gui would have made life lot easier. With the container approach, I cannot imagine how much resource it will take just to have everything under one umbrella.

I am not leaving Neth so easily. I am going to keep active eye on the progress of NS8. May be not for email, but there are other things neth can be quite useful.

5 Likes

Cool, I don’t know much about ceph. Need to test it.
Please check compatibility with podman: Compatibility and Stability — Ceph Documentation

It’s a new world, from OS to a containerized app server.
It’s slower but much more flexible and…we can use Debian now!

I think there’s an issue with the installation. Usually the UI isn’t that slow. I suspect unreachable nodes.

It will take more ressources but podman is well optimized in using CPU and RAM.
The advantage is that you get secured separated applications. This approach makes it much easier to backup/restore or uninstall an app because the app not scattered all over the system, it’s just containers located in /home, secured by it’s own user.
NS8 also provides moving apps to other nodes or create user domain replicas.
Most projects provide docker images that can be run by podman in NS8.

That’s nice. I hope you’ll give it another try…

4 Likes

I try to be open for constructive criticism because it takes us further.

Is it about calling NS8 stable to early?

I came from the Windows world where one is beta tester for life so if you ask me…

I can understand to expect a fully working system like NS7 and if that’s not the case then it can’t be stable but it’s really strict when there’s a new container concept and NS7 had a complex config.
I can also understand to call a system stable when 90% of the adaptable NS7 base is available and working in internal tests and bugs were fixed in external tests (betas, RCs).

What I wanted to say is that no matter if stable was announced too early or not,

I really think the best way is to raise issues, ask for support or create howtos about customizing apps and open feature requests to inspire the (community) devs. I’m convinced that these things also powered NS7 to get that powerful and stable.

7 Likes

I think the definition of stable has nothing to do with the set of supported features. It is about whether the features work as expected and as documented in all supported scenarios, and whether there is proper error handling in all other cases. A stable system is predictable (it does what is documented) and reliable (it “just works”, always).

2 Likes

@jaywalker i can understand your definition, but I dont share it as mine.
Currently… how can I “migrate” from a “working as expected and desired, but soon unsupported” system, to a stumbling, lacking of features, lacking of refinement system, which works roughly the 75/80% of the scenarios of the older one? (and I’m excluding from the equation the current unavailable ones due to lacking firewall feature/integration with current generation).

How a customer can consider that acceptable (with all the hiccups and support ticket generated from missing pieces here and there) “for having the new toy”?

This lack of continuity has a simple yet complex response: this is a new product.
Has been called NethServer8, but (IMO) it’s closer to a 0.8 version of Container Orchestrator, if a “good one version” is NS7. Lacking features, lacking functions, a lot of options must be “invoked” via shell and not from the web interface and still it’s rough the migration in a not small set of cases. I’m glad for all the flawless executions happened, but the experience that @wahmed shared is not “that” uncommon.

I’m critic, I’m trying to stating some facts.
Don’t read grudge in my words: the job done until now is huge, and still commitment and competence of the dev team it’s here, without any doubt. Some decisions about wording into product definition, features, promotion… let’s say are a tad on the optimistic side.

@wahmed thanks for sharing your experience.

This part is one that… won’t change. At all.

Will scale higher during time, every occasion you will add a module (container), the disk occupation, ram usage, CPU request will increase. NS7 is a single system with services installad and configured, with only one container: AD DC.

NS8 is a container-based service suite, any module is closer to a VM than a service; consider that for the next metal-iteration of proxmox VE.
On one hand, it’s not pleasant: using more RAM, more CPU, more disk to do “the same thing” that the older version did it’s not that nice.
On the “infrastructure side”, also there’s a powerful switch, a router, a VPN system that’s necessary for all the cogs spinning as expected. That cannot come without “cost” in terms of resources.
On the other hand, you have a powerful tool to split load among different VMs or different host. I hope, for the next future, some kind of NethServer-level HA. I mean… the underlying architecture should allow it.

Also another huge advantage is the possibility to have some different kind of application managed from NS8 that on NS7 could not be installed due to incompatiblity/different version of packages on the underlying system (which is debian, if i recall correctly for your setup).
Currently several projects/products do not bother to deliver packages anymore, only docker/podman/kubernetes recipes.

It will be a long road… and now it’s way bumpier.

3 Likes

Thank you Wasim to have spent you time testing the product and also reporting back!
We always think that this kind of feedback is invaluable and the force of our community.

I’m not going through the exposed problems, because I think that @mrmarkuz already did an excellent job, as always!

I have seen this scenario many times: on every major change in any software product.
It’s hard to get familiar with new ways to do things, even more if you are a sysadmin or a developer and your comfortable with existing tools!
Think a about ifconfig vs ip commands: ifconfig has been deprecated by years but I still struggle when using ip :roll_eyes:

I fully agree on this and we are working hard on this side. When using NS8, sometimes I still feel some raw edges but IMO this is why we are used to old UIs.
I know that people that never used NS7, feel very comfortable with NS8 and they find it quite easy to configure. So for me, this is a huge kudo to all the design and dev team!

We hope to meet your expectations very soon in the near future :wink:

7 Likes

That is really what I meant. A complete new product such NS8 I dont expect it to have all the features of NS7, but the features that are officially released should be usable without much issue. I can live with less features but if i cannot use what is given that makes the long journey quite unbearable.

Container based design does have it’s advantages. With new product comes new features, higher resource requirement. Thats to be expected. Thats why everytime new Windows comes out, computer sales goes up and recycle plants gets new shipment of old computers. I am not sure why my vanilla setup had issues with the unstable GUI even with more than needed resource allocation. But certainly i will continue my test.

Nethserver has a great community and amazing Dev team. I forgot for how long I have been following the forum and have seen the no B.S. support community member gives and receives. Not always I post something. But with the NS8 experience I felt to voice my opinion.

The criticism and feedback had been aboslutely constructive. Thats what makes the life a product long term. Any team can make a great product but not everyone can make an amazing community. I have no doubt Neth Dev were and will be hard at work making NS8 a gem. It got some good things going and it would be great to see the result.

There are features in Proxmox i waited years before they became a reality. But I stuck with them for over a decade for the great community. Neth sits in the same level in my book.

6 Likes

A very good mindset!

That’s one part of a great community!

I’m also a Proxmox “fan”, just like NethServer.

My 2 cents
Andy

4 Likes

Just a screenshot from ceph<->podman compatibility from the link posted by Markus:

3 Likes

Does this apply to putting podman directly on Ceph or it affects podman inside a VM? Inside the VM podman should really be concern about what file system is in use such as ext4/xfs etc. Podman should not know what sort of storage the VM is really running on.

Yes, that makes sense.

What type of storage drives are used? NVMe, SAS, SATA…?

1 Like

The local storage is RaidZ3 ZFS comprised of Enterprise SAS spinners.

Nice, but still too slow, IMHO…

:slight_smile:

I’ld suggest adding in a PCIx card with two or 4 NVMEs…
Enterprise SAS spinners (7200 RPM) don’t reach 600 MBit/S specified for SATA / SAS.
SSDs can reach that speed.

Current NVMEs can easily reach over 10000 !!!

Put OS (and maybe large DBs on the NVMEs, and Storage on the SAS…

Your server will feel better than new!

My 2 cents
Andy

5 Likes

Wow thanks a lot. It sounds like a big shout-out to our community

3 Likes

I agree. For performance, I have Ceph. The local storage is only for temporary stuff such as experiment with new product or use it transient storage VMs. Enterprise SAS also spins at 10,000 rpm vs 7200 rpm SATA. They are not SSD but not so bad for occasional storage use.

In context of NS8, this obviously does not matter. NS7 as mail server works quite well even on desktop class drive based ZFS.

I agree!