Nethserver NS8 Gateway timeouts with Nextcloud

Hi everyone,

Hope you’re all having a fantastic Valentine’s day. I have a strange issue with the NS8 version of nextcloud.

I’ve rebuilt the cluster on Debian 12 (I couldn’t get it going on Rocky Linux as it never created the user admin, and failed to load the WebDAV service, even on a new install of Rocky Linux with latest updates) and every runs okay. I’ve got an 8 core 3.50GHz machine with 32GB RAM and it’s installed on bare metal. (Proxmox was retired due to issues with ZFS. Basically it was doing large pauses on the VM’s even though CPU,IO, and RAM were very low usage)

Nextcloud runs well but when anyone upload files via the web browser (more than ten) the browser locks up and gateway timeouts appears after a bit…even with just a single user using Nextcloud. This is rather frustrating and the only way to get nextcloud back is to go into the nextcloud cluster settings and press save (no changes entered). This appears to restart the service until someone tries to upload files again.

Any ideas on where to look to fix this, find an error log as Nextcloud logs seems to have very little relating to this.

Thanks

Turbond

Hi @Turbond

Very strange, I’ve over 50 Proxmox running, over 30 just for clients. 90% use only ZFS, on hardware from Low range (min 32 GB RAM) to fairly high end servers from HP, Dell & Lenovo.
None have issues with ZFS or performance.

Typical issues are:

  • Wrong hardware
  • RAID Hardware used
  • No Swap partition (Not file) created
  • Wrong settings for /tmp
  • RAM Limitation not or wrongly set.
  • Not enough RAM

I’ld NEVER go back to bare metal installs.

Bare metal has too many issues with disaster recovery on different hardware.

My 2 cents
Andy

Hi Andy,

I agree with you, I prefer virtual machines but when I have 30 staff screaming at me as the invoice system takes five minutes to submit one invoice, I have to act (re-act) and get the system running. So the invoice system is working well, and the cluster on bare metal is also working well. Just this nextcloud glitch so hopefully someone can help :smiley_cat:

1 Like

Hi @Andy_Wismer and @Turbond Turb,

that is curious. I had a similar problem with NC, Collabora and Webtop with a virtual NS8 machine using Debian 12 as a substructure. Everything ran slowly, even though 20 GB RAM is reserved for the VM in Proxmox. I then installed a new VM with Rocky and everything is running fine.

Regards…

Uwe

Screenshot 2024-02-14 203104

Hi UWE,

Except in my case when I tried doing a new install of NS8 on Rocky Linux with latest updates of Rocky (7th Feb 2024) the install script completed but WebDav service did not start, nor was the admin user created and no web interface was available which is why I had to go back to Debian 12. I tried it twice with both VM and bare metal and couldn’t get NS8 going with Rocky :disappointed: If I could have got a Rocky Linux system going with NS8 I would have…the reason I changed over was NS8 failed totally on a live system, with the same WebDav.service error after Rocky Linux was updated on the VM, which then caused active directory to fail, and several other services (I still have the vm and journal logs I can post if required) and I couldn’t access the webpage to do a cluster backup…so had to manually copy over 512,000 files plus redo the active directory as cluster restore wasn’t playing nice either.

I really have tried to use Rocky Linux but due to the above I’ve thrown it out and gone to a more stable debian until Rocky WebDAV.service issues are sorted.

I seem to be encountering strange issues like the above, which others have not, and wonder if it is New Zealand time zone/hardware differences lol.

Turbond
PS maybe it’s due to the fact I have over 500,000 files and each users email box is around 68GB on average

Please contact @Andy_Wismer. He really is a guru when it comes to Linux and Proxmox in particular. Together with him it should be possible to find a workable solution. Thanks to him, I have become a fan of Proxmox through and through and use it intensively. Together with the Proxmox Backup Server PBS, this is a really useful and energy-efficient solution.

2 Likes

I have three clients, where the boss is the premium culprit. All three are fighting hard for pole position…

Average Notebook usage (new powerful hardware!):

1-3 Firefox Windows open, with on average 100+ tabs open, a lot of youtube videos.
Thunderbird open, 60-150 GB Mail, but with over 100-300 Mails open for weeks on end.
Word / XL open, with over 100 Docs open, also for weeks on end.

Any such use is beyond normal use. I’m surürised hat it manages - more or less - to handle this load, more the Windows box than the NethServer behind!

There is always a worse user, than the worst you’ve ever seen!

My 2 cents
Andy

I’ld have tuned the system and got it running smoothly as one of my cats purring… :slight_smile:

My 2 cents
Andy

Hi @Turbond

If you’re interested, feel free to drop a PM, and I can offer assistance with tuning Proxmox, also with several different scenarios, depending on use case.

Open Source is generally free.
I don’t bitch about anything, but do suggest improvements and twitches.
I don’t consider myself a coder, even if I can do coding.
But to give back, i provide the community help, relieving the real devs, and provide know-how and advice here, and in other places…

My 2 cents
Andy

Except that the Ricky Ricity (Rocky) Linux system was broken after update and couldn’t be repaired, and wouldn’t work on either bare metal or the vm. But we’re getting side tracked…the issue is with nextcloud gateway timeouts :smiley_cat:

Once this is fixed I’ll probably go back to virtualbox which is what I was using with NS7…at least it works with only a small decrease in performance… In fact it was running the accounting system at least ten times faster than my super power Proxmox machine (64GB RAM, Raid 10, 6TB HDD SSD, 12 Core HP workstation server) on an old dell 6 core, 2.8Ghz server with only 32GB RAM which also held the entire NS7 install with Nextcloud running flawlessly with all apps, webtop and mesh Central.

Even the fastest, most powerfulö Ferrari won’t budge, until you put in the right gear (Or any gear).
In neutral, you can waste as much CO2 as you want and rev her up.

Same goes for Proxmox.

:slight_smile:

Virtualbox is a toy compared to Proxmox.
But you’re free to choose.

My 2 cents
Andy

Hi Andy,

You already have a few screen shots of my Proxmox when I had Rocky Linux fail the first time on me and I needed to rebuild back in December/early Jan. I did a standard install, with ZFS and setup the raid. But then maybe 64GB ram isn’t enough with 12TB (6TB available due to RAID 10) of hard disk. :smiley_cat:

Hi Matthew

Rule of Thumb for RAM limits on Proxmox:

(Without any limits, ZFS alone will eat up 50% of RAM for ZFS, add in 2 GB RAM of ZFS system, and 2 GB for Proxmox.)

2 GB RAM for Proxmox PVE (With or without ZFS) - This is not really ZFS usage…

2 GB RAM for the ZFS subsystem
1 GB RAM per TB Storage in ZFS (Here the available matters, not the used disks total volume!).

So in your case, ZFS would use 8 GB RAM if optimized,
34 GB RAM if not optimized.

Note: It seems that since the latest updates, Proxmox handles the RAM limiting correctly from installing on ZFS with boot. But it doesn’t handle swap. This depends heavily on the ISO used.

If you did not leave aside say 8 GB Diskspace per Disk during install, you will not have any disk space free to create a swap partition… :frowning:

No matter if you have only 1 MB of RAM - or a full Petabyte of RAM, any UN*X / Linux runs better with a swap partition, even if it’s hardly ever used!

ZFS on Proxmox has additional caveats for the TMP area, this needs to be handled extra on ZFS booted systems…

Please see also here:

https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage

and here:

https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Snapshot_of_LXC_on_ZFS

Those two pages contain a lot of valuable information!

My 2 cents
Andy

2 Likes