Neth7 migration with replica on proxmox in ZFS

france · November 6, 2021, 7:28am

Hello everyone, as a result of disk latencies on my Neth7 server often the Apache service does not start. I assume that during startup it goes slightly in timeout, I have to intervene manually to start it. I had thought of migrating the disk of the neth7 server that is currently on a node in ZFS and replicating to two others. This migration I was thinking of doing it on a nas with an nfs share and a 1GB network card. Could I have your advice or opinion? Thank you all.

Ps: I’m sure that from systemctl it is possible to increase the waiting time for Apache …

Andy_Wismer · November 6, 2021, 7:44am

@france

Hi Francesco

90% of my 30 clients are using Proxmox exclusively with external storage. This is a Synology NAS (Most 4-8 bays, no smaller NAS used), and the NAS is connected to LAN (for Administration) and directly to the Proxmox. The larger models have 4 NICs, I use these then BONDED…

One of the larges sites has about 35 employees, using NethServer as AD, fileserver, NextCloud, Zabbix and more. This is running on a fairly powerful HP Proliant ML380 Gen10 Server. The VMs are all stored on a Synology DS1817+ NAS with Disks in HybridRAID.

This NethServer is actually used as AD from three sites, connected via VPN.
No performance issues…

My Personal Suggestion:

Connect a “decent” NAS (It does not need to be dedicated for Proxmox with lower usage!) to your LAN - and directly to a seperate NIC in your Proxmox - or via a Switch to all nodes in your cluster.
Define this as Storage in Proxmox, and migrate your VM disk there.
Start the VM and observe! (CPU, Load, etc over a cople of days, and give some Feedback!

This is the network of a smaller company using a HP Microserver Gen 10 as Proxmox.
The Server has only 2 small SSDs, used as System disks, all Storage is on NAS.
The NAS is also used for other stuff…

A slightly larger network, using 2 proxmox in Cluster, a dedicated Synology NAS-PVE, and a seperate PBS (Proxmox Backup Server). There is a seperate (same modell) NAS for general usage, also for NethServer backups.
All Storage still done on disks, not SSDs.
Only PBS and Proxmox have SSD for the OS.
Proxmox have a BONDED LAN, a Storage Network and a Cluster Network.
Backups are done to PBS over the bonded LAN.

→ Extra Note:

The Synologys are all set to do updates (Not Upgrades) - over the Weekend.
No VMs are ever shut down during this.
No issues in 5 years - Proxmox “caches” the disks of the VMs for this time.

My 2 cents
Andy

france · November 6, 2021, 8:04am

Hi Andy! Thank you as always for responding in super detail. Your configuration is similar to mine … Zabbix, AD, NAS. Well I accept your advice and as soon as possible migro the record on nfs and then we will see the results. My Nas is an XIGMANAS on microPc hardware with 8GB of memory and a 4-core CPU. I hope to get better results. I also avoid replication in HA for the server and its bugs (as I reported on the proxmox forum) and confirmed by them. See you soon and thank you.

pike · November 6, 2021, 10:22am

@france how many and what model of network adapters has these XIGMANAS installation?

france · November 6, 2021, 10:54am

re0@pci0:1:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x15 hdr=0x00
vendor = ‘Realtek Semiconductor Co., Ltd.’
device = ‘RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller’
class = network
subclass = ethernet

re1@pci0:2:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x15 hdr=0x00
vendor = ‘Realtek Semiconductor Co., Ltd.’
device = ‘RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller’
class = network
subclass = ethernet

pike · November 6, 2021, 11:04am

Realtek is a nice chip/ICs producer for networks. But IMVHO these network adapters start to show a bit more of issues when it is asked to produce sustained load or faster response. Moreover, the CPU use for running these adapters is quite superior to Intel 210, for example.

Therefore, still in my humble opinion, use this kind of hardware for running a NAS is… nice. But NFS is quite different than a simple NAS, moreover if it’s used as “system storage” for emulated/virtualized systems.
1GBPs is a interesting network speed, but compared to SATA3 is still 1/6 of the peak transport capacity, with higher latencies ad a bit more of overhead (SATA is Point-to-point, ethernet+tcp often has some delays due to switches).

After “not requested advice and cosiderations”, you could reach your result (start of Apache) follwing five paths:

optimize/improve network performance of your adapters tuning driver or replacing that with a better one, if the adapter and the system allow that (i had a massive improvement of the stability into my test-setup after installing a specific Realtek driver, but it’s CentOS 7 related only)
upgrade the XIGMANAS hardware (you’re going to do that, if i understand correctly)
increase caches, RAM or SSD if XIGMANAS could manage that as read-ahead cache
improve the level of the hardware (higher computational speed of newtork adapters, higher switching capacity of the switch, higher network connection like 2.5G or 5.0G or 10G ethernet)
roll back to local storage for Proxmox part of the filesystem of your nethserver install

Andy_Wismer · November 6, 2021, 11:46am

@pike

Hi Michael

On lower powered Proxmox using ZFS, the CPU will often go up to loads of 10…
This will entail IO delays on the ZFS and any involved VMs…

This is a new 8-core ( 4 real cores & Hyperthreading ) Intel Xeon E3 with 64 GB RAM, 2 SSDs as system, and 2x4 TB Disks in ZFS…
The “only” intensive job is a single VM backing up…
CPU and IO Delay both go up to 90% and latency is high.
On this image, the CPU Loads are: 18.54, 8.63, 4.67. That is VERY high!

→ Even the VMs, such as non involved NethServers here, show a very high CPU load on the Cockpit…

Just to be relative…

A NIC will not have much load during this.

However, I fully agree as to Realtec NICs being way lower in thruput than any Intel!

My 2 cents
Andy

france · November 6, 2021, 12:54pm

Thank you @Andy_Wismer @pike! You are perfectly aware that you can get the best by replacing the hardware with e.g. SSDs and other network adapters. As far as the ZFS cache is concerned, as @Andy_Wismer says, I have long modified the max and min values of the system and now it’s fine. But I want to try to transfer the disc to Nas and then see the performance status. It is understood that I will report my feedback.

pike · November 6, 2021, 1:23pm

@france ProxMox and more other tools can “move” out of the hardware the storage, but it’s really important to understand, evaluate and size the NAS/SAN subsystem for allowing the necessary performances.
Moreover, ZFS is really a powerful tool, but also a priceful one, so if your system cannot afford load, speed or latency of the filesystem… EXT4 or XFS could serve a bit better the purpose. Price is not money-related but performance related.
I mean… I have a really old laptop named “Casualty”, which have 3gb of DDR2, a 2.0GHZ Core 2 Duo T7250, a “dead” battery and a Samsung 850 EVO as disk, with Windows 10 (32bit). The SSD is wasted into that computer, and it’s closer to recycle bin than the desk. But still works, slowly. I cannot bear a thing about that machine, it’s doing the best performance possible. But i cannot consider that more than a casualty ready to be dead soon… Or maybe the next scapegoat for test uses

france · November 6, 2021, 11:11pm

Hi Andy, I migrated the RAW disk to the NAS and precisely to the internal datastore with an internal 2.5" disk. The performance was mediocre. After a while I migrated the RAW disk back to the NAS but in QCOW2 format, and to the internal SSD datastore disk. Incredible! At the moment it’s very fine, never went like this,… I also activated the write back cache.

Andy_Wismer · November 6, 2021, 11:51pm

Just see the size difference between RAW & qcow2…

→ On high performance systems, .raw is faster…

My 2 cents
Andy

france · November 7, 2021, 8:52am

But Andy, I don’t know if as a first test or transferring the record. RAW on the mechanical datastore the performance was mediocre. Now as previously written, the disk has been converted to qcow2 and migrated to the ssh datastore. Is my doubtful performance due in part to the SSD or qcow2? The difference is considerable, my server has never gone so fast, despite being now on nfs. Thank you

Andy_Wismer · November 7, 2021, 9:23am

@france

Salutti Francesco

Scusa, my Italian is sufficient to order something when in Italy on vacation - but simply not good enough to understand the - someimes important - details of technical stuff!

Possibile de fare in inglesi?

Andy_Wismer · November 7, 2021, 9:32am

Using Proxmox VMs on real SSDs (In Proxmox or Storage):

Always set these 2 options on the advanced Options of your VMs disk:

With these two options, your VMs OS will treat the disks as SSDs, which is correct.
Without these, they will treat disks as “somewhat faster” rotating disks…

My 2 cents
Andy

france · November 7, 2021, 12:03pm

Thank you very much, I try as soon as possible.

france · November 7, 2021, 2:56pm

Sorry, I didn’t notice I posted in Italian!

Andy_Wismer · November 7, 2021, 3:37pm

@france

Hi

As to your question:

I’d say in part both:

a SSD is roughly 100x faster than a hard disk (moving disk).
RAW is MUCH larger than a .qcow2 file - but can’t handle snapshots if the filesystem can’t (Like NFS can’t handle snapshots, wheras ZFS has that function “built-in”). On NFS, but also locally if using XFS, .qcow2 is the preferred format for the Diskfile.

A .raw file is the same size as the allocated diskspace for that VMs disk.
A .qcow2 file corresponds with the used space of that same VM, usually much smaller!

A major advantage is - if you have a cluster of Proxmox (min 2!) - you can now do “live migration”.
Move a running VM from Proxmox A to Proxmox B can take about 90 seconds, as both Proxmox can access the VM file, only the memory contents are actually transferred.

Then choose the Proxmox you want to migrate to:

Hope this answers your questions!

My 2 cents
Andy

france · November 7, 2021, 4:10pm

Yes Andy realized that I’m doing tests with too much difference in performance on nfs looks at the performance: