Setup questions: Nethserver/VMs/RAID - Proxmox?

Support
41

Hi @ylavi

If you’re going for a single power machine, by all means, do it!
I also have some clients with a single power server, but a PBS (Older server with newer disks and system-SSD) is available for very fast backups and recovery. These clients do have a NAS, but that is only used once a week for secondary backups of VMs, as it’s just too slow (I can’t backup all over night, the system has to be ready by morning 06:00!). PBS is fantastic, and really fast. An incremental backup of a 500 GB VM server takes about 2-3 minutes (To hard disks on PBS, not SSDs!). Fast enough to do a backup over lunch break… The NAS is more used for other office stuff than for VMs, since we have PBS.

TrueNAS with Passthru would not be for VMs, as the passsthru is not migrateable… But it can be used as a powerful NAS…

Backups to local USB drives tend to be slow, but work. It depends a lot on the disksize of the VM in question.

This is correct, Passthru is not compatible with HA and fast migration, but can work and can make sense, depending on your environment. Passthru works only for whole disks, not partitions.

Even with HA on Proxmox, not all VMs need / must use HA… Some can be excempt from HA functionality…

I’d strongly suggest sticking only to Proxmox supplied tools for backup of VMs (& Containers) - it just works without headaches! And options like PBS is just unbeatable!

Firewall:

I tend to prefer a seperate box as firewall, I use a lot f PCengines APU4 for OPNsense. This has the major advantage that whatever is screwed up on your VMs or NethServer, you always have Internet to troubleshoot and get help!

OPNsense does offer HA out of the box, and the cool thing is you can use ANY combinations of real hardware or virtual hardware. Also HA cluster with 3 or more nodes work with OPNsense…

→ Combining a hardware OPNsense firewall with a virtual one on Proxmox as a HA cluster for OPNsense gives you the best of both worlds, but saves energy and power! But you get very high availability on your firewall!

Combining the right things together:

A friend of mine has a powerful main, single Proxmox server and a second server, less powerful, running PBS and Proxmox PVE on the same box. The grafik card is passed thru, so he uses that box as power woirkstation, but other than his workstation there are no other VMs running on this box normally. It’s main use is PBS and Workstation… Proxmox runs the workstation, but can also be used as stand in, if the main server needs maintenence…

Hope I didn’t leave out any questions… :slight_smile:

My 2 cents
Andy

42

Can PBS back up a VM with its passthru disks? If so, do they have to be restored to passthru or can they become regular volumes (with ZFS still inside them) sitting on non-ZFS storage? That would probably suit me quite well if I need to fall back on the secondary server.

43

@ylavi

Yes, PBS can also backup VMs with Passthru disks, just as it can also provide a good backup solution for other Linux boxes (Non VMs). It can also backup Windows, but no GUI available.

A restore can go back to the VM with Passthru, or you can create a complete VM out of those passthru disks when restoring…

Note:
ZFS on ZFS isn’t a good solution. In case you want / need a NAS like storage, you could use OpenMediaVault, this runs well in Proxmox with Passthru or normal VM Disks…

My 2 cents
Andy

1 Like
44

The parts are starting to fit together now!

I did understand no ZFS on ZFS (which is why we were talking passthru). Will have to think about how to partition my secondary machine so that I will have non-ZFS partitions on separate drives onto which to restore the passthru volumes (containing ZFS) so that they can remain mirrrored. If restoring a passthrough drive do I need to create files the same size as the physical drives?

Alternatively, would it be a bad idea to do ZFS on something non-ZFS (like ext4) instead of ZFS on passthru? I’m thinking that I don’t really want to dedicate entire disks to TrueNAS at this point, especially if I’d have to restore volumes so unnecessarily large.

BTW in Proxmox is there any GUI for managing the physically-attached storage? The install process set up one drive but the other one needs partitioning and setting up with a file system. IIRC I can see it’s there but not use it. Do I need to do it all at the command line?

45

Hi

Proxmox does have a neat GUI for creating and managing ZFS, but it has no GUI capabilities for formatting a drive, erasing a drive or so on. These mundial tasks are best done on the console.
Mostly fdisk and formatting tools are needed / used.
Also setting up a swap disk needs console, but it’s easy to do. And even if you have 2 TB RAM in your box you’re NEVER going to use, it’s still better to run Linux with a swap partition than without! Linux generally runs better with swap than without. And as a Proxmox usually has enough RAM, it’s hardly ever used. 2 x 8 GB spare space on ZFS volumes are enough as swap, if using ZFS-mirror for your boot devices.

There’s no GUI to eg unmount a USB drive, nor any GUI tools to format / partition or whatever.
Console tools do work.

As for TrueNAS…
As much as I like it, if you’re not planning on shared storage anyways, but just need some form of storage, try OMV (OpenMediaVault).

Advantages: It’s just a simple Linux VM, disks can be enlarged on the fly when needed. Additional disks can be added. Passthru can be used, if needed. It uses ext4 or XFS, like NethServer. The backups are small, using Proxmox compression, and on PBS additionally with deduplication. No headaches.

Yes, but these can and should be on thin allocated storage. Meaning they will only use a mite more than you have data on those disks…

My 2 cents
Andy

46

Not a good idea, too much CPU / Interrupt overhead…

47

We were discussing TrueNAS as I would like to put my data directly on ZFS so that I could use snapshots for “previous versions” and share the ZFS volumes straight to SMB. From a quick look it seems that I could do that with OMV as well.

But it seems that I have a few conflicting “like-to-haves” (some resulting particularly from the above) and will need to resolve that, probably by adjusting those requirements

48

I have my data on NethServer (except large volume data like foto collections, etc.
The NethServer is on Proxmox (With ZFS, but not directly used for “versioning”) as a VM.
Backups (Snapshots) are done incrementally by Proxmox to PBS.
PBS uses a single SSD for System (Could also be mirrored), and 2 mirrored 6 TB disks (Or whatever) eg WD Red Plus or Pro.

This gives me at least twice daily “Versions” of every file / folder on NethServer.

The Backup of my NethServer takes eg 10 Minutes (twice that daily)…

I get the same as you (more or less), but much less headache with disks / storage…
This works with low-powered or very high powered systems - and anything in-between…

:slight_smile:

My 2 cents
Andy

1 Like
49

PBS:

My largest VM NethServer at a client has 1.2 TB disks, about 900 GB full.

The PBS has 2*4 TB Disks in ZFS mirror.

I have a total of 21 “Versions” of that VM, until 3 Months ago…

Bildschirmfoto 2022-10-31 um 16.10.39
Bildschirmfoto 2022-10-31 um 16.10.392868×698 228 KB

And there another 10 VMs on those disks, each also 21 versions…

Bildschirmfoto 2022-10-31 um 16.13.35
Bildschirmfoto 2022-10-31 um 16.13.351370×246 32.4 KB

The compression / deduplication is fantastic with PBS !!!

:slight_smile:

50

Yes I think I will forget the shadow copy/previous versions idea
It’s not like I’ve needed to dig files out of backups, and I understand that it should be possible to retrieve individual files from the sorts of backups you describe if I really need to

So your have Proxmox with the ZFS on those disks and QCOW2(?) files appearing to the Nethserver VM which sees them as drives which have partitions formatted to… ext4?

Does your Nethserver VM then have more than one drive with some of them dedicated to data?

51

Not for my personal / cloud usage. I do have such setups for certain clients, but not much, more for friends in home environments with eg large video collections…

Proxmox has itself ZFS, and stores files there in RAW format, not qcow2. On a few Proxmox, I still have some running qcow2, which is more than good enough. The cited example above with 1.2 TB is running on shared storage, a dedicated NAS with 4 Disks (Not SSDs) in RAID10, and dual BONDING for networking… Here qcow2 is used…

PBS stores in it’s own format, no need to worry about it, it works well, has usually the highest uptimes in alll my networks.

PBS, even if running the same basis Debian as Proxmox PVE, has much less reboots over the years. It does not run VMs or Containers, and can “restart” the new Kernel on the fly, which Proxmox PVE (Due mainly to Containers) can’t.

PS: Using a Synology NAS as shared Storage - and having automatic Updates installed on Weekends on the Synology - including the needed reboot…

What happens with the VMs in that time?

No issues - Proxmox can “cache” these until the NAS is back up & running and NFSis available…
→ Good to know…

My 2 cents
Andy

52

NethServer uses XFS as default fille system, it’s quite solid and rarely needs repairs. These must be done on an unmounted system, so I boot up the VM with SystemRescueCD for repairs.

Proxmox also defaults to XFS, by the way…

1 Like
53

It’s almost a year later and I’m back - this time with some time I can invest in this…
Having set the servers up once, I’m about to reinstall them with the latest version of Proxmox.

PVE servers (two) will be installed (again) with ZFS (four drives each, as two ZFS mirrors).
VMs will be given virtual disks in raw format (converting the NS pre-built image from zcow2 to raw before starting). This seems to be what is advised above.
PBS will itself be a VM on one of the servers. I intend to do some experimentation with managing this environment (backups, migrations, failure recovery etc) before setting up anything for production use.

I’m excited to be able to go straight to NS8 whose architecture seems so much more appropriate. My understanding is that I should set up larger VMs with NS8 under PVE, and those will manage containers inside themselves and between connected nodes which may be on the same PVE host or a different one. Is that correct?

54

Hello, do not use a laptop if you want to keep your data safe.
An used hp microserver costs about 200€ + 2 hdd 2tb another 100€ + USB external hdd 2 Tb 60€ (for backup!).
Install ns7 on mdadm raid1, USB backup every night and oven cooking at 180° for 25minutes and you have your home data safe for years.
PS.
Backup on external disk Is mandatory. Raid Is not a backup! Buy new not-cheap disk. Avoid wd green or similar.

Want to do Better?
4 hdd + Proxmox on xfs Raid10. Better performances and proxmox experience, but should not your target i think. Bare metal ns7 rocks.

55

@sarz4fun

Bare metal?

What a dated concept.

Only older people still use such concepts. Virtualize!

Crappy and hardware fussy on disaster recovery.

Better?

Proxmox, ZFS (Not XFS!), Backups to a dedicated PBS, eg on an Odroid H3…

My 2 cents
Andy

56

Apparently @sarz4fun didn’t read the whole thread… That laptop is long gone now.

I think I will make my backups to a virtual PBS on the little host and perhaps then sync those backups to another PBS on a cloud VM - rather than having a dedicated machine locally. Maybe I will also test direct backup to a cloud PBS.

But my main question this time was about NS8:

I’m excited to be able to go straight to NS8 whose architecture seems so much more appropriate. My understanding is that I should set up larger VMs with NS8 under PVE, and those will manage containers inside themselves and between connected nodes which may be on the same PVE host or a different one. Is that correct?

But on reflection I’ll ask that in a new thread

57

Sorry, i’ve Lost just a couple of comments…

58

One size do not fits all, @Andy_Wismer :slight_smile:

Virtualize is one way… not the “correct one” :wink:

59

For those still stuck in pre 2000 days…

Virtualization was around before PCs, it started with mainframes towards the end.
Mainframes then could run several “members”…

It stopped when PCs and PC-Servers came, but returned with a vengeance since 2000 or later.

All major, current companies use virtualization, those who don’t are on the downturn cycle, have old school bosses (usually in that age group somewhere between seniority and senility…), are not current, or a combination of several major issues…

If they could, some would probably prefer a single core solution, maybe even running goold old Dos…

:slight_smile:

My 2 cents
Andy

2 Likes
60

Please… this is now completely unrelated to the reason I revived this thread - and even to the original reason for it
Can this thread be closed without marking a specific post as the solution?

1 Like