The starting / shutdown feature of Proxmox is quite powerful, and is all in Web…
I’m using a Raspberry here in Switzerland (Etwa gleiche Stromverfügbarkeit wie in D, die Stadt mit den meisten grossen Ausfälle ist ausgerechnet Zürich!) for UPS, it controlls the NAS, the Proxmox and the VMs. Proxmox tells all VMs when to shutdown, according to the startup list. (In reverse oreder of starting).
Hello Andy
first a hello from the new MaskLand … Burkas for everybody … 
same problem very similar solutions
A solution is 90% H2O, but not by russian ther it is Vodka …
some day the elektric Energy price on Stockeexchange is under zero , like US Oil last days …
Virus times crasy times …
Hello Axel
Our government missed the boat and didn’t have enough masks. They even sold the governmental dept for controlling alcohol (Früher: Eidg. Alkoholverwaltung) - they had storage tanks around the country. After privatization, the new owners sold the alcohol, and then came corona, we didn’t have enough alcohol to make desinfectant!
As german, you may or may not be aware of this one:
Enter in Google “Wir liefern das gas von morgen” and turn to pictures (Bilder)…
Peinlich (In 2006!)…
Ein “Digitalisierungsfehler” - Die Prüfleser sahen die Seite - mit ausgeblendeter Werbung, zwecks Ablenkungen minimisieren.
Und die Werbung war - wie Google - kontextsensitiv, darum kein Strom- oder Öl-werbung…
English translation:
A german weekly did a several page article on the attrociaties of the Nazi Death Camps (Auschwitz).
On a double page, with pictures of murdered clildren on top, was a large ad on the right hand side, by E-ON, germanys largest utility and power company: We supply the gas for tomorrow…
An error of “Digitalisation”: The proofreaders didn’t get to see the Ads, they were blanked out to increase concentration on the articles - and the advertising used google like context sensitive advertising, so no ads for electricity or oil, but: Gas (!)
Embarassing, in 2006!
I should specify that I’m talking about XOA installed from source, which is free and full-featured:
It has solid backup and reporting capabilities, and lets you do pretty much anything else you could do through the Proxmox web GUI. But there is a chicken/egg problem in that it runs in a VM, so you need a way to set up that VM.
No, I had it right–with that version of XOA installed, you can do pretty much anything on your Xen/xcp-ng server that you could do on Proxmox through the Proxmox web GUI.
I think I may have been misunderstood. I read that VMs under some circumstances can mess up an mdadm RAID on the host if it is somehow given direct access to the storage as a result of how things are configured. Nothing to do with a RAID within the VM.
I picked up on NS handling RAID1 automatically on install as a plus when I was thinking that I’d install NS directly on the machine. If in a VM obviously not relevant.
@Andy_Wismer I’m in Israel. Aside from being the type who prefers to use old equipment as long as I can and not throw it away, my purchase options are limited as we don’t have anywhere near the range available in Europe to start with, and the second-hand market is weak.
@Andy_Wismer @danb35 @Axel Thanks for the input on host alternatives, I’ll do some more general research to understand the pros and cons because it sounds as if for me it would just be a matter of preference.
Hi Yitzchok
I assumed, from your jewish name, then again a lot are in Europe, US and elsewhere. Shalom!
You’re telling me about using old equipment?
I am running at home two 9 year old servers, I’m writing this on an 8 year old Macbook.
Besides my iPhone, the newest IT equipment is a Raspberry 4…
It’s just my technology, know-how and methods which are new and up to date, my hardware at home isn’t!
My 2 cents
Andy
Art of buying is not the lowest price, but the wiser choice.
Sometimes it’s useless spend more. Sometimes is necessary pay for what you need.
Shalom Yitzchok
I am using a more than 7 years old dual core AMD Low Power Opteron server with 16G (32G) Supermicro server. My notebook is a old Thinkpas x230 Tablet with Linux and Android second is a X61T “Basestation” is a HP8300 with a I7 860…
If you like to by a new old one Xeon L5630 are cheap and low power… With 2 of them you had 8 Real Core with Hyperthrading 16 Core price in europe around 200€ with 16G DDR3 ECC or better more RAM …
and its better to test software with small and old Servers. If its flying with a small server, than its fliying with new hardware only higher and faster
With a lill more money you can buy a usesd workstation very near to a Server but more silence HP620 or HP820
Hello Dan
XOA is the VM part of it. I you put the chick out of VW everthing is fine. I Put it on a a thinClient but you can use a old notebook or slim PC. Orchstra is the same like XOA but not running in a VM. My Orchestra is a old HP8000 or a HP Thinclient. Indstalling a fresh debian and use the installer from Jarli01. he had a updater to
With a external Orchestra it is easy to handel a UPS. Than you need some scripts do stop the VM in the rigth order. So it is not a good idea to let run your PBX is the same VM like the other parts of Neth
What i forgot, with a external Orchestra PC you can connect a USB3 HDD and store the beackup there. Than change the drive for next days
I think it’s more accurate to say that Orchestra is the software itself. XOA, which I was incorrectly using to represent Orchestra generally, is a VM appliance of Orchestra (and thus Xen Orchestra Appliance = XOA), but non-Appliance installation of Orchestra can still run in a VM. So this:
should instead refer to “Orchestra” installed from source. XOA can handle backups, but I believe you’re looking at a paid subscription in that case.
you are rigth the XSAN and the Patchservice for Orchestra is to pay (or doing by them selfs)
The patches for the VM are included …
My reasion to let it run external was chick/egg and the low speed to a USB HDD
Its ligth around 8G HDD with debian and Orchestra and a local HDD can use for Backups. With a NFS or Samba you can handel easy the install ISOs for VMs …
Hi all!
Two years later I’m finally making some progress and reviving my thread.
In the meantime, having heard your feedback, and in between waves of Covid and various family issues, I have managed to acquire more hardware and swap other things around and I now have better equipment to work with.
I’d appreciate your feedback on a few things.
I now plan to install Proxmox on a Dell Poweredge T410 (a proper server!) and also an early i3 desktop (consumer motherboard with 16GB RAM). Both of these will have at least a couple of drives but assorted sizes. My idea is that the second machine will allow me to run at least a core set of services if I need to take down the main host for maintenance. It would also be a target for backups. I don’t think I’d bother with mirrors or RAID, I’d make more of an effort to automate backups; machines to the other host, and data probably to the cloud.
My first questions relate to Proxmox configuration:
I’m also thinking about my network layout (this is for a home setup).
First of all - should I be trying to implement ipv6 yet? I’m inclined to, but it seems that various things on server side might still not work well on ipv6 yet (did I read that about Nethserver even?) and we might not have ipv6 on cellular devices for a long time yet (on our networks here, anyway).
At one point I thought it would be nice to have the whole network behind pfSense (or alternative) (virtual) but then if that host or VM went down that would surely complicate internet access for our computers and other devices. Also I don’t think I can easily run the cables necessary to put the VM host physically in between the workstations (and wifi) and the internet connection.
So it seems that our devices will be directly behind NAT, and the various virtual servers would be on a different subnet or something (behind the same NAT), using a firewall to protect them alone. I’d continue with ipv4, using port forwarding.
Would anyone suggest a different approach? I’m having an incoming fibre optic line and internal ethernet cables installed in the next couple of days. Right now I intend to keep the router in the same location as with the current VDSL, but that might turn out not to be possible, and maybe your feedback will encourage me to do something more complex.
I apologise if my questions aren’t specific enough - I’ll add more detail (and maybe more questions) as needed
Thanks
Hi Yitzchok
To answer your questions:
Yes, I’d suggest long term planing for IP adresses for Proxmox.
I use the IPs x.x.x.61-66 for Proxmox PVE Hypervisors.
Proxmox Backup Servers get x.x.x…78-79.
And I’d also suggest - if possible - creating seperate networks eg for:
Note: There are dual / quad NICs available on the market, just choose the right, suitable bus / bracket…
Firewalling:
I strongly advise using a dedicated box as firewall. I tend to use PCEngines apu4d4, or larger, faster models for larger installations. I do not use PFsense, but only OPNsense.
Why a decdicated box?
No matter what screws up, it’s usually never the dedicated firewall box (unless lightning struck your power / DSL lines…). That leaves you at least with working Internet, also Internal DHCP / DNS services will still be up & running. VPNs are also all available.
Note: Using HA / CARP you can set up a full high available firewall with failover. One box can be hardware, the other one can also be in hardware - or a VM in Proxmox.
Storage:
Proxmox makes for a very good and stable storage for VMs and more.
I’d strongly suggest using zfs (Reserving RAM according to the rule of thumb (2 GB for Proxmox zfs services, 1 additional GB RAM for eact TB of storage).
→ Use ZFS with RAID and remove the HW RAID controller! (For the Dell Server)
My 2 cents
Andy
My primary hardware’s not done yet (I’m going to buy more drives to make mirrors with ZFS after all).
If I want to base my regular file storage on ZFS (because of snapshots and versions etc) and then share that over SMB which NextCloud would also have access to, would it be unwise to do that directly from the Proxmox machine? I suspect it might be tricky to manage that storage from the perspective of permissions and so on. I imagine I don’t want to run ZFS inside a VM which uses volumes sitting on ZFS… I don’t really want to have separate box/SAN running for storage.
At the same time I’m thinking about how to configure the network.
I have in mind to go for a network topology where my virtual machines are protected by a virtual OPNSense firewall and PCs etc connect directly to the wireless router as is customary in a home setup. My router (running DD-WRT) should work as a VPN server so supposedly I could have a VPN which isn’t dependent on the VMs. I thought this was what Andy mentioned above (Setup questions: Nethserver/VMs/RAID - Proxmox? - #3 by Andy_Wismer) and that’s why I started writing here (instead of asking on a general networking forum as I intended). Looking again at the diagram I see that it’s different but I’ll ask here anyway.
My idea was to put the VMs (at least the servers) on a separate subnet and set up routing (including on the WAN router) and firewall rules through OPNSense to limit access to those and also to prevent them initiating unwanted connections to my LAN. I’d use port forwarding to send external traffic to the internal IPs of the relevant servers and that server LAN would be a DMZ of sorts. Is that a good idea? Or unnecessary? Can OPNSense act as a switch between different parts of a LAN where all clients are on the same subnet and still restrict the traffic?
Andy mentioned HA/clustering but if that needs (as I was told on the PVE forums) 3 always-on machines. That’s too much for me. If I want to move VMs temporarily to the secondary server I’ll back them up and restore them. Or is there some way to make use of the clustering functionality to move machines around without having all the servers running all the time?
I hope my thoughts are clear enough to respond to.
Thanks!
You’ve conflated two things that are not the same. HA does, as you’ve been told, require three always-on machines (though one of those can be a Raspberry Pi, if you happen to have one of those), but clustering doesn’t. But to be useful, yes, all members of a cluster need to be powered on pretty much all the time.
Hi @ylavi
Storage:
I would not suggest using Proxmox directly as share for SMB. It’s possible, but compared with the rather comfortable WebGUIs you have for adminisrating PVE, the File Sharig would have NO GUI…
It would make more sense to add in a couple of disks as “Passthru”, so you could eg install a TrueNAS inside Proxmox, but using it’s own dedicated disks. The “System” disk of TrueNAS could be an ordinary VM Disk in Proxmox, the Storage of TrueNAS using passthru.
Cluster / Fast-Migration
Clusters can be used for full High Availability, but also “lesser” things come out of the box…
True that you need at least 2 nodes and a “voting” node eg as VM or a Raspberry, better is three full nodes, but hey, you get full High Availability!
But even without HA, or for those on the way to full HA:
For one, it makes managing VMs and VM-IDs much easier, especially on several Proxmox and PBS.
Secondly, this already gives you fast live migration between nodes!
Using a seperate 1 GB/S “Cluster” Network, it takes 90 seconds to live migrate a 200 GB VM with 12 GB RAM from one node to another…
Proxmox PVE does once in a while upgrade the kernel (much less often on the paid versions). PBS does not require a reboot, but PVE will (Due to VM usage). Using live Migration, the VMs running inside do not need to worry about downtime - just move the VMs on to another Proxmod, upgrade the server and reboot, then live migrate your VMs back… Really cool!
Networking
The more Proxmox, the more you want to think and plan your networking options.
Best would be a 10 GBE for your LAN, a seperate 10 GBE for Cluster-Network and maybe a 10 GBE as storage and another as Backup-Network.
I’m thinking here of shared storage for VMs (A requirement for fast Migration!) and Backups going via it’s own network, as these can generate high network loads.
That said, you can use anything from 1 GBE upwards, add BONDING or change NICs to 2.5 GBE or 10 GBE NICs…
Note:
A common mistake: BONDING 2 x 1 GBE will not allow a datastream running with 2 GB/S, but will allow 2 datastreams with each 1 GB/S… This is NOT the same thing!
Internal Security
Depending on your planned Environment, I would not overdo “internal security”. Keep your perimeter security top, I use a dedicated OPNsense box as hardware firewall protecting my Networks and also providing DHCP / DNS for all internal networks and also VPN services (I use both IPsec and OpenVPN, IPsec is my preference for site2site, and OpenVPN for RoadWarriors. Next up may be Wireguard, also available in OPNsense).
If you’re in an educational environment, or in some form of public WAN, by all means, treat each internal the same as external, top security.
In an Enterprise sort of environment, I would not place a firewall between clients and servers, unless you’re in a secure financial / military environment.
OPNsense
OPNsense is a great firewall, and the “official” successor to the M0n0wall system. I am known here on this Forum as a great fan of OPNsense, I also used the older M0n0wall. I only looked at PF-Sense, never really used it…
However, and this is valid for all BSD systems: While Linux has no issues distributing all routing an port forwarding CPU tasks to all cores available, BSD will allocate a single core for this task!
That has the great disadvantage of requireing a higher speed CPU, the amount of cores are fairly irrelevant.
In that sense, using eg DD-WRT or OpenWRT (Which I use at home) as a VM Router / Firewall may make more sense, at least performance wise.
However, and here are the big “gotchas”…
Both DD-WRT and OpenWRT are optimized as WiFi Routers on low levelhardware and do that well. But even here, where they are optimized, the GUIs are rather lacking…
Both can easily do firewalling and port-forwarding, also DNS and DHCP and even all three VPNs are available, from IPsec, OpenVPN and Wireguard. However, the GUIs are tendentially error-prone and errors are easily overseen. One notices, that these boxes aren’t optimized for firewall / router usage. And errors in security tend to be more dangerous than an unaccessible WiFi access point!
I do use several networks internally on Proxmox, often enough I do use an “internal” OPNsense to connect these, but in my case, it serves more LAB useses or emergency recovery, where I have to emulate a clients network rather than purely security needs.
Here’s a sample of a “hosted” environment at Hetzner (Finnland): Two AMD Ryzen Servers, one as Proxmox, the other as PBS, with a second PBS at my Home.
Networking on Proxmox at a client of mine:
Proxmox Cluster for SME, Home and LAB environments
Concept:
2 decent powered servers, to be used as full HA capable Proxmox PVE.These have plenty of RAM.
1 third, lesser server, to be used principally as Proxmox Backup Server (PBS) with enough storage space. A third Proxmox PVE runs also here, but normally no running VMs, only as voting member of the cluster to achieve a full cluster Quorum (=working and functioning cluster with voting capability.)…
Hope this is somewhat understandable…
My 2 cents
Andy
@Andy_Wismer and @danb35, thanks for your answers. I’m sure they’re time-consuming (especially Andy’s) and I hope you also manage to get some work done… I answer questions in other forums.
Your answers are informative but they seem to be dragging me out of scope a bit.
If a cluster (and especially easy migration between nodes) is going to involve keeping multiple machines running, I’m not going that way. I invested (for now and ideally the next couple of years at least) in one main server-class machine which is intended to be good for running the VMs and holding the storage. I don’t intend to run a separate device for storage (which would appear to be a requirement for live migration) nor a third one for voting (or even backups).
The second machine is really intended to be there in case for some reason I take down the main server and can’t start it again. Maybe I’ve been very lucky but my current server (an old desktop-class machine) has been running pretty-much non-stop for years and I’d be happy to see the new machine do that too. I concede that if I have to stop my VMs altogether for restarts and so on of my host I might be a bit reluctant to do it, but if I know that I have made backups of those machines I can restore to the other host if anything goes wrong I think I’ll be prepared to do that.
Given the warnings of bit-rot and the fact that my secondary machine will probably not have mirrored drives and maybe not even ZFS on the single drives, I am a little reluctant to move volumes over to it just temporarily - I’ll take my chances on a backup if need be but would rather just wait during the maintenance.
Running TrueNAS with Passthru makes a lot of sense. But I also get the idea that use of drives in passthru mode is rather incompatible with migration of VMs between hosts - and perhaps even with clustering in the first place? I read somewhere something like that the storage configuration is the same for all nodes but even so the physical configuration and the content can be different. I wasn’t sure what to make of that. Can passthru be used with partitions, or only with entire devices?
Backups I was thinking I’d make to local USB drives which I’d switch every so often and also I’d make complementary (differential/incremental) backups to the cloud, perhaps more frequently. What would be ideal would be to be able, if required, to do a “full” (but a little dated) restore from (relatively fast) USB drive and then get the difference from the cloud. I see so many backup tools out there which can work with ZFS and make use of the snapshots - is there something which can do that? Or alternatively will PBS help? Are there any backup techniques to avoid which would only restore the current state and not restore the ZFS snapshot history too?
Just a note about the networking - yes the firewall belongs on the perimeter but while I’m getting used to things (both the firewall and the configuration of the new VMs) I think I’d rather not have my regular internet access dependent on a new firewall running as a VM. I can switch things around later when I understand better.
After all that - if I’m going for a single always-on machine and another just-in-case machine, where the former has at least one VM with pass-thru, what strategies might you suggest for being most ready with the just-in-case machine? Is there any way to replicate ZFS periodically with a scheduled “power on, transfer, power off” process? Especially in a way which would allow switching to use the replica and then replicate back again? Or is that all HA-type stuff? If backup-and-restore, which tools are reliable?
I realise that some of these questions are still very open-ended but hope you can tailor answers to the direction I’m trying to indicate.
Thanks!
