NethServer Hotsync [Needs Testing]

HotSync is a set of scripts that leverage NethServer backup to copy a snapshot of a NethServer over another one every 15 minutes. In case of a disaster, it’s possible to promote slave to master in a few minutes. We call it “Poor men HA”.
It was used few times in production for the enterprise, and AFAIK it works quite well.

If you want to take a look at it, here you can find a draft of documentation, including new features that we’d like to implement.
You can try current version installing it from nethserver testing repository or take a look at the code

Now, we’d like to publish it for community and start working on it again. Maybe rewrite it if it’s needed.

Any idea/request/opinion is appreciated

14 Likes

reading through right now…
just a small typo with ssh port (22, 222) on Current implementation configuration section.

:+1: Fixed

Good job, it makes me think you created an affa like. The ‘rise’ feature is really interesting

It looks promising but we need more tests and scenarios!
Tagging some people that may be interested in this new module!
@GG_jr @wbilger @m.traeumner @aaronux @Walter_Schoenly @Ctek @jelle @WolfAddict
@rowihei @enrilor @kunstlust

1 Like

Wow, this sounds VERY interesting, and exactly what I am looking for, I would love to test this.
Couple questions,

  1. How do I get started on testing, is this just built in to the new version? Do I just follow the “Current implementation configuration” instructions.
  2. is it suggested to use port 222, what if I alredy have ssh server running on 22?
  3. Is there any way to do more critical data more often, or should I just do a regular lvm snapshot using my own scripts outside of this?
  4. when promoting the slave to master, is the IP address and hostname updated?
2 Likes

Glad to be useful sometimes :slight_smile:
@Stll0 @nrauso can you provide some additional info?

install package from nethserver-testing repository and follow the “Current implementation configuration”

there is port 222 in the example to show how to change ssh port, but default is 22. You can skip that config setprop. I’ll try to write it better on doc

how often? Now syncing is every 15 minutes. You can change it with a template custom of this nethserver-hotsync/root/etc/e-smith/templates/etc/cron.d/hotsync/10hotsync at master · NethServer/nethserver-hotsync · GitHub
But keep in mind that a mysqldump is done every time, if you have a big mysql database it is time consuming.
If your important data are on mysql and you whant to have a fast sync of it, or if you have a huge mysql db, I suggest to disable mysql syncing at all and setup a mysql clustering

BTW, it’s safe to use short time, since there’s a lock that prevent to run more instances at the same time

Yes: promoting a slave to master consist of a configuration-backup restore. If you have more than one network interface, you are prompted to chose which network assign to each network device. If you have one, IP is automatically assigned

2 Likes

I think 15 minutes would be fine for email etc, I just have a couple of things I would want to be every couple minutes, for axample, an access database that I have stored in a samba share. My samba shares are on there own lvm, so I can do an lvm snapshot and then rsync the files necessary, but could this possible conflict with the Hotsync?

I don’t know, you should try. I guess it depends on snapshot size.
But rsync isn’t efficient on binary files. For MySQL and PostgeSQL (the second is in development) Hotsync make a dump of database and copy the dump for this reason.
You can dump db content and write a custom script (also custom scripts execution is in development)

I will test this as soon as I can. I have an issue with webtop updates, so I a can’t update with the latest updates on my test machine until I get that sorted out. Does this need to have the latest nethserver updates to install? And if so, what needs to be udpated, so I can leave WebTop at it’s current version?

It’s[quote=“wbilger, post:11, topic:8391”]
Does this need to have the latest nethserver updates to install?
[/quote]

No, you can take the package from 7.4 testing repo and install it on any 7.x
But Webtop syncing is in development at the moment, you should have it on testing package in a few days

Ok, I’ll give it a shot. I will test for a couple weeks on a couple test VM’s.
Any idea how safe this would be to run on production, giving that the production server would be the ‘master’ and a test machine would be the ‘slave’? As of right now, the production server runs in a VM and just has a nightly snapshot backup. I wouldn’t do it until at least after a couple weeks of testing on 2 test servers, but just wondering.

I think safe for the production server, unsure about the restore of the slave.
Quite sure that sync of your access file doesn’t work :sweat_smile:

That’s what I was hoping, but still better than just a daily snapshot I have now.
Not sure what you mean about access file, Does this sync the samba shares in /var/lib/nethserver/ibay/ ?

Yes, but if file changes during sync, it will be very likely corrupted

Not sure I am understanding. In the normal process of the day, this HotSync will be running every 15 minutes correct? Are you saying that if a file in a samba share is open during this, it will be corrupted?

it will likely be… and since you told something about an access db, forget to sync it every few minutes if it’s open… it’s not a database, it’s a binary file and so it will be copied everytime…

What would be a reason not to set up a HA virtualization backed by NAS storage and let the HA part be done there? This can be done for about the same amount of money as two servers with proper UPS and whole shebang or am I missing some of the constraints people have to work with ?

2 Likes

On a production server there are going to be many, many files open, how can
this work then?
Just wondering how this works, is there not a snapshot taken before the
HotSync? Or is this basically dumping databases, rsyncing them, and then
importing at the other side, and then using rsync to send all files over?
If there is not a snapshot taken, could there be right before HotSync runs,
then rsync the snapshot data?