Suggestions on how to sync Nextcloud data to external S3 storage

Hello Team,

I hope everyone his keeping healthy and safe during this crazy time!

I’m hoping this community has suggestions on how I can automatically copy files from my locally hosted Nextcloud to an External S3 share. Our small office employees are using our local Nextcloud (hosted on Nethserver) for all their document storage and editing tasks. I’ve recently enabled external storage to S3 and I’d like to put in place a daily sync of all employees folders/files to this external S3 storage. I’m the Admin so I have the access to all folders and files. I was looking for a scheduled automated way to sync to this S3. I’ve looked at flow but can’t see the option there. I’m not aware of any Apps that can assist. Searching forums I see people using rsync with a cronjob to sync to S3. Does Nethserver have a way or app I could use to assist with ensuring our locally hosted folders and files are saved to external S3?

Any ideas or options available that I’ve missed?

Thank you in advance for any information you can provide me.

Nethserver backup is able to save to S3 but it’s not a real file sync.

I think that’s the way to go.

Thank you @mrmarkuz for this reply. I will look into testing out rsync with cron.

I’ve noticed that Nethserver has a backup app as well and I see mention of restic? Since I use Nextcloud on my Nethserver, couldn’t the nethserver using restic backup to S3 for me where I can select my Nextcloud folders (and perhaps even my Nethserve File share folders) to send data (deup) to my S3?

Thoughts?

As far as I understand it you can do a Nethserver full backup with restic to s3. I never tested with S3 but it should work.

I’m looking to backup my Nethserver Shared folders and my Nextcloud data folders for each user. Being able to have multiple versions of these backups would be valuable. Perhaps I’ll need to use restic or Duplicity. I’m trying to test setting up a backup scheduled task in Nethserver (cockpit) GUI. I know my S3 storage connection and have confirmed this in logging into S3 using Cyberduck. But when I use these same connection settings in Nethserver GUI to create a backup task my Check keeps failing with configuration errors. There is no other message to provide what the error is. Is there a log file I can read to help determine why Nethserver can’t connect to my providers S3 storage?

Thank you.

Did you go through these steps?

http://docs.nethserver.org/projects/nethserver-devel/en/latest/nethserver-backup-data.html#s3

https://restic.readthedocs.io/en/stable/080_examples.html

Should be logged to /var/log/messages:

http://docs.nethserver.org/projects/nethserver-devel/en/latest/nethserver-backup-data.html#logs-and-wrapper

Thank you @mrmarkuz for you continued assistance!

I’m tailing /var/log/messages and I’m using the ‘Check’ button on the S3 configuration (we don’t use Amazon S3 but my provider has given me my Access Key and Secrety Key and I’ve confirmed my access using Cyberduck) and when I press on Check the only message I see in the messages log is:

Mar 20 12:19:36 swasmainserver1 cockpit-bridge: No entry for terminal type “unknown”;
Mar 20 12:19:36 swasmainserver1 cockpit-bridge: using dumb terminal settings.

Is there another log to look for when setting up backups to S3 that can provide for me a reason why the check is failing?

Thank you.

Hello @mrmarkuz, My S3 provider uses path style and not subdomain style for connecting. What style does Nethsever require?

I’ve also successfully connected my Nextcloud (running on Nethserver) to my S3 provider using path style. In fact Nextcloud allows me the option to use either path style OR subdomain when connecting to S3.

Is Nethserver locked to only using Amazon S3? My S3 provider is not Amazon.

Thank you.

I don’t know, I assume both should work. Which S3 provider do you use?
Unfortunately I have no S3 to try. Is there a good free (without need for credit card data) S3 service?
If it’s ok for you, PM me your S3 login so I can try and maybe find out what’s going on.

From the docs it should be compatible with others:

Amazon S3 (https://aws.amazon.com/s3/) compatible (like https://www.minio.io/).

EDIT:

We managed using the S3 server via CLI and that worked. So the cockpit check seems to have some problems with S3.

http://docs.nethserver.org/projects/nethserver-devel/en/latest/nethserver-backup-data.html#s3

We got the same error with working credentials:

1 Like

Thanks to @mrmarkuz testing the connection, it seems from the command line a Restic backup can be made to my providers S3 storage. Seems there may be an issue in the GUI that doesn’t allow nethserver to connect to my S3. Hopefully this issue can be resolved.

Now that I’ve taken a look at Nethserver backup and tested it I can see what it does. I have a couple of questions/comments to make:

Nethserver backup is perfect for backing up important data from Nethserver to my S3 as encrypted files. What I was hoping to see though was to be able to see my folders and files by version on my S3 and restore which files I want. For example, an employee deletes a file by mistake from their nethserver file share. I would like to view the files backed up and restore the one file. In its current state it appears nethserver backup and restore is a full restore of everything. Is what I’m looking to do currently possible or could this be a feature request? It appears that almost everything is already in place on Nethserver backup. We just need to have the selective restore with a backup browser system added to give us full remote backup with versioning with file restore ability.

Thank you.

1 Like

I’m wondering if the problem in the GUI is because my S3 provider uses path style and not subdomain style for connecting. Perhaps @support_team could confirm what S3 connection nethserver uses to S3. If only subdomain is allowed could nethserver add in the ability for path style? I see that Nextcloud allows both path style and subdomain to connect to S3 external storage.

Hello Nethserver Support Team @support_team, would you be able to confirm what would cause the issue that Markus found when trying to connect to our S3 provider using the Nethserver Backup GUI? If you need any help from me or a test access key please let me know.

Thank you.

I finally found the code snippet:

In /usr/libexec/nethserver/api/system-backup/check-s3 the validation is defined.
The check uploads a file and deletes it afterwards. Both is done with curl.

Upload line:

curl -s -X PUT -T "${tmp_file}" -H "Host: ${BUCKET}.${HOST}" -H "Date: ${date}" -H "Content-Type: ${content_type}" -H "Authorization: AWS ${ACCESS_KEY}:${put_signature}" "https://${BUCKET}.${HOST}/${file_to_upload}"

Delete:

curl -s --fail -X DELETE -H "Host: ${BUCKET}.${HOST}" -H "Date: ${date}" -H "Authorization: AWS ${ACCESS_KEY}:${delete_signature}" "https://${BUCKET}.${HOST}/${file_to_upload}"

If you like you can PM me a test access to your S3 so I can check…

1 Like

Thank you @mrmarkuz, I will PM you with login details to my providers S3 test account.

Thanks!

Hello @mrmarkuz and Nethserver forum,

I’ve provided Markus with a test S3 account from my S3 provider. Markus confirmed that Nethserver uses ‘subdomain style’ - https://{BUCKET}.{HOST}/${file_to_upload} connect to S3. My S3 provider uses ‘Path Style’ for S3 connections so this is the reason that Nethserver backup GUI cannot connect to my S3.

Nextcloud allows for both types of S3 connections so my hope is that Nethserver is able to update the backup GUI to also allow for both S3 connection types.

Many thanks to Markus for his continued support and diligence in confirming root cause.

1 Like

Just to recap, it seems some S3 providers use different https hosts like https://host instead of https://bucket.host (Amazon style). Nethserver-backup assumes Amazon style but the certificate for https://bucket.host is not valid and therefore the check when setting up S3 backup with non-amazon providers fails.

A S3 backup set on command line works without problems:

The path style is reflected for example in Nextcloud too.

From the Nextcloud docs

Enable path style is usually not required (and is, in fact, incompatible with newer Amazon datacenters), but can be used with non-Amazon servers where the DNS infrastructure cannot be controlled. Ordinarily, requests will be made with http://bucket.hostname.domain/ , but with path style enabled, requests are made with http://hostname.domain/bucket instead.

My proposal:
Add the other path style to the S3 check, like check both https://${BUCKET}.${HOST}/${file_to_upload} and `“https://{HOST}/{file_to_upload}” ways and report good if one test succeeds.

Lazy method:
Add -k to the curl commands in /usr/libexec/nethserver/api/system-backup/check-s3 to surpress cert check.

@giacomo What do you think? Should I open an issue/PR?

3 Likes

Yes thank you! I thought the host name format was part of S3 protocol :slight_smile:

2 Likes

The PR is not necessary. The S3 providers SSL cert is now valid and therefore the check is working correctly.

2 Likes

Hello @mrmarkuz and @giacomo,

This is correct. My S3 provider has figured out how to have both subdomain and path style enabled on their S3 storage service. So while Markus and I were testing we discovered this change and now Nethserver works with my S3 provider.

Really appreciate your help with all my recent questions! During my forced quarantine/seclusion of Covid-19 I’m at home a lot more and I’m digging into how to make Nethserver more useful for our office!

Thanks!

2 Likes