Nethserver Backup - Help me understand it better

greavette · March 27, 2020, 2:33am

NethServer Version: NethServer release 7.7.1908 (final)
Backup and Restore

Hello Team,

I’d like to better understand the Backup/Restore module in Nethserver.

First Question: When I create a new Backup I see what looks like an option that should let me select the type of backup I want:

restic
duplicity
rsync

But when I click on resitc like in the picture I’ve attached I’m not provided an option to change the type of backup I want to use? Why is there a hand to choose?

Second question I have is related to the other locations I can choose to be backed up:

This is great! Now I can include locations from my Nethserver and/or shares from other servers. But when I select Restore…what exactly will happen to these Custom Included paths during the restore? Will everything in that Custom Included Path get restored?

Third Question: What I’m really looking for (since Nethserver has restic already installed) is the ability to use the restore feature of Nethserver to restore what I want and where I want to restore it. I don’t want to restore my entire Nextcloud folders/files. For example…an employee needs to recover a file that Nethserver backed up for us last week. I want to be able to use the Nethserver Restore to choose that one file and restore it for this employee. Can Nethserver backup/restore be used like this?

Thank you for helping me better understand how to use this module.

mrmarkuz · March 27, 2020, 4:46am

The engine to choose depends on the storage backend you selected a step before.

I assume you chose a cloud storage like S3 as storage backend and in this case you only get the engine restic.

If you choose CIFS as storage backend for example, you get Duplicity and Restic as possible engine.

https://docs.nethserver.org/en/latest/backup.html#settings

Yes, everything in custom included paths will be restored.

Yes, that’s possible.

For web UI:

https://docs.nethserver.org/en/latest/backup.html#selective-restore-of-files

Command line:

http://docs.nethserver.org/projects/nethserver-devel/en/latest/nethserver-backup-data.html#restore-from-command-line

greavette · March 28, 2020, 6:01am

Thank you @mrmarkuz!

I’ve tested the backup to my providers S3 and my backup to S3 from Nethserver is working well.

I had a test document file on my Nextcloud. I used the Restore in Nethserver and was able to search for the file. Nethserver Restore looked in my backup and found the file. I restored it first without overwriting. The restore said it was successful but on my Nextcloud the restored file with the new name did not appear. I checked through the terminal and also did not find the restored file there.
So I did another test and this time I deleted my test file from Nextcloud. I restored the file and again did not choose overwrite (since there is nothing there to overwrite…I deleted it). The restore of the file said it completed successfully but the file never showed up in my Nextcloud and was not in the folder location Nethserver restore said it should be.
So I tried a third time and this time I used the overwrite checkbox. Nethserver restore said it was successful and this time through the terminal I could see my restored file. BUT my nextcloud did not show this file?

Am I restoring files wrong in Nethserver restore?

Thanks!

greavette · March 29, 2020, 5:16am

I think I may know the problem here with restoring my Nextcloud files using Nethserver restore for selective files. My testing is showing that nextcloud files are being restored to their correct locations when I navigate through the terminal. But Nextcloud can’t see these new files. I think that Nextcloud database isn’t aware of these new files and some kind of ‘sync’ needs to occur in Nextcloud to see these files. Perhaps that someone from Nethserver can confirm this for me.

if this is true then the Nethserver selective file restore needs to have a sync command added to it so that the Nextcloud database will see the new restored file(s).

What do you think?

dnutan · March 29, 2020, 11:27am

If permissions and ownership of restored files are correct, since Nextcloud 14.0.3 NethServer ships a cron job for Nextcloud that, between other tasks, will rescan for new files.
If you want to do it manually you can use the occ files:scan command.

danb35 · March 29, 2020, 11:40am

I don’t think so. The cron task handles general system maintenance functions, but I don’t believe it includes a file scan–the file scan wouldn’t be appropriate with the way Nextcloud is designed to be used in Neth.

greavette · March 29, 2020, 2:10pm

Thank you @danb35 and @dnutan for your input on my post.

So is this “sync” of files in Nextcloud missing during the file restore? Could the Nethserver file restore be updated to include a sync command so that I can see my restored file(s) in Nextcloud?

Thank you,

danb35 · March 29, 2020, 5:39pm

I wouldn’t think that would be appropriate–many installations wouldn’t have Nextcloud at all, and the failure of that script to run would likely be fatal to the restore process.

greavette · March 29, 2020, 6:22pm

Thank you @danb35, you make some good points. But I’m hoping my discussion will find a solution.

Am I correct ie: have others tested the restore of a file from Nextcloud and found it not visible in Nextcloud upon restore?
Is there a command I could run manually on Nethserver that would allow my Nextcloud data files to re-sync so my restored file(s) are available?
I’m not seeing any special instructions on the Nethserver documentation for restore so perhaps @support_team could weigh in and let me know if they are already aware of what I found or perhaps correct me so that if I’m doing this wrong I can restore files correctly.

While it may not be the correct course of action to automatically add a sync job to run when doing a restore…perhaps Nethserver team can know when the restore is from Nextcloud and only run the script. Or even if there is manual step that we can take to run a sync job manually, documenting that would be valuable for all of us to know.

Thank you.

danb35 · March 29, 2020, 6:26pm

Yes, the command @dnutan gave in his post this morning, with a link to the Nextcloud docs explaining it. Though to properly run it in the Neth environment, see howto:nextcloud [NethServer Wiki]

mrmarkuz · March 29, 2020, 10:31pm

I never tested but you may try to add the “files:scan” line to the /sbin/e-smith/restore-file script.

The end of the file should look like this:

if ($ret != 0) {
    exit 1;
} else {
    system("sudo -u apache scl enable rh-php73 -- php -dmemory_limit=512M /usr/share/nextcloud/occ files:scan --all");
    exit 0;
}

greavette · March 30, 2020, 12:49am

Hello @danb35 and @dnutan and @mrmarkuz , I’ve tried to issue the following commands but seem to be doing something wrong. I found where occ is on Nethserver. Here is what I’m running and the error I see when I run it:

[root@nethserver share]# /usr/share/nextcloud/occ php files:scan --all

PHP Parse error: syntax error, unexpected ‘class’ (T_CLASS), expecting identifier (T_STRING) or variable (T_VARIABLE) or ‘{’ or ‘$’ in /usr/share/nextcloud/console.php on line 96

Can you see where I’m going wrong to issue the files:scan on Nethserver?

danb35 · March 30, 2020, 12:50am

From the wiki page I linked above, you’d invoke the occ command with:

sudo -u apache scl enable rh-php72 -- php -dmemory_limit=512M /usr/share/nextcloud/occ (command/parameters)

mrmarkuz · March 30, 2020, 12:52am

@greavette I already posted it.

sudo -u apache scl enable rh-php73 -- php -dmemory_limit=512M /usr/share/nextcloud/occ files:scan --all

EDIT:

I had an error in the scan line in the script but it’s fixed now and it works.
Backuped my Neth with Nextcloud. Deleted a file.
Restored from backup and the file was back in Nextcloud.
You have to do an overwrite else a hidden file is restored that seems to be not scanned by Nextcloud.

greavette · March 30, 2020, 5:04am

Thanks Team for these replies. not sure how I missed these commands.

I wasn’t sure which version of PHP I have installed (I installed this nethserver new 3 days ago to test backup and Talk) so I checked to see what version(s) of php I have installed and it seems I have only 7.2:

[root@nethserver share]# rpm -qa php7
rh-php72-php-gd-7.2.24-1.el7.x86_64
rh-php72-php-bcmath-7.2.24-1.el7.x86_64
rh-php72-php-soap-7.2.24-1.el7.x86_64
rh-php72-php-pdo-7.2.24-1.el7.x86_64
rh-php72-php-pgsql-7.2.24-1.el7.x86_64
rh-php72-1-2.el7.x86_64
rh-php72-php-json-7.2.24-1.el7.x86_64
rh-php72-php-mysqlnd-7.2.24-1.el7.x86_64
rh-php72-php-mbstring-7.2.24-1.el7.x86_64
sclo-php72-php-tidy-7.2.24-1.el7.x86_64
rh-php72-php-enchant-7.2.24-1.el7.x86_64
sclo-php72-php-imap-7.2.24-1.el7.x86_64
nethserver-rh-php72-php-fpm-1.1.1-1.ns7.noarch
rh-php72-php-common-7.2.24-1.el7.x86_64
rh-php72-php-fpm-7.2.24-1.el7.x86_64
rh-php72-php-cli-7.2.24-1.el7.x86_64
rh-php72-php-pecl-apcu-5.1.12-1.el7.x86_64
rh-php72-php-xml-7.2.24-1.el7.x86_64
sclo-php72-php-smbclient-1.0.0-1.el7.x86_64
rh-php72-php-zip-7.2.24-1.el7.x86_64
rh-php72-php-ldap-7.2.24-1.el7.x86_64
rh-php72-php-pear-1.10.5-1.el7.noarch
rh-php72-runtime-1-2.el7.x86_64
rh-php72-php-intl-7.2.24-1.el7.x86_64
rh-php72-php-process-7.2.24-1.el7.x86_64
rh-php72-php-opcache-7.2.24-1.el7.x86_64
sclo-php72-php-pecl-imagick-3.4.4-1.el7.x86_64

The following command did the trick and ran successfully:

[root@nethserver share]# sudo -u apache scl enable rh-php72 – php -dmemory_limit=512M /usr/share/nextcloud/occ files:scan --all
Starting scan for user 1 out of 5 (A165EA09-9DA5-4606-9300-4EDDD4443627)
Starting scan for user 2 out of 5 (admin)
Starting scan for user 3 out of 5 (BCED6330-8C3D-4DE7-97BD-20A435A17271)
Starting scan for user 4 out of 5 (CEE99C82-6D68-4ECB-8E83-2968EA586005)
Starting scan for user 5 out of 5 (D0F46740-8392-4E7E-A9BF-E0BD6FEFE885)
±--------±------±-------------+
| Folders | Files | Elapsed time |
±--------±------±-------------+
| 23 | 81 | 00:00:02 |
±--------±------±-------------+

And now all my restored test files I’ve been restoring through Nethserver GUI are in my Nextcloud once again! Nice job Team!

So I’ve continued my testing now…

I’ve taken the suggestion from @mrmarkuz to modify the /sbin/e-smith/restore-file script and added to the end of it the code you’ve suggested.

Next I created test files on my Nextcloud and added text to these files. I backed up my Nethserver to S3 using Nethserver backup. I then deleted some test files and began restoring though Nethserver (using overwrite and not-overwrite options). The files are now showing up in my Nextcloud but what’s curious is that the text I had in each file are not there? In fact all restored files are empty.

I think I’ll remove the changes I made to /sbin/e-smith/restore-file script and issue the manual command for now. It’s not a common task (I hope!) to have to restore files that are deleted/corrupted so issuing the manual command isn’t hard. I’m curios though if Nethserver Team will add a permanent change to the restore process to issue this files:scan command IF the restore includes Nextcloud?

I’ll keep you posted on my testing. Thanks!

giacomo · March 30, 2020, 7:24am

I do not know.
To obtain such thing, we should add an hook at the end of restore-file. After the restore of files, the command could execute all scripts from a well-known directory.
Then, all applications should add their own hook if a special actions is needed.

AFAIK Nextcloud is the only application which could benefit from such implementation.