Migrating Nextcloud data?

v7
nextcloud
migration
openldap

(Dan) #1

NethServer Version: 7.4.1708
Module: N/A

As I mentioned yesterday, I’m considering migrating from SME Server to NethServer. I’ve installed Nextcloud 12.0.3 manually on my SME server, and it’s running fine there, with about 130 GB of data. Installing Nextcloud in Neth is a simple matter of point and click, but I’m concerned about migrating the data so that (1) all the data transfers over, (2) all the data belongs to the right users, and (3) though not as important as the first two, all the sharing permissions are correct. I see that nextcloud.com has some migration instructions, but I’m concerned about the users. I’m using local LDAP authentication for Nextcloud on my SME server, and plan to do the same on Neth. But that seems to turn user IDs from simple usernames like admin into long hex strings like 740b50cc-4fb4-1037-814c-6dbb6d41fba0. Presuming that the long hex strings are what’s stored in the Nextcloud database, this sounds like a problem. Any thoughts or pointers?


phpMyAdmin - possible to find/replace throughout database?
(André Wismer) #2

Hi

Also a longtime SME user…

AFAIK: The long Hexstrings came with an update of owncloud, and was taken over in NextCloud.
Just to make sure you don’t have two users Joe Smith…

Not very elegant, but it actually works fairly well. The only real PITA (Pain in the A… - Not the front!) is entering the connecton data on a mobile device like Android or iOS… I send the User a mail with the appropriate link(s) to the calender / contacts app. Copy / Pasting works better from Mail than from the browser, which tends to open up that link…

You did see this?:

http://docs.nethserver.org/projects/nethserver-devel/en/v7/migration.html

NextCloud will need a bit of manual handling - but all in all, it’s worth it.
Less reboots for updates, latest libraries and such, full AD and lots of features the good ol SME just couldn’t keep up with.

My 2 cents
Andy


(Davide Principi) #3

There’s also a migration script that could be tweaked for your case

http://docs.nethserver.org/en/v7/upgrade.html#owncloud-and-nextcloud

Refer also to

http://docs.nethserver.org/en/v7/migration.html


(Dan) #4

Yes, I did, thanks–haven’t yet looked through it in detail, but I’d seen it was there, and will be testing that procedure out (hopefully) thoroughly before I migrate my production system.

I’d figured they were related to using LDAP authentication. I’d previously had an installation of Owncloud on the SME server, set up to do its own authentication. When it was configured that way, there were data directories with each user’s username–data/fred/, data/wilma/, etc. Wanting to link it to the SME accounts database, I reconfigured it to use LDAP authentication; at that time, the directories became data/0e12fec8-1bf2-1037-93df-cbaeb7430477 and such.

What’s more, the directory names are not determined (at least solely) by the username. I have a user “dan” on both my SME server and my test Neth install. On the SME server, dan’s Nextcloud data folder is 878bc1ca-05e0-1034-9e9f-4fbf87d6134f. On Neth, it’s 740b50cc-4fb4-1037-814c-6dbb6d41fba0. Even that wouldn’t be insurmountable; I don’t have that many users on the SME server.

But the database is a real concern. There are about 130k rows among the various tables in the Nextcloud database, and most of them have at least one of these long hex UIDs inside. I’m sure that could be fixed, but it’s sounding at least a little scary.


(Dan) #5

Ah, I hadn’t seen that yet–thanks.


(André Wismer) #6

@danb35

There’s also https://wiki.nethserver.org/doku.php?id=phpmyadmin

Actually there’s already quite a few plugins in NethServer:

https://wiki.nethserver.org/doku.php?id=modules

Besides the ones already available with a click in the Web-GUI…

And, coming from SME, “yum install …” is not very far away for your favorite tools.

Mine are:

yum install screen htop nano mc (For almost all UNIXes…)

My 2 cents
Andy


(Dan) #7

Found that list already, and would probably be using a number of those. Surprised that nano wasn’t installed by default, though.


(André Wismer) #8

I think nano is actually per default included.
I’m not sure as I “sledgehammer” my favorite tools in - not bothering to check if thy’re there before - as soon as one isn’t there when I call it.

:wink:

Andy


(Dan) #9

Nope, just had to install it manually. yum install nano is easy enough, though, as you point out.


(Stéphane de Labrusse) #10

Ns is modular and want to be a clone of centos, it doesn’t come with a bunch of application. You need one then install it.

With centos, vi is your friend but i prefer to use vim…a matter of choice


(André Wismer) #11

@stephdl

Editor of choice…

I used to advocate usage of vi / vim, whatever available on the platform. After all, it was more or less simple and available on all UN*Xes, available as open source. And shortcut commandos were cool…

Eight megabyte and constantly swapping (EMACS) was never an option for me.

I always thought vi is vi, no matter what platform…

And platforms were there: Linux, OpenBSD, FreeBSD, NetBSD, Solaris (Slow war es) MacOSX, QNX and other exots…

The day I discovered that vi uses different shortcuts on different platforms (Solaris was the eye opener for me) - that’s the day vi died for me. I dabbled around with mc’s editor, joe and pico but my choice of editors is presently nano!

My 2 cents
Andy


(Rob Bosch) #12

As I come from Ubuntu (where it is installed by default) first thing after installing NS is do yum install nano… I find it a VERY easy to use and the easiest to get used to of all commandline editors I have seen.


(Dan) #13

So now having done a bit of testing on the Nextcloud data migration, here are my notes FWIW. I haven’t found any specific instructions on doing this, so the odds are good I’m missing something.

My intent is to migrate from my SME Server. I installed Owncloud 7, IIRC, on that machine, upgraded through the latest version before the Nextcloud fork, migrated to Nextcloud, and have upgraded that progressively up to the latest version. No doubt there’s a lot of legacy cruft in there. That also means that it’s still running on a database named owncloud. I’m not testing using my main machine (that would be insanity), but my testing environment is trying to somewhat duplicate this.

So, I set up two test VMs. One running SME 9.2, one running Neth 7.4. On the SME VM, I created a half-dozen users, installed Nextcloud 12.0.3 (using a database named owncloud–which will turn out to be very helpful), and uploaded some data to those users’ directories. Also on that server, I logged into Nextcloud as the admin user, and went to the Users page–that lists all the usernames along with their corresponding UUIDs. Those got copied into a spreadsheet.

On the Neth VM, I set it up as its own domain controller (something I hadn’t realized when I started this thread that I would need to do if I wanted to authenticate SMB access), and installed most of the packages I wanted from the software center–backup, email, file server, mysql server, web server, etc. I did not install Roundcube or Nextcloud at this time, because their database users get clobbered in the migration from SME.

I migrated the data from the SME server using the rsync-migrate script. Once that finished, I installed Nextcloud and Roundcube from the software center. Also installed phpMyAdmin and mc.

I logged into Nextcloud on the Neth VM using admin/Nethesis,1234, went to the Users page, and got the new list of usernames and UUIDs. Copied those to the spreadsheet too.

Now all the pieces were in place. Using mc, I moved the data directories from /var/lib/nethserver/ibay/owncloud/html/data to /var/lib/nethserver/nextcloud, renaming them from the old UUIDs to the new ones in the process. Then fired up phpMyAdmin.

In the owncloud database (i.e., the old one), I went into the oc_activity table. One at a time, I replaced the old UUIDs with the new ones on both the user column and the affecteduser column. The oc_files_trash table was empty; if not, I should have done the same there. I then copied the oc_activity and oc_filecache tables into the nextcloud database, replacing the existing tables there.

The result is that I’m able to log into Nextcloud on the Neth VM as any of the users I set up on the SME VM, and the files I uploaded to those users’ Nextcloud folders on the SME VM are now available through Nextcloud on the Neth VM. I’m provisionally considering this successful.

What I’m not sure of is whether I should be copying data in any other tables into the nextcloud database. Thus far, my inquires on the Nextcloud forums have not gotten any answers.

Edit: Well, I think I’ve found one other table. In the oc_share table, the same find/replace would be needed in uid_owner and share_with. Wasn’t an issue on my test install as I hadn’t shared anything.

Also in oc_share_external, the user column should be updated. Looks like the same is needed for the oc_uid column in both oc_storagecharts2 and oc_storagecharts2_uconf.