Sogo contacts lost after data-restore!

rowihei · October 29, 2016, 9:28am

NethServer Version: 6.8
Module: sogo + restore data

Good morning,
yesterday in the afternoon I have to change the 2xhdd > 2xssd (software-mirror)

made an backup-data
changed hd >< ssd
install nethserver 6.8 from CD to new software-mirror on ssd
config IP + root-password + localhost-name + localhost-domain + backupdata like the old one
restore-config
restore-data

After this it was all OK ! sogo-mails and sogo-contacts all visible. Then manuell start pop3-connector to transfer new emails … 120 mails

After that I start manual as root on console backup-data >>> failed
I tried again some hours later >>> failed
Looking for /var/log/last-backup.log >> not exist !
Now the sogo-contacts are empty, nothing to see … !!!
I think there is a mysql-database corrupt

wich database of mysql can be corrupt ?
How can I restore only this one ?

Thanks for every tip/help !

mark_nl · October 29, 2016, 1:54pm

Don’t know about v6.8, at the moment i’m working on the SOGo package for v7 and noticed there is no nethserver-sogo-restore-user action.

Does your backup set contain a sogo-backup.tgz ?

Is that’s the case we just have to untar it and figure out the right sogo-tool command.

http://wiki.sogo.nu/backupRestore

On 6.8 sogo-tool has to be executed by sogo (user); it is going to be somthing like

su -s '/bin/bash' -c 'sogo-tool restore .........' - sogo

rowihei · October 29, 2016, 2:45pm

Good afternoon,
thanks to mark_nl:
Where can I find sogo-backup.tgz ? make backup inside nethserver with backup-data - works in background with duplicity I think …
My backup files are on NAS in /Public/BAK-MAIL/msrv - looks like million files … ??

mark_nl · October 29, 2016, 3:07pm

In the nethserver data backup from the HDD you transferred to SDD ?

rowihei · October 29, 2016, 3:20pm

I restored via restore-config , then restore-data from NAS >> nethserver, now installed on SSD
See my first text …

mark_nl · October 29, 2016, 3:59pm

So your backup set is on the NAS, the location is where you restored it from. Right?

EDIT:Ah its between a million files…

mark_nl · October 29, 2016, 4:08pm

On the old server the back-up is in

/var/lib/nethserver/backup

Maybe it is restored on the same location on the new install…

Did you log in to SOGo to see if the contact are listed there?
And you are ablosulty sure they where present first, this puzzles me a bit…?

rowihei · October 29, 2016, 9:21pm

good evening,

yes, I stated restore of data at 15:49 … ready at 16:15. then I installed mc+ncdu+htop from epel-repo. After that I started from root-console backup-data >>> error. Second try some hours later same error.
My friend checked in this time emails and adresses. He said at 18:30 his contacts were visible, because he send an email.
I removed nethserver-backup and installed again, after that backup-data >> Success.
Only find /var/lib/nethserver/backup/sogo-backup.tgz 1569 Byte from 12:30, 29.10.2016 - this is time of the last backup, and no contacts visible.
In NAS-backup-folders only files duplicity …bla.bla-bla.volxx.difftar.gz. Inside this gz no sogo-backup.tgz to find.
There are 403 files, most of them 250MB. I’m not able to look in all this, to find sogo-backup.tgz from 27.10. or 28.10. 12:30…
Is there a tool for duplicity-files to find easier, what I’m looking for ??

23:41: find only this
The -t argument will tell duplicity from what time to restore.
duplicity -t 3D --file-to-restore FILENAME scp://uid@other.host/some_dir /home/me/restored_file

so there is hope to find an older sogo-backup.tgz

Good night - till next time
…
Good morning, bad news:

tried this:
duplicity -t 5D restore --file-to-restore sogo-backup.tgz file:///mnt/backup/msrv /root/test

but after a while;
…
Copying duplicity-new-signatures.20161028T115204Z.to.20161029T093149Z.sigtar.gz to local cache.
Copying duplicity-new-signatures.20161029T093149Z.to.20161029T103007Z.sigtar.gz to local cache.
Last full backup date: Fri Oct 28 13:52:04 2016
GnuPG passphrase:
sogo-backup.tgz not found in archive, no files restored.
…

That is it, what I’m afraid - sogo-backup.tgz is not in nethserver-backup !!
Now I have to “roll back” - means I have to change the new SSDs to old HDDs and hope I find on the old HDDs sogo-backup.tgz in folder /var/netserver/backup
Nice sunday to everyone…

mark_nl · October 30, 2016, 8:03am

You cloud try to mount your HDD…

Keep in mind receiving and sending mail are not done by SOGo

Just to be sure we are not barking on the wrong tree, you are able to make new contacts in SOGo?

And I advise to wait if some can confirm the SOGo data is not restored in NS 6.8.
Iv’ been trough the code and looked back in time, and can not find it. (@Stll0, @giacomo)

Good luck!

rowihei · October 30, 2016, 9:39am

good morning again,

my friend send mails only via sogo-webfrontend and used contacts inside …
and yes it’s possible to create new contacts, we tried it…
now I copied the sogo-backup.tgz (144kB) from old HDD to new Nethserver /var/lib/nethserver/sogo20161028
I read restore is possible via terminal with sogo-tool - but there are some problems:
"
[root@msrv tmp]# mkdir test
[root@msrv tmp]# ls
sogo-integrator.05T9MPle sogo-integrator.a_bUMx7i sogo-integrator.IgzfZBM7 sogo-integrator.kek0DNQv sogo-integrator.mkqVBRP0 test
sogo-integrator.5j2Ihyxz sogo-integrator.FRVLkHiW sogo-integrator.j1kzBOrk sogo-integrator.kZETqDqd sogo-integrator.Y71BhKjp
[root@msrv tmp]# sogo-tool restore -l /tmp/test user1
2016-10-30 10:36:02.497 sogo-tool[13287] ERROR(+[GCSFolderManager defaultFolderManager]): default ‘OCSFolderInfoURL’ is not configured.
2016-10-30 10:36:02.510 sogo-tool[13287] specified directory does not exist

"
Howto restore ??
anyone knows a better link as backupRestore - SOGo Wiki ??

mark_nl · October 30, 2016, 9:44am

This is the contend of the un-tarred sogo-backup.tgz ?

rowihei · October 30, 2016, 9:49am

No - content of tmp-folder - I created a folder /tmp/test - checked it … then tried "sogo-tool restore -l …"
but /var/tmp/test is empty, but exists…

Now I extract sogo-backup.tgz to this folder:
[root@msrv var]# ls -lh /var/tmp/test
total 708K
-rw-r–r-- 1 clam cgred 4.3K Oct 28 13:52 admin
-rw-r–r-- 1 clam cgred 375 Oct 28 13:52 anonymous
-rw-r–r-- 1 clam cgred 573K Oct 28 13:52 user1
-rw-r–r-- 1 clam cgred 42K Oct 28 13:52 user2
-rw-r–r-- 1 clam cgred 11K Oct 28 13:52 user3
-rw-r–r-- 1 clam cgred 21K Oct 28 13:52 user4
-rw-r–r-- 1 clam cgred 5.9K Oct 28 13:52 user5
-rw-r–r-- 1 clam cgred 12K Oct 28 13:52 user6
-rw-r–r-- 1 clam cgred 4.3K Oct 28 13:52 user7
-rw-r–r-- 1 clam cgred 4.6K Oct 28 13:52 user8
-rw-r–r-- 1 clam cgred 1.5K Oct 28 13:52 user9

And now ?

mark_nl · October 30, 2016, 9:58am

For every user (if it works, try one first)

su -s '/bin/bash' -c 'sogo-tool restore -p user1' -  sogo
su -s '/bin/bash' -c 'sogo-tool restore -f user1' -  sogo

first cd in to /var/tmp/test

rowihei · October 30, 2016, 10:07am

Must be a missunderstanding …

I work as root (no need su …)
if I copy your commandline - this is the result?:

su -s ‘/bin/bash’ -c ‘sogo-tool restore -p user1’ - sogo
restore [-l|-p|-f/-F folder/ALL|-p] directory user

_ directory the directory where backup files were initially stored_
_ user the user of whom to restore the data_
_ -l flag used to list folders to restore_
_ -p flag used to restore only the user’s preferences_
_ -f/-F flag used to specify which folder to restore, ALL for everything_

Examples: sogo-tool restore -l /tmp/foo bob
_ sogo-tool restore -f Contacts/personal /tmp/foo bob_
_ sogo-tool restore -p /tmp/foo bob_

Now I found a little bit…
must be something like this:
[root@msrv test]# sogo-tool restore -l /var/tmp/test user1
2016-10-30 11:17:46.616 sogo-tool[15153] ERROR(+[GCSFolderManager defaultFolderManager]): default ‘OCSFolderInfoURL’ is not configured.
<0x0x259e678[SOGoUserManager]> No authentication sources defined - nobody will be able to login. Check your defaults.
<0x0x2541a88[SOGoCache]> Cache cleanup interval set every 300.000000 seconds
<0x0x2541a88[SOGoCache]> Using host(s) ‘localhost’ as server(s)
2016-10-30 11:17:46.658 sogo-tool[15153] Restorable folders:
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/1D60-55F18980-4D-725F230 (WΓ¶hner D.)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/295B-55EC7B00-1-41608D00 (Familie)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/3E49-56B8B000-35-415A1A00 (???)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/446D-56AF6D00-37-2865F6C0 (Service)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/5AC-55ED6B00-19-4A95E280 (Bianka)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/6B52-56B87700-1-627B6180 (??? Susanne)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/6B52-56B8AD80-39-627B6180 (??? Florian)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/6B52-56B8B180-57-627B6180 (??? JΓΌrgen)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/73F8-55F19780-49-3C5C9840 (??? JΓΌrgen)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/7439-5720BA00-29-36F9C500 (??? Sylvio)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/7B21-56B0CB80-103-2348C180 (??? Oliver)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/DE3-55F19900-73-43C4F300 (??? Robin)
2016-10-30 11:17:46.659 sogo-tool[15153] Calendar/personal (PersΓ¶nlicher Kalender)
2016-10-30 11:17:46.659 sogo-tool[15153] Contacts/personal (PersΓ¶nliches Adressbuch)
[root@msrv test]#

I substituded names with ???

[root@msrv test]# sogo-tool restore -p /var/tmp/test user1
2016-10-30 11:23:22.642 sogo-tool[15273] ERROR(+[GCSFolderManager defaultFolderManager]): default ‘OCSFolderInfoURL’ is not configured.
<0x0x1d3d678[SOGoUserManager]> No authentication sources defined - nobody will be able to login. Check your defaults.
<0x0x1ce0a88[SOGoCache]> Cache cleanup interval set every 300.000000 seconds
<0x0x1ce0a88[SOGoCache]> Using host(s) ‘localhost’ as server(s)
<0x0x1f300c8[SOGoSQLUserProfile]> failed to acquire channel for URL: (null)
<0x0x1f74008[SOGoSQLUserProfile]> failed to acquire channel for URL: (null)

Looks like FAILED ??? what to do ??

mark_nl · October 30, 2016, 10:28am

Yes on 6.8 you do EDIT you have to execute sogo-tool as user sogo, sogo can’t login.

we are coming close let me think about it

try it with full path

su -s ‘/bin/bash’ -c ‘sogo-tool restore -p /var/tmp/test user1’ - sogo
su -s ‘/bin/bash’ -c ‘sogo-tool restore -f ALL /var/tmp/test user1’ - sogo

EDIT see below

rowihei · October 30, 2016, 10:40am

AND NOW USER1 SEE ALL his old CONTACTS + the new test-contact

Thank you very much - I learned a lot …
I would say, it’s important to include sogo-backup.tgz in nethserver-backup data

Have I nice weekend - like I NOW

mark_nl · October 30, 2016, 10:41am

Thank you too, I learned a lot for SOGo on ns7

Nice Weekend

alefattorini · November 8, 2016, 3:51pm

If I remember correctly, It’s already included. At least in the nethserver-sogo for 6.8
@mark_nl could you check please?

rowihei · November 8, 2016, 4:53pm

you are right - I tried to restore sogo-backup.tgz via duplicity directly. But without correct passphrase there is only the same message: …" sogo-backuo.tgz could not be found …" or so.
Now I read the complete wiki backup/restore and found the correct commandline:

restore-file -t 10D /test “/var/lib/nethserver/backup/sogo-backup.tgz”

It is not possible to restore via duplicity directly - better to say, not needed.
Excuse me, typical error number 1: I not read this manual … many times lost, because I was excited to lost contacts …

It’s all over - contacts are alright now … but I don’t understand how we lost it during restore-data.

Thanks to all, especially to mark_nl

mark_nl · November 11, 2016, 12:39pm

I’m pretty sure nethserver-sogo has a script and hooks for backing up the (mysql) user data.
AFAIK a restore script (and hooks) is missing.