Upgrade to AD after update from NS6 to NS7.4 failed

flatspin · September 29, 2017, 2:05pm

NethServer Version: NS 7.4
Module: Account Provider

Hi friends,

I tried to upgrade a existing NS 6 to NS 7.4 with rsync method. That worked perfectly!
After that I had LDAP account provider and all my users, groups etc.

Than I wanted to upgrade to AD

but that ended up with this failure:

Also the IP of the green LAN wasn’t in database, so i had to do db networks setprop br0 ipaddr 192.168.0.xxx and signal-event update-interface.

But I didn’t get the sssd to work and the admin interface wasn’t reachable also.

Tried it twice, but got two time the same mess. => Is there a Bug in the upgrade script?

TIA Ralf

EDIT: I found that on the ported machine the home directories dont have a user or group. They are reported with unknown user and unknown group. Ldap users are present.

davidep · October 1, 2017, 7:57pm

Hi @flatspin thank you for sharing this!

We should try to reproduce it to see if it’s a bug or not /cc @quality_team

Could you attach the nsdc journal? Please upload it (on gist.github.com for instance)

journalctl -M nsdc >nsdc.log

davidep · October 2, 2017, 10:36am

I couldn’t reproduce the problem on my VM (but could be my VM that is not on a clean state).

Please share also the content of /var/log/messages!

BTW I could have found another problem during the upgrade

In /var/log/messages:

Oct  2 12:12:34 nethservice esmith::event[2858]: Action: /etc/e-smith/events/nethserver-samba-update/S30nethserver-samba-libwbclient FAILED: 2 [0.08398]

[root@nethservice ~]# bash -x /etc/e-smith/events/nethserver-samba-update/S30nethserver-samba-libwbclient ev
+ set -e
+ event=ev
++ /sbin/e-smith/config getprop smb Libwbclient
+ prop=samba
++ alternatives --list
++ grep -o -P '^libwbclient\.so\.[^\t]+'
+ target='libwbclient.so.0.12-64
libwbclient.so.0.13-64'
+ [[ -z libwbclient.so.0.12-64
libwbclient.so.0.13-64 ]]
+ [[ samba == \s\s\s\d ]]
+ [[ samba == \s\a\m\b\a ]]
++ alternatives --display libwbclient.so.0.12-64 libwbclient.so.0.13-64
++ cut -f 1 -d ' '
++ grep '^/usr'
++ grep -F /samba/
+ option_winbind=
+ alternatives --set 'libwbclient.so.0.12-64
libwbclient.so.0.13-64' ''

…it seems alternatives detects multiple versions of libwbclient

flatspin · October 2, 2017, 10:56am

Hi Davide,

sorry for late response. I did a roll back and try it again.
At this pouit the account provider shows this:

ldap is working and every thing seems to work, exept the shared folders. They all have “unknown user”.

What I did:

installed a fresh VM with NS 7.4 b1 iso. Did all updates and than did the rsync.
After that finished I did the rsync -u and shut down the old (physical) machine.
Now I will do the upgrade to AD again.

flatspin · October 2, 2017, 11:56am

2nd try, same result.

Here is [quote=“davidep, post:2, topic:7938”]
journalctl -M nsdc >nsdc.log
[/quote]

-- Logs begin at Mon 2017-10-02 12:59:07 CEST, end at Mon 2017-10-02 13:22:46 CEST. --
Oct 02 12:59:07 nsdc-nethserver.ad.jeckel.local systemd-journal[13]: Runtime journal is using 8.0M (max allowed 391.0M, trying to leave 586.6M free of 3.8G available → current limit 391.0M).
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd-journal[13]: Permanent journal is using 8.0M (max allowed 4.0G, trying to leave 4.0G free of 650.7G available → current limit 4.0G).
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd-journal[13]: Time spent on flushing to /var is 1.824ms for 2 entries.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd-journal[13]: Journal started
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Flush Journal to Persistent Storage...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Flush Journal to Persistent Storage.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Create Volatile Files and Directories...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Create Volatile Files and Directories.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Update UTMP about System Boot/Shutdown.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Reached target System Initialization.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting System Initialization.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Daily Cleanup of Temporary Directories.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Daily Cleanup of Temporary Directories.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Reached target Timers.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Timers.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Listening on D-Bus System Message Bus Socket.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting D-Bus System Message Bus Socket.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Reached target Sockets.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Sockets.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Reached target Basic System.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Basic System.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local ntpd[22]: ntpd 4.2.6p5@1.2349-o Wed Apr 12 21:24:06 UTC 2017 (1)
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Network Time Service...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Domain controller provisioning...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Permit User Sessions...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local ntpd[26]: proto: precision = 0.106 usec
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started D-Bus System Message Bus.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local ntpd[26]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local ntpd[26]: MS-SNTP signd operations currently block ntpd degrading service to all clients.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting D-Bus System Message Bus...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Network Service...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Login Service...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Network Time Service.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Permit User Sessions.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Cleanup of Temporary Directories...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Console Getty.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Console Getty...
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Reached target Login Prompts.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Login Prompts.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd-networkd[27]: host0           : Cannot configure IPv4 forwarding for interface host0: Read-only file system
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd-networkd[27]: host0           : Cannot configure IPv6 forwarding for interface: Read-only file system
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd-logind[28]: New seat seat0.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Login Service.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd-networkd[27]: Enumeration completed
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Network Service.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd-networkd[27]: host0           : link configured
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Reached target Network.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Network.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd-networkd[27]: host0           : gained carrier
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Cleanup of Temporary Directories.
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local samba-tool[23]: Reading smb.conf
Oct 02 12:59:08 nsdc-nethserver.ad.jeckel.local samba-tool[23]: Provisioning
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: failed to bind to server ldap://192.168.0.235 with dn="cn=samba,dc=directory,dc=nh" Error: Can't contact LDAP server
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: (unknown)
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: pdb_init_ldapsam: WARNING: Could not get domain info, nor add one to the domain. We cannot work reliably without it.
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: pdb backend ldapsam:ldap://192.168.0.235 did not correctly init (error was NT_STATUS_CANT_ACCESS_DOMAIN_INFO)
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: ERROR(<class 'passdb.error'>): uncaught exception - Cannot load backend methods for 'ldapsam:ldap://192.168.0.235' backend (-1073741606,Configuration information could not be read from the domain controller, either because the machine is unavailable or access has been denied.)
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: File "/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py", line 176, in _run
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: return self.run(*args, **kwargs)
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: File "/usr/lib64/python2.7/site-packages/samba/netcmd/domain.py", line 1584, in run
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: useeadb=eadb, dns_backend=dns_backend, use_ntvfs=use_ntvfs)
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: File "/usr/lib64/python2.7/site-packages/samba/upgrade.py", line 485, in upgrade_from_samba3
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: s3db = samba3.get_sam_db()
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: File "/usr/lib64/python2.7/site-packages/samba/samba3/__init__.py", line 390, in get_sam_db
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local samba-tool[23]: return passdb.PDB(self.lp.get('passdb backend'))
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: samba-provision.service: main process exited, code=exited, status=255/n/a
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: Failed to start Domain controller provisioning.
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: Unit samba-provision.service entered failed state.
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: samba-provision.service failed.
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: Reached target Multi-User System.
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Multi-User System.
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: Reached target Graphical Interface.
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Graphical Interface.
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting Update UTMP about System Runlevel Changes...
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: Started Update UTMP about System Runlevel Changes.
Oct 02 12:59:24 nsdc-nethserver.ad.jeckel.local systemd[1]: Startup finished in 16.878s.
Oct 02 13:20:13 nsdc-nethserver.ad.jeckel.local systemd[1]: Started /usr/bin/samba-tool domain passwordsettings set --min-pwd-age=0 --max-pwd-age=0 --complexity=on --history-length=default.
Oct 02 13:20:13 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting /usr/bin/samba-tool domain passwordsettings set --min-pwd-age=0 --max-pwd-age=0 --complexity=on --history-length=default...
Oct 02 13:20:13 nsdc-nethserver.ad.jeckel.local systemd[1]: run-19869.service: main process exited, code=exited, status=255/n/a
Oct 02 13:20:13 nsdc-nethserver.ad.jeckel.local systemd[1]: Unit run-19869.service entered failed state.
Oct 02 13:20:13 nsdc-nethserver.ad.jeckel.local systemd[1]: run-19869.service failed.
Oct 02 13:22:45 nsdc-nethserver.ad.jeckel.local systemd[1]: Started /usr/bin/samba-tool user create admin --random-password --must-change-at-next-login --login-shell=/usr/libexec/openssh/sftp-server --unix-home=/var/lib/nethserver/home/admin --given-name=NethServer Administrator --use-username-as-cn.
Oct 02 13:22:45 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting /usr/bin/samba-tool user create admin --random-password --must-change-at-next-login --login-shell=/usr/libexec/openssh/sftp-server --unix-home=/var/lib/nethserver/home/admin --given-name=NethServer Administrator --use-username-as-cn...
Oct 02 13:22:45 nsdc-nethserver.ad.jeckel.local systemd[1]: run-19943.service: main process exited, code=exited, status=255/n/a
Oct 02 13:22:45 nsdc-nethserver.ad.jeckel.local systemd[1]: Unit run-19943.service entered failed state.
Oct 02 13:22:45 nsdc-nethserver.ad.jeckel.local systemd[1]: run-19943.service failed.
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: Started /usr/bin/samba-tool group listmembers Account Operators.
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting /usr/bin/samba-tool group listmembers Account Operators...
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: run-19960.service: main process exited, code=exited, status=255/n/a
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: Unit run-19960.service entered failed state.
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: run-19960.service failed.
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: Started /usr/bin/samba-tool group addmembers Account Operators NETHSERVER$.
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: Starting /usr/bin/samba-tool group addmembers Account Operators NETHSERVER$...
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: run-19971.service: main process exited, code=exited, status=255/n/a
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: Unit run-19971.service entered failed state.
Oct 02 13:22:46 nsdc-nethserver.ad.jeckel.local systemd[1]: run-19971.service failed.

I have a github account, but don’t know how to upload a file via gist.
Can you please give me a hint.

TIA

davidep · October 2, 2017, 12:39pm

http://docs.nethserver.org/en/v7/upgrade.html#sync-and-upgrade

Did you restored on a machine with a different IP? I guess yes, as you’re using rsync. Did you fixed the network configuration after rsync-upgrade step? Did you run post-restore-data?

flatspin · October 2, 2017, 1:23pm

yes. I did db networks setprop.... and than signal-event interface-update.

yes, I did (I followed the docs. ), but I before changing IP

After that the LDAP is running and reachable.
The failure comes with upgrading to AD.
Maybe I was unclear. The upgrade itself from NS6 to NS7 works I think.
But I do not have the users in the upgraded machine to give those folders the appropiate owner.
The users are only in LDAP not in CLI.
Is V7.4 different to V7.3?

davidep · October 2, 2017, 1:28pm

No, there aren’t any changes between 7.3 and 7.4 on the Account provider.

From your journal I see a connection error from the container to the LDAP server. The connection is required to import Samba users from OpenLDAP to Active Directory database. I’m suspecting an IP/firewall configuration error.

Before starting the upgrade to AD, please ensure the LDAP service is still accessible from the green network. Shorewall service must be up.

This is a symptom of a failed AD provisioning

flatspin · October 2, 2017, 1:35pm

I don’t have them before updating to AD.
This is before AD:

I’ll send you a pm with link to gist for the /var/log/messages/. I found out how to. Just drag 'n drop. Easy.

davidep · October 2, 2017, 2:01pm

Are you saying Unix users are in LDAP db but you cannot see them - say - with getent passwd ralf or similar? It could be an sssd cache problem.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sssd-cache.html

…however your container log is clear: the LDAP server didn’t answer. But on the host log there’s no evidence of issues to it, it only states the provision procedure failed.

Please note down the shorewall status before starting the ad upgrade procedure:

systemctl status shorewall

flatspin · October 2, 2017, 2:23pm

I did a rollback again. This is from NS7.4 with local LDAP-Accountprovider before trying to upgrade to AD.
I got this:

If I follow the link, I get

Is this relevant? Don’t think so. With a config-backup it is gone.

getent passwd jeckel
jeckel@jeckel.local:*:5001:502:Ralf Jeckel:/var/lib/nethserver/home/jeckel:/bin/bash

[root@nethserver /]# systemctl status shorewall
● shorewall.service - Shorewall IPv4 firewall
   Loaded: loaded (/usr/lib/systemd/system/shorewall.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/shorewall.service.d
           └─nethserver-firewall-base.conf
   Active: active (exited) since Mon 2017-10-02 14:06:26 CEST; 2h 6min ago
 Main PID: 1444 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/shorewall.service

Oct 02 14:06:25 nethserver.jeckel.local shorewall[1444]: Setting up Route Filtering...
Oct 02 14:06:25 nethserver.jeckel.local shorewall[1444]: Setting up Martian Logging...
Oct 02 14:06:25 nethserver.jeckel.local shorewall[1444]: Setting up Proxy ARP...
Oct 02 14:06:25 nethserver.jeckel.local shorewall[1444]: Preparing iptables-restore input...
Oct 02 14:06:25 nethserver.jeckel.local shorewall[1444]: Running /sbin/iptables-restore ...
Oct 02 14:06:26 nethserver.jeckel.local shorewall[1444]: IPv4 Forwarding Enabled
Oct 02 14:06:26 nethserver.jeckel.local shorewall[1444]: Processing /etc/shorewall/start ...
Oct 02 14:06:26 nethserver.jeckel.local shorewall[1444]: Processing /etc/shorewall/started ...
Oct 02 14:06:26 nethserver.jeckel.local shorewall[1444]: done.
Oct 02 14:06:26 nethserver.jeckel.local systemd[1]: Started Shorewall IPv4 firewall.

Should I disable shorewall to upgrade to AD?

EDIT:
Now it worked. But I changed 2 things. So I will do it again to verify which change was the right one.
Will report ASAP.

davidep · October 2, 2017, 2:37pm

No, it’s fine: the unit is not masked, and has no defined startup conditions.

If you started from a past VM snapshot I’d go with a poweroff + new snapshot + poweron againt, to run further tests on a clean state (clock included).

You could also try to bind the LDAP server from a host in LAN/green network, to see if it is reproducible.

flatspin · October 2, 2017, 4:00pm

This installation needs 2 changes before changing to AD from LDAP:

disable firewall, then upgrade to AD works, but I get a failure with Sogo
I must not use the default suggested AD-name. If I use the default ad.domain.tld Sogo doesn’t work after upgrade to AD. If I use only domain.tld everything is o.k.

But why?

davidep · October 2, 2017, 8:49pm

It’s hard to say… Let’s watch the forum if similar issues arise!

flatspin · October 3, 2017, 11:08am

O.k. it may be related to my specific setup…

But I have another issue. I’ve to change the owner of the home directories manually, to get access to the folders from Windows machines. All home directories have no owner after rsync-upgrade.
Manually chown jeckel /home/jeckel give access with JECKEL\jeckel + passwd.
In this case no big problem, cause there are only 17 users, but what if would be a big installation with hundrets of users.

Can I give you something to find out why this happens?

davidep · October 3, 2017, 12:55pm

Your SSSD service cannot resolve numeric filesystem ids to user names. This could be an sssd problem or an OpenLDAP one.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system-level_authentication_guide/trouble

You could start by increasing the debug and search in sssd logs for any relevant error…

flatspin · October 3, 2017, 4:09pm

Thansk for your help. You are great man!
So I’ll digg the logs…