NS8: urlopen error timed out during migration from NS7 to NS8 with remote AD

mrmarkuz · May 12, 2023, 10:51pm

NS8 test environment:

I did setup two Proxmox VMs from Debian 11.7.0 ISO and installed NS8 on them.
Got a letsencrypt certificate for NS8 Master node1
Joined node2 to node1

I tried to migrate a from another NS7 Proxmox VM with remote AD. I was told to create the same remote AD on NS8 so I did it and it worked, I can see the users on the NS8 node1.

But the migration didn’t work, on NS7 I still got

NS7 /var/log/messages:

May 13 00:16:42 testserver esmith::event[27756]: Event: nethserver-ns8-migration-save SUCCESS
May 13 00:17:01 testserver agent: Task queue pop error: dial tcp 10.5.4.1:6379: i/o timeout
May 13 00:17:03 testserver cockpit-bridge: Traceback (most recent call last):
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/urllib/request.py", line 1349, in do_open
May 13 00:17:03 testserver cockpit-bridge: encode_chunked=req.has_header('Transfer-encoding'))
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/http/client.py", line 1254, in request
May 13 00:17:03 testserver cockpit-bridge: self._send_request(method, url, body, headers, encode_chunked)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
May 13 00:17:03 testserver cockpit-bridge: self.endheaders(body, encode_chunked=encode_chunked)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
May 13 00:17:03 testserver cockpit-bridge: self._send_output(message_body, encode_chunked=encode_chunked)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
May 13 00:17:03 testserver cockpit-bridge: self.send(msg)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/http/client.py", line 974, in send
May 13 00:17:03 testserver cockpit-bridge: self.connect()
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/http/client.py", line 946, in connect
May 13 00:17:03 testserver cockpit-bridge: (self.host,self.port), self.timeout, self.source_address)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
May 13 00:17:03 testserver cockpit-bridge: raise err
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
May 13 00:17:03 testserver cockpit-bridge: sock.connect(sa)
May 13 00:17:03 testserver cockpit-bridge: socket.timeout: timed out
May 13 00:17:03 testserver cockpit-bridge: During handling of the above exception, another exception occurred:
May 13 00:17:03 testserver cockpit-bridge: Traceback (most recent call last):
May 13 00:17:03 testserver cockpit-bridge: File "/usr/sbin/ns8-action", line 104, in <module>
May 13 00:17:03 testserver cockpit-bridge: resp = request.urlopen(req, timeout = 20)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
May 13 00:17:03 testserver cockpit-bridge: return opener.open(url, data, timeout)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/urllib/request.py", line 526, in open
May 13 00:17:03 testserver cockpit-bridge: response = self._open(req, data)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/urllib/request.py", line 544, in _open
May 13 00:17:03 testserver cockpit-bridge: '_open', req)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
May 13 00:17:03 testserver cockpit-bridge: result = func(*args)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/urllib/request.py", line 1377, in http_open
May 13 00:17:03 testserver cockpit-bridge: return self.do_open(http.client.HTTPConnection, req)
May 13 00:17:03 testserver cockpit-bridge: File "/usr/lib64/python3.6/urllib/request.py", line 1351, in do_open
May 13 00:17:03 testserver cockpit-bridge: raise URLError(err)
May 13 00:17:03 testserver cockpit-bridge: urllib.error.URLError: <urlopen error timed out>

NS8 /var/log/messages:

May 13 00:16:35 node1 traefik1[821]: 192.168.3.162 - - [12/May/2023:22:16:35 +0000] "POST /cluster-admin/api/login HTTP/1.1" 200 269 "-" "-" 2427 "ApiServer-https@redis" "http://127.0.0.1:9311" 15ms
May 13 00:16:35 node1 traefik1[821]: 192.168.3.162 - - [12/May/2023:22:16:35 +0000] "POST /cluster-admin/api/cluster/tasks HTTP/1.1" 201 450 "-" "-" 2428 "ApiServer-https@redis" "http://127.0.0.1:9311" 35ms
May 13 00:16:35 node1 redis[753]: 1:M 12 May 2023 22:16:35.504 * 1 changes in 5 seconds. Saving...
May 13 00:16:35 node1 redis[753]: 1:M 12 May 2023 22:16:35.505 * Background saving started by pid 147
May 13 00:16:35 node1 redis[753]: 147:C 12 May 2023 22:16:35.529 * DB saved on disk
May 13 00:16:35 node1 redis[753]: 147:C 12 May 2023 22:16:35.530 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
May 13 00:16:35 node1 traefik1[821]: 192.168.3.159 - - [12/May/2023:22:16:35 +0000] "GET /cluster-admin/api/cluster/task/41397e14-bbe3-414f-adb0-6b027675b3ac/context HTTP/2.0" 200 364 "-" "-" 2429 "ApiServer-https@redis" "http://127.0.0.1:9311" 20ms
May 13 00:16:35 node1 traefik1[821]: 192.168.3.159 - - [12/May/2023:22:16:35 +0000] "GET /cluster-admin/api/cluster/task/41397e14-bbe3-414f-adb0-6b027675b3ac/context HTTP/2.0" 200 364 "-" "-" 2430 "ApiServer-https@redis" "http://127.0.0.1:9311" 40ms
May 13 00:16:35 node1 redis[753]: 1:M 12 May 2023 22:16:35.606 * Background saving terminated with success
May 13 00:16:35 node1 traefik1[821]: 192.168.3.162 - - [12/May/2023:22:16:35 +0000] "GET /cluster-admin/api/cluster/task/41397e14-bbe3-414f-adb0-6b027675b3ac/status HTTP/1.1" 400 82 "-" "-" 2431 "ApiServer-https@redis" "http://127.0.0.1:9311" 1ms
May 13 00:16:36 node1 traefik1[821]: 192.168.3.159 - - [12/May/2023:22:16:36 +0000] "GET /cluster-admin/api/cluster/task/41397e14-bbe3-414f-adb0-6b027675b3ac/context HTTP/2.0" 200 364 "-" "-" 2433 "ApiServer-https@redis" "http://127.0.0.1:9311" 28ms
May 13 00:16:36 node1 traefik1[821]: 192.168.3.159 - - [12/May/2023:22:16:36 +0000] "GET /cluster-admin/api/cluster/task/41397e14-bbe3-414f-adb0-6b027675b3ac/context HTTP/2.0" 200 364 "-" "-" 2432 "ApiServer-https@redis" "http://127.0.0.1:9311" 53ms
May 13 00:16:36 node1 traefik1[821]: 192.168.3.159 - - [12/May/2023:22:16:36 +0000] "GET /cluster-admin/api/cluster/task/41397e14-bbe3-414f-adb0-6b027675b3ac/status HTTP/2.0" 200 275 "-" "-" 2434 "ApiServer-https@redis" "http://127.0.0.1:9311" 12ms
May 13 00:16:41 node1 traefik1[821]: 192.168.3.162 - - [12/May/2023:22:16:41 +0000] "GET /cluster-admin/api/cluster/task/41397e14-bbe3-414f-adb0-6b027675b3ac/status HTTP/1.1" 200 350 "-" "-" 2435 "ApiServer-https@redis" "http://127.0.0.1:9311" 17ms
May 13 00:16:41 node1 redis[753]: 1:M 12 May 2023 22:16:41.053 * 1 changes in 5 seconds. Saving...
May 13 00:16:41 node1 redis[753]: 1:M 12 May 2023 22:16:41.054 * Background saving started by pid 148
May 13 00:16:41 node1 redis[753]: 148:C 12 May 2023 22:16:41.070 * DB saved on disk
May 13 00:16:41 node1 redis[753]: 148:C 12 May 2023 22:16:41.071 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
May 13 00:16:41 node1 redis[753]: 1:M 12 May 2023 22:16:41.156 * Background saving terminated with success

EDIT:

It’s working without account provider:

mrmarkuz · May 13, 2023, 4:48pm

Migration is working now.

I think it was about DNS issues, what I did:

Set static IPs for the NS8 nodes
Edit /etc/hosts on NS8 nodes and set the right IP instead of 127.0.0.1
Check DNS: NS7 machine and NS8 nodes hostnames should be resolvable from each machine

giacomo · May 15, 2023, 10:23am

Thank you for the test!

Do you think we should improve the manual or the inline doc inside the migration module?

mrmarkuz · May 15, 2023, 2:51pm

Yes, I think correct DNS setup should be mentioned in the manual because it’s is more important with NS8 than it was with NS7.

This one should be clear to avoid problems but I think it’s also worth to be mentioned in the manual.

giacomo · May 16, 2023, 6:51am

I just added a new card to improve the doc on this part.