Configure backup to S3 Minio, validation fails

NethServer Version: 8.0

Setting up NS8 to use iDrive E2 S3-compatible storage as a backup target was fairly straightforward, but what I really want is to back up NS8 to my NAS, like I was/am doing with NS7. And since my NS8 server isn’t on my LAN, it looks like setting up the NAS as S3-compatible storage is the only option.

Well and good, I’m running TrueNAS SCALE, and TrueCharts has an app available for Minio. It’s a little finicky, but now up and running. Following their installation guide, the Minio console is available at minio.mydomain, and any buckets at bucketname.minio.mydomain. A bucket is set up for ns8, a user with access to that bucket, DNS entries made, reverse proxy entries added, etc.

Now comes configuring NS8, which should be easy, right? If it were, I wouldn’t be writing this post. First try: follow the recommendation on the “Add repository” card, and enter the bucket address as minio.mydomain/ns8. That doesn’t work; the API doesn’t listen on minio.mydomain.

Second attempt: bucket address of ns8.minio.mydomain/ns8 (which seems redundant, but whatever). Again, this fails. The error it lets me copy to my clipboard reads: {"context":{"action":"add-backup-repository","data":{"name":"Minio on TrueNAS","parameters":{"aws_access_key_id":"ns8-user","aws_secret_access_key":"(redacted)"},"password":"XXX","provider":"generic-s3","url":"s3:ns8.minio.familybrown.org"},"extra":{"isNotificationHidden":true,"title":"Add backup repository"},"id":"e026a391-ef94-4195-bef6-77618a4a7263","parent":"","queue":"cluster/tasks","timestamp":"2024-02-19T16:20:27.426776226Z","user":"admin"},"status":"validation-failed","progress":0,"subTasks":[],"validated":false,"result":{"error":"","exit_code":2,"file":"task/cluster/e026a391-ef94-4195-bef6-77618a4a7263","output":[{"error":"backup_repository_not_accessible","field":"parameters","parameter":"parameters","value":"Traceback (most recent call last):\n File \"/usr/local/agent/bin/rclone-wrapper\", line 52, in <module>\n s3_endpoint, s3_path = upath.split('/', 1)\n ^^^^^^^^^^^^^^^^^^^^\nValueError: not enough values to unpack (expected 2, got 1)\n"}]}}

Acting on a hunch, I set a region of “home” for my Minio instance–by default, it doesn’t have a region at all. Tried again to add the repo, failed again: {"context":{"action":"add-backup-repository","data":{"name":"Minio on TrueNAS","parameters":{"aws_access_key_id":"ns8-user","aws_secret_access_key":"(redacted)"},"password":"XXX","provider":"generic-s3","url":"s3:ns8.minio.familybrown.org/ns8"},"extra":{"isNotificationHidden":true,"title":"Add backup repository"},"id":"c258c4e6-bd37-415d-8d8b-c2cb2787c6d6","parent":"","queue":"cluster/tasks","timestamp":"2024-02-19T16:53:05.67198408Z","user":"admin"},"status":"validation-failed","progress":0,"subTasks":[],"validated":false,"result":{"error":"","exit_code":2,"file":"task/cluster/c258c4e6-bd37-415d-8d8b-c2cb2787c6d6","output":[{"error":"backup_repository_not_accessible","field":"parameters","parameter":"parameters","value":"2024/02/19 16:53:07 Failed to size: AuthorizationHeaderMalformed: The authorization header is malformed; the region is wrong; expecting 'home'.\n\tstatus code: 400, request id: 17B551CFEF843A5A, host id: 761cee22934e0df6eacf290198c5f0c67d0cd497199d46bf130b1ed24327c551\n"}]}}

The next logical thing to do, I’d think, would be to tell NS8 what region to use, but I don’t see that option anywhere. So what next?

As a side note, I’m getting kind of frustrated with the error handling in NS8:

  • There doesn’t seem to be any way to get the system to show you what the error is. The most it will do is copy it to the clipboard, in which case it barfs JSON–often lots of it–at you. Then, of course, you can paste that into whatever you want.
  • OK, I’ll assume JSON is easiest for the devs to deal with in support requests. So why isn’t anything done to sanitize it? Why are things like passwords (or “secret access keys”) included in plain text?
  • This might be specific to setting up a backup target (which NS8 calls a repository, overloading that word with too many meanings), but when there’s an error, there’s no way to see what the error is without closing the window. So even assuming I could see the error, and figure out from it what’s wrong, I can’t just go back and fix it; I need to start over.
1 Like

Danb did you setup inio as s3 storage on ns8 or as s3 compatible storage?

S3-compatible–it’s definitely not Amazon S3.

Any ideas?

In TrueNAS docs they write

API ingress: bucket_name.minio.mydomain.com (one entry for each bucket)

So if your bucket is named ns8, you Bucket address should be ns8.minio.mydomain.com (without /ns8 path). I’m pretty sure such use case was not tested, as our reference S3 implementation was our MinIO module.

Do you want to try a workaround? Make a backup of the original script, then edit /usr/local/agent/bin/rclone-wrapper. In line 53:

-     rclone_path = ':s3:' + s3_path
+     rclone_path = ':s3:' + (s3_path or "/")

Tried this change. Bucket address of ns8.minio.mydomain. It fails. Error appears to be the same as before:

{"context":{"action":"add-backup-repository","data":{"name":"Minio instance on TrueNAS","parameters":{"aws_access_key_id":"ns8-user","aws_secret_access_key":"(redacted)"},"password":"XXX","provider":"generic-s3","url":"s3:ns8.minio.familybrown.org"},"extra":{"isNotificationHidden":true,"title":"Add backup repository"},"id":"4cf30f7a-f2da-444b-9f02-474d72316bd4","parent":"","queue":"cluster/tasks","timestamp":"2024-03-18T19:39:15.682252636Z","user":"admin"},"status":"validation-failed","progress":0,"subTasks":[],"validated":false,"result":{"error":"","exit_code":2,"file":"task/cluster/4cf30f7a-f2da-444b-9f02-474d72316bd4","output":[{"error":"backup_repository_not_accessible","field":"parameters","parameter":"parameters","value":"Traceback (most recent call last):\n File \"/usr/local/agent/bin/rclone-wrapper\", line 52, in <module>\n s3_endpoint, s3_path = upath.split('/', 1)\n ^^^^^^^^^^^^^^^^^^^^\nValueError: not enough values to unpack (expected 2, got 1)\n"}]}}

1 Like

Eh, if you think putting all that on one line makes it easier to read, go for it, I guess–I thought it was easier to read with word wrap.

With the /ns8 on the end of the path, same error as I’d received previously: "value":"2024/03/18 19:43:50 Failed to size: AuthorizationHeaderMalformed: The authorization header is malformed; the region is wrong; expecting 'home'.\n\tstatus code: 400, request id: 17BDF360DF69E197, host id: 761cee22934e0df6eacf290198c5f0c67d0cd497199d46bf130b1ed24327c551\n"}

1 Like

Right, I reverted my reformatting attempt :frowning:

I thought the JSON highlighter would also prettify it…

True, the highlighting helps–but taking away word wrap hurts more than the highlighting helps, I think. JSON is a decidedly mixed blessing IMO.