Docker toomanyrequests error

Unfortunately this core-update fails here with errors. These are the details:

What does “You have reached your pull rate limit” mean?

Summary
{
  "context": {
    "action": "update-core",
    "data": {
      "nodes": [
        1
      ]
    },
    "extra": {
      "description": "Processing",
      "eventId": "0a725235-c452-4b44-8ebb-e23fe4c5f65c",
      "title": "Update core"
    },
    "id": "31b09ece-d726-47fd-a230-0552fb9b428a",
    "parent": "",
    "queue": "cluster/tasks",
    "timestamp": "2023-08-02T15:30:20.862256489Z",
    "user": "admin"
  },
  "status": "aborted",
  "progress": 0,
  "subTasks": [
    {
      "context": {
        "action": "list-actions",
        "data": {},
        "extra": {},
        "id": "bf6d9578-cecb-4253-a934-6a5d778aa54c",
        "parent": "31b09ece-d726-47fd-a230-0552fb9b428a"
      },
      "status": "completed",
      "progress": 100,
      "subTasks": [],
      "result": {
        "error": "",
        "exit_code": 0,
        "file": "task/node/1/bf6d9578-cecb-4253-a934-6a5d778aa54c",
        "output": [
          "get-firewall-status",
          "get-name",
          "get-node-status",
          "update-core",
          "remove-custom-zone",
          "remove-module",
          "remove-public-service",
          "add-custom-zone",
          "add-module",
          "add-public-service",
          "get-facts",
          "get-info",
          "set-name",
          "list-actions",
          "cancel-task"
        ]
      }
    },
    {
      "context": {
        "action": "update-module",
        "data": {
          "instances": [
            "loki1"
          ],
          "module_url": "ghcr.io/nethserver/loki:1.0.0"
        },
        "extra": {},
        "id": "25c7a439-051a-4ca5-9605-ddd524b3a958",
        "parent": "31b09ece-d726-47fd-a230-0552fb9b428a"
      },
      "status": "aborted",
      "progress": 30,
      "subTasks": [
        {
          "context": {
            "action": "list-actions",
            "data": {},
            "extra": {},
            "id": "cf31bbbf-87ec-4ada-95fd-c47bd0c9a990",
            "parent": "25c7a439-051a-4ca5-9605-ddd524b3a958"
          },
          "status": "completed",
          "progress": 100,
          "subTasks": [],
          "result": {
            "error": "",
            "exit_code": 0,
            "file": "task/module/loki1/cf31bbbf-87ec-4ada-95fd-c47bd0c9a990",
            "output": [
              "destroy-module",
              "get-status",
              "create-module",
              "configure-backup",
              "list-volumes",
              "run-backup",
              "set-name",
              "clone-module",
              "transfer-state",
              "import-module",
              "list-service-providers",
              "restore-module",
              "update-module",
              "get-name",
              "list-actions",
              "cancel-task"
            ]
          }
        },
        {
          "context": {
            "action": "update-module",
            "data": {
              "module_url": "ghcr.io/nethserver/loki:1.0.0"
            },
            "extra": {},
            "id": "97d3a9cd-b968-441d-a0b1-83d013d1c95d",
            "parent": "25c7a439-051a-4ca5-9605-ddd524b3a958"
          },
          "status": "aborted",
          "progress": 57,
          "subTasks": [],
          "result": {
            "error": "<7>podman-pull-missing ghcr.io/nethserver/loki:1.0.0\n<7>podman-pull-missing docker.io/traefik:v2.4 docker.io/grafana/loki:2.2.1\nTrying to pull docker.io/library/traefik:v2.4...\ntime=\"2023-08-02T17:31:00+02:00\" level=warning msg=\"Failed, retrying in 1s ... (1/3). Error: initializing source docker://traefik:v2.4: reading manifest v2.4 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\ntime=\"2023-08-02T17:31:33+02:00\" level=warning msg=\"Failed, retrying in 1s ... (2/3). Error: initializing source docker://traefik:v2.4: reading manifest v2.4 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\ntime=\"2023-08-02T17:32:06+02:00\" level=warning msg=\"Failed, retrying in 1s ... (3/3). Error: initializing source docker://traefik:v2.4: reading manifest v2.4 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\nError: initializing source docker://traefik:v2.4: reading manifest v2.4 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\nTraceback (most recent call last):\n  File \"/usr/local/agent/bin/podman-pull-missing\", line 35, in <module>\n    subprocess.run(['podman', 'pull', image_url]).check_returncode()\n  File \"/usr/lib64/python3.9/subprocess.py\", line 460, in check_returncode\n    raise CalledProcessError(self.returncode, self.args, self.stdout,\nsubprocess.CalledProcessError: Command '['podman', 'pull', 'docker.io/traefik:v2.4']' returned non-zero exit status 125.\nTraceback (most recent call last):\n  File \"/usr/local/agent/actions/update-module/05pullimages\", line 66, in <module>\n    agent.run_helper('podman-pull-missing', *extra_images,\n  File \"/usr/lib64/python3.9/subprocess.py\", line 460, in check_returncode\n    raise CalledProcessError(self.returncode, self.args, self.stdout,\nsubprocess.CalledProcessError: Command '('podman-pull-missing', 'docker.io/traefik:v2.4', 'docker.io/grafana/loki:2.2.1')' returned non-zero exit status 1.\n",
            "exit_code": 1,
            "file": "task/module/loki1/97d3a9cd-b968-441d-a0b1-83d013d1c95d",
            "output": ""
          }
        }
      ],
      "result": {
        "error": "<7>podman-pull-missing ghcr.io/nethserver/loki:1.0.0\n<3>Module instance \"loki1\" update failed with code 1\nAssertion failed\n  File \"/var/lib/nethserver/cluster/actions/update-module/50update\", line 134, in <module>\n    agent.assert_exp(errors == 0)\n",
        "exit_code": 2,
        "file": "task/cluster/25c7a439-051a-4ca5-9605-ddd524b3a958",
        "output": ""
      }
    },
    {
      "context": {
        "action": "update-module",
        "data": {
          "instances": [
            "traefik1"
          ],
          "module_url": "ghcr.io/nethserver/traefik:1.0.0"
        },
        "extra": {},
        "id": "da440d19-3d70-453d-82c0-12bdd1d0e807",
        "parent": "31b09ece-d726-47fd-a230-0552fb9b428a"
      },
      "status": "aborted",
      "progress": 30,
      "subTasks": [
        {
          "context": {
            "action": "update-module",
            "data": {
              "module_url": "ghcr.io/nethserver/traefik:1.0.0"
            },
            "extra": {},
            "id": "7529638c-ca1c-4711-b945-97d25a772f35",
            "parent": "da440d19-3d70-453d-82c0-12bdd1d0e807"
          },
          "status": "aborted",
          "progress": 57,
          "subTasks": [],
          "result": {
            "error": "<7>podman-pull-missing ghcr.io/nethserver/traefik:1.0.0\n<7>podman-pull-missing docker.io/traefik:v2.9\nTrying to pull docker.io/library/traefik:v2.9...\ntime=\"2023-08-02T17:31:00+02:00\" level=warning msg=\"Failed, retrying in 1s ... (1/3). Error: initializing source docker://traefik:v2.9: reading manifest v2.9 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\ntime=\"2023-08-02T17:31:33+02:00\" level=warning msg=\"Failed, retrying in 1s ... (2/3). Error: initializing source docker://traefik:v2.9: reading manifest v2.9 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\ntime=\"2023-08-02T17:32:06+02:00\" level=warning msg=\"Failed, retrying in 1s ... (3/3). Error: initializing source docker://traefik:v2.9: reading manifest v2.9 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\nError: initializing source docker://traefik:v2.9: reading manifest v2.9 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\nTraceback (most recent call last):\n  File \"/usr/local/agent/bin/podman-pull-missing\", line 35, in <module>\n    subprocess.run(['podman', 'pull', image_url]).check_returncode()\n  File \"/usr/lib64/python3.9/subprocess.py\", line 460, in check_returncode\n    raise CalledProcessError(self.returncode, self.args, self.stdout,\nsubprocess.CalledProcessError: Command '['podman', 'pull', 'docker.io/traefik:v2.9']' returned non-zero exit status 125.\nTraceback (most recent call last):\n  File \"/usr/local/agent/actions/update-module/05pullimages\", line 66, in <module>\n    agent.run_helper('podman-pull-missing', *extra_images,\n  File \"/usr/lib64/python3.9/subprocess.py\", line 460, in check_returncode\n    raise CalledProcessError(self.returncode, self.args, self.stdout,\nsubprocess.CalledProcessError: Command '('podman-pull-missing', 'docker.io/traefik:v2.9')' returned non-zero exit status 1.\n",
            "exit_code": 1,
            "file": "task/module/traefik1/7529638c-ca1c-4711-b945-97d25a772f35",
            "output": ""
          }
        },
        {
          "context": {
            "action": "list-actions",
            "data": {},
            "extra": {},
            "id": "511747e1-9cc7-48e7-83d0-9cfb2e036114",
            "parent": "da440d19-3d70-453d-82c0-12bdd1d0e807"
          },
          "status": "completed",
          "progress": 100,
          "subTasks": [],
          "result": {
            "error": "",
            "exit_code": 0,
            "file": "task/module/traefik1/511747e1-9cc7-48e7-83d0-9cfb2e036114",
            "output": [
              "list-certificates",
              "create-module",
              "destroy-module",
              "get-status",
              "run-backup",
              "get-name",
              "list-volumes",
              "delete-route",
              "upload-certificate",
              "set-acme-server",
              "configure-backup",
              "import-module",
              "delete-certificate",
              "get-certificate",
              "transfer-state",
              "update-module",
              "get-acme-server",
              "get-route",
              "clone-module",
              "list-service-providers",
              "restore-module",
              "set-name",
              "list-routes",
              "set-certificate",
              "set-route",
              "list-actions",
              "cancel-task"
            ]
          }
        }
      ],
      "result": {
        "error": "<7>podman-pull-missing ghcr.io/nethserver/traefik:1.0.0\n<3>Module instance \"traefik1\" update failed with code 1\nAssertion failed\n  File \"/var/lib/nethserver/cluster/actions/update-module/50update\", line 134, in <module>\n    agent.assert_exp(errors == 0)\n",
        "exit_code": 2,
        "file": "task/cluster/da440d19-3d70-453d-82c0-12bdd1d0e807",
        "output": ""
      }
    }
  ],
  "validated": false,
  "result": {
    "error": "Task cluster/update-module run failed: {'output': '', 'error': '<7>podman-pull-missing ghcr.io/nethserver/loki:1.0.0\\n<3>Module instance \"loki1\" update failed with code 1\\nAssertion failed\\n  File \"/var/lib/nethserver/cluster/actions/update-module/50update\", line 134, in <module>\\n    agent.assert_exp(errors == 0)\\n', 'exit_code': 2}\nTask cluster/update-module run failed: {'output': '', 'error': '<7>podman-pull-missing ghcr.io/nethserver/traefik:1.0.0\\n<3>Module instance \"traefik1\" update failed with code 1\\nAssertion failed\\n  File \"/var/lib/nethserver/cluster/actions/update-module/50update\", line 134, in <module>\\n    agent.assert_exp(errors == 0)\\n', 'exit_code': 2}\nAssertion failed\n  File \"/var/lib/nethserver/cluster/actions/update-core/50modules\", line 64, in <module>\n    agent.assert_exp(update_module_errors == 0)\n",
    "exit_code": 2,
    "file": "task/cluster/31b09ece-d726-47fd-a230-0552fb9b428a",
    "output": ""
  }
}

Error excerpt:

Trying to pull docker.io/library/traefik:v2.4...
time="2023-08-02T17:31:00+02:00" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://traefik:v2.4: reading manifest v2.4 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"
time="2023-08-02T17:31:33+02:00" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://traefik:v2.4: reading manifest v2.4 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"
time="2023-08-02T17:32:06+02:00" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://traefik:v2.4: reading manifest v2.4 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"
Error: initializing source docker://traefik:v2.4: reading manifest v2.4 in docker.io/library/traefik: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

I saw that error in the past in Travis-CI logs.

Maybe your environment hit that limit?

The Docker documentation points also to this interesting blog post

Anonymous free users will be limited to 100 pulls per six hours, and authenticated free users will be limited to 200 pulls per six hours. If you would like to avoid rate limits completely, you can purchase or upgrade to a Pro or Team subscription

If you have a Docker Hub account, I know it was possible to set up Registry authentication credentials in NS8 Podman client.

I got a free Docker Hub account now.
Where do I set up Registry authentication credentials in NS8 Podman client?

This might be a common scenario for medium/large organizations (e.g. education, health…).

If NS8 shares its public IP with other systems it might hit the Docker Hub rate limit¹ for image pulls.

To avoid hitting the limit, it is possible to set up Docker Hub credentials. At a root shell prompt type this command²:

runagent podman login docker.io

The command asks for Docker Hub credentials. It could be a good idea to generate and use a read-only access token³ for the NS8 system only – refer to Docker Hub documentation about how to do it.

The access token returned by docker.io is saved in /etc/nethserver/registry.json. Ensure the file is still world-readable because every Unix user of the system (NS8 modules) needs to read it.

chmod -c a+r /etc/nethserver/registry.json

Repeat the procedure for every node of the cluster.

Further references:

  1. Download rate limit | Docker Documentation
  2. podman-login — Podman documentation
  3. Create and manage access tokens | Docker Documentation
1 Like

Thank you so much!

1 Like