Test your NS8 module on a real node with GitHub Actions

Development
1

For a developer, Renovate and automated build testing are becoming almost mandatory: they let you keep dependencies up to date without turning every security update into a manual testing exercise. But updating a dependency is only half the job, you also need to know that your module still works afterwards.

Renovate opens pull requests continuously, a Docker tag here, a UI dependency there. Today, nothing proves the module still installs, configures and answers after them.

The Test module workflow we all inherited delegates to DigitalOcean, which needs the Nethesis account, its NS8-CI project and the ci.nethserver.net domain.

Outside Nethesis, it has never run. Not once.

stephdl/ns8-ci-actions replaces it with a reusable workflow that boots a real NS8 node under KVM on a public GitHub runner.

A Rocky 9 and a Debian 13 guest are started in parallel. The NS8 core is installed, a single-node cluster is created, and your own tests/ are run against it.

No secret.
No Nethesis infrastructure.
Works from a fork.

And this is not a mocked NS8 environment: the tests run against an actual NS8 node.

Adopting it is one file

.github/workflows/test-module-qemu.yml, 46 lines.

It waits for Publish images to succeed, then tests the image it just published. A broken build never reaches the test.

Your module needs only what it already has:

  • build-images.sh
  • test-module.sh
  • tests/

What the runner gives you

A GitHub-hosted ubuntu-24.04 runner reports 4 vCPU, 15 GiB usable out of the 16 GiB advertised, 3 GiB of swap, and 87 GiB free on disk.

The guest takes 8 GiB and 4 vCPU by default, on a 30 GiB disk.

The Rocky 9 and Debian 13 ISOs are also cached between runs, avoiding repeated downloads of the installation media.

Around 12 GiB is the practical RAM ceiling, the host needs the rest, and vm_mem is an input, so a module starting several JVMs can ask for more.

For scale: Pi-hole’s node reported 873 MiB in use on a full run.

The default is therefore generous for most modules.

Jobs run in parallel up to a per-account limit:

  • 20 on the Free plan
  • 40 on Pro
  • 60 on Team
  • 180 on Enterprise

One module costs three jobs, the image lookup plus one guest per distribution, so a Free account can test about six repositories at once.

Beyond that they queue rather than fail.

On a public repository, the minutes are free and unmetered. There is no daily cap, only that concurrency limit.

What you can actually assert

Everything your backend exposes, through api-cli on a real node.

Start small.

tests/pihole.robot installs the module, sends a valid configure-module payload, reads it back, and greps the virtual host:

curl -fsS -H 'Host: ${TEST_HOST}' http://127.0.0.1/admin/
Should Contain    ${output}    <form id="loginform">

Four cases, and that grep alone already catches a broken Traefik route, a container that will not start, and an upstream image that changed under you.

A status check would not: /admin/ answers a 302 with an empty body, which curl -f reports as success.

Then go as far as your module deserves.

ns8-webserver could assert every PHP version answers and every virtual host resolves. sftpgo could test eachlogin page.

ns8-mail could test the whole delivery path.

Anything reachable from a shell on the node is one test away: ports, file permissions, container state, a service on its control socket, generated configuration, and so on.

The UI is reachable too, with the Browser library and screenshots.

It is more work, and the backend is where the value is cheapest.

Full documentation: docs/test-on-qemu.md

A wired-up module, file by file

Everything below is from ns8-pihole, which runs this on every published image.

Path What it does
.github/workflows/test-module-qemu.yml The only file you add. Triggers, concurrency group, distribution matrix, and the workflow call
tests/ The whole suite, three files
tests/pihole.robot The cases: install, configure, read back, assert, remove
tests/__init__.robot Opens the SSH session to the node and waits for systemd. Unchanged from the template
tests/pythonreq.txt Two lines, unpinned. Worth checking and pinning if you want a reproducible test environment
test-module.sh Starts the Robot container and points it at the node. Verify the Robot version
build-images.sh Must honour REPOBASE and report what it built on the images output. Already does

Only the first two are really yours to write.

The rest you already have. Think of it as a checklist to verify that the existing files contain what the workflow expects.

The interesting part is that there is no special CI environment to maintain.

You publish an image. GitHub starts two real NS8 nodes. Your module is installed on both. Your existing tests run against both.

If Renovate changes something that breaks the module, you should find out before merging the PR, not after someone installs it.

6 Likes
The NethServer agent skills should be findable from the developer manual
2

Presentation of the github action at nethesis

2 Likes
3

Thank you for this.

THis is great. @kemboielvis22 We could test this with a small module, then work out a plan to do the same for other modules as well.

1 Like
4

Update: UI screenshots and upgrade testing now supported

Following up on the original post: ns8-ci-actions now covers two things that were missing.

Interface screenshots

A Robot Framework case tagged ui drives a real browser against cluster-admin and captures what it sees. Enable it with one input:

uses: stephdl/ns8-ci-actions/.github/workflows/test-module-qemu.yml@v1
with:
  ui_tests_strategy: on_renovate_ui_change  # always | on_ui_change | never

on_renovate_ui_change is the sane default: screenshots only run on a renovate branch that touched ui/ or build-images.sh, so a dependency bump gets a visual check without commenting on every push. The images get posted straight into the pull request:

[Tags]    ui
Import Library    Browser
New Browser    chromium    headless=True
Go To    https://${NODE_ADDR}/cluster-admin/#/apps/${module_id}
Take Screenshot    filename=${OUTPUT DIR}/browser/screenshot/1._Status.png

Working example: tests/25__ui.robot in ns8-postgresql18.

Upgrade testing
A second scenario installs the last published release first, then upgrades it to the image under test, and checks the module survives that:

with:
  scenarios: '["install","update"]'

Opt-in on purpose: your test suite needs to branch on $SCENARIO first, so nothing breaks for modules that don’t. Working example: tests/10__install.robot. It seeds a probe row before the update and checks it’s still there after, catching both config and data regressions.

Full setup
Real caller in production: .github/workflows/test-module-qemu.yml
Docs: docs/test-module-qemu.md · Inputs reference

Cost, and what’s not covered yet

With the default two distros (rocky9, debian13), install alone runs 2 jobs in parallel, one QEMU VM each. Turning on scenarios: '["install","update"]' runs 4, still in parallel, same wall-clock time, but double the runner-minutes billed.

Plenty of headroom either way: GitHub caps concurrent jobs per account, not per workflow, 20 on the Free plan, 40 on Pro, 60 on Team, up to 500 on Enterprise Cloud. A handful of NS8 modules testing at once, on a public repo, won’t come close.

ns8-postgresql18 is the only module running update today, and it goes one step further: it seeds a row before the upgrade and checks it’s still there after (tests/10__install.robot), so a broken migration fails loudly instead of quietly. That’s not something every module needs to do, a config-survival check alone already catches most regressions. My other repos still test install only, for now.

Feedback welcome, especially from anyone testing a module with a different upgrade shape than a plain add-module → update-module.

1 Like
5

Thanks for the information

1 Like