Cross-compiling multi-arch RPMs

danb35 · June 23, 2021, 12:42pm

I realized the other day that, though most of the RPMs I’ve built have been .noarch, I have a couple (caddy and acme-dns) that are compiled binaries. Since Neth on ARM is a thing, it’d probably be good to build for that arch too. But as often seems to be the case with building RPMs, I’m not finding much in the way of clear or current documentation. So,

Is it possible to build both architectures using a single .spec file, such that rpmbuild -ba foo.spec or make-rpms foo.spec would build both? I’d think it’d have to be; in the old days it was very common to build for i386 and x64 at the same time.
If it’s possible, is there a relatively easy way to do it?

mark_nl · June 23, 2021, 1:52pm

Hi @danb35,

In general it is not possible to cross-compile userspace programs/binaries. This is not because a cross-compiler is not able to do it but because of (dynamically) linked libraries used in the build process (glibc to name one).
Because x64_86 can handle/run i386 it is possible to “cross”-compile i386 on x64.

This being said AFAIK caddy and acme-dns are written in golang. The engineers @google designed golang so it can be cross compiled. (quite simple: include the source-code of all the libraries and statically link them in one executable binary)

Here an example (first I found happen to be cady)

Note: never tried it myself because have little armhfp and aarch64 build nodes up 24/7;
interesting experiment though.

danb35 · June 23, 2021, 5:27pm

I’ve understood that it’s difficult (though not necessarily impossible) to do this for a variety of reasons, including what you cite, but as you observe, the particular software I’m interested in is written in Go, which apparently makes cross-compiling (among other things) quite a bit easier. So I guess there are two distinct questions to consider:

How do I compile Go software for ARM on a x64 machine?
What do I need to do, in a .spec file or otherwise, to build, on a x64 machine, a RPM for ARM architecture?

The first one looks pretty straightforward–find the go build command, and preface it with env GOOS=linux GOARCH=arm.

The second looks messier. From what I’m finding (and again, documentation on building RPMs seems sparse and old), it looks like I’d need to do something like this:

%build
%ifarch armv7hl
env GOOS=linux GOARCH=arm go build -v -o acme-dns
%endif
%ifarch x86_64
go build -v -o acme-dns
%endif

…and then manually specify the target as an argument to rpmbuild: rpmbuild -bb --target armv7hl-redhat-linux, to build separately from the x86_64 package (which would be built by default, as that’s the “native” arch for my build machine).

Is that it? Seems a little clunky, but feasible. But my .spec file would need modification to compile natively on a Pi, I’d think.

mark_nl · June 23, 2021, 5:51pm

rpmbuild takes the argument --target=xxx, and mock takes the argument --arch=xxx;
to build for arm xxx = arm or aarch64. (reference preserved in arm-dev)

I have to think a bit (understatement) longer how this can direct the build in the desired direction ie arch for golang… (really do not know… seems not unsolvable )

EDIT:
rpmbuild takes the argument --target=
mock takes the argument --arch=

danb35 · June 23, 2021, 8:23pm

Well, a little bit of trial and error. Added this section to the .spec file:

%build
%ifarch aarch64
env GOOS=linux GOARCH=aarch64 go build -v -o acme-dns
%endif
%ifarch x86_64
go build -v -o acme-dns
%endif

Invoked with rpmbuild -bb --target=aarch64 acme-dns.spec. And initial indications looked promising:

➜  SPECS rpmbuild -bb --target=aarch64 acme-dns.spec
Building target platforms: aarch64
Building for target aarch64
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.VtiYKn

But the result wasn’t as hoped:

Wrote: /var/lib/nethserver/home/dan/rpm-build/RPMS/x86_64/acme-dns-0.8-1.el7.x86_64.rpm
Wrote: /var/lib/nethserver/home/dan/rpm-build/RPMS/x86_64/acme-dns-debuginfo-0.8-1.el7.x86_64.rpm

Back to the drawing board. I suspect I need a different operator/macro than %ifarch.

mark_nl · June 23, 2021, 8:29pm

did you remove BuildArch: x86_64 from acme-dns.spec ?

mark_nl · June 23, 2021, 8:36pm

from the top of my head you may want:

%ifarch %{arm}
env GOOS=linux GOARCH=arm build -v -o acme-dns
%endif
%ifarch aarch64
env GOOS=linux GOARCH=arm64 go build -v -o acme-dns
%endif

(also note in the above for aarch64 GOARCH=arm64)

danb35 · June 23, 2021, 9:01pm

Nope, I’d overlooked that. And yes, I also needed to update GOARCH. And it is now compiling the binary properly:

➜  acme-dns-0.8-1.el7.aarch64 cd usr/local/bin
➜  bin ls
acme-dns
➜  bin file acme-dns
acme-dns: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped

But now I’m running into another issue:

+ install -D -m0644 acme-dns.service /var/lib/nethserver/home/dan/rpm-build/BUILDROOT/acme-dns-0.8-1.el7.aarch64/etc/systemd/system/acme-dns.service
+ /usr/lib/rpm/find-debuginfo.sh --strict-build-id -m --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 50000000 /var/lib/nethserver/home/dan/rpm-build/BUILD/acme-dns-0.8
extracting debug info from /var/lib/nethserver/home/dan/rpm-build/BUILDROOT/acme-dns-0.8-1.el7.aarch64/usr/local/bin/acme-dns
*** ERROR: No build ID note found in /var/lib/nethserver/home/dan/rpm-build/BUILDROOT/acme-dns-0.8-1.el7.aarch64/usr/local/bin/acme-dns

I expect this is where it tries to build the acme-dns-debuginfo RPM, and (ignorant as I am of such things) I’m not surprised it’s failing in a cross-compiling scenario. What I am unsure of is why it’s trying to build such a thing in the first place. But I was able to disable it with %define debug_package %{nil} in the .spec file, and now it builds the RPM. Woohoo!

Looks like I can’t set multiple values for BuildArch:–it will default to whatever the build system is; I can specify others with --target. But I’m putting acme-dns up on my repo for x86_64 and aarch64; they should be available in an hour or so.

mark_nl · June 23, 2021, 9:06pm

yes, golang packages do not deliver debuginfo

(re)building acme-dns on armv7hl (32bit) (natively) I have an build error:

golang.org/x/text@v0.3.1-0.20180807135948-17ff2d5776d2: invalid version: git fetch --unshallow -f origin in /builddir/go/pkg/mod/cache/vcs/38515699458adac9c8b61a0b44f9ad7a5f6edd7bcc2d7fae95930ec78f71e1b4: exit status 128:
fatal: git fetch-pack: expected shallow list

caddy build fine (natively) on armv7hl (32bit) :
https://vps01.havak.nl/nethserver/7/devel/armhfp/Packages/caddy/

danb35 · June 23, 2021, 9:08pm

Interesting, acme-dns did for x86_64 (even though I never asked it to, AFAIK)–but caddy didn’t. But that’s disabled now. I was also able to build for armv7hl; I’ll have that up shortly.

mark_nl · June 23, 2021, 9:14pm

AFIAK, they are empty…

danb35 · June 23, 2021, 10:28pm

Nope, acme-dns-debuginfo-0.8-1.el7.x86_64.rpm contains:

[root@neth test]# ls -alRh
.:
total 12K
drwxr-xr-x   3 root root   4.0K Jun 23 18:19 .
dr-xr-x---. 31 root root   4.0K Jun 23 18:19 ..
drwxr-sr-x   3 root apache 4.0K Jun 23 18:19 usr

./usr:
total 12K
drwxr-sr-x 3 root apache 4.0K Jun 23 18:19 .
drwxr-xr-x 3 root root   4.0K Jun 23 18:19 ..
drwxr-sr-x 3 root apache 4.0K Jun 23 18:19 lib

./usr/lib:
total 12K
drwxr-sr-x 3 root apache 4.0K Jun 23 18:19 .
drwxr-sr-x 3 root apache 4.0K Jun 23 18:19 ..
drwxr-xr-x 4 root root   4.0K Jun 23 18:19 debug

./usr/lib/debug:
total 16K
drwxr-xr-x 4 root root   4.0K Jun 23 18:19 .
drwxr-sr-x 3 root apache 4.0K Jun 23 18:19 ..
drwxr-xr-x 3 root root   4.0K Jun 23 18:19 .build-id
drwxr-xr-x 3 root root   4.0K Jun 23 18:19 usr

./usr/lib/debug/.build-id:
total 12K
drwxr-xr-x 3 root root 4.0K Jun 23 18:19 .
drwxr-xr-x 4 root root 4.0K Jun 23 18:19 ..
drwxr-xr-x 2 root root 4.0K Jun 23 18:19 a5

./usr/lib/debug/.build-id/a5:
total 8.0K
drwxr-xr-x 2 root root 4.0K Jun 23 18:19 .
drwxr-xr-x 3 root root 4.0K Jun 23 18:19 ..
lrwxrwxrwx 1 root root   30 Jun 23 18:19 ef5eefb7f10973319b10637f2dc26a7d0ccd1c -> ../../../../local/bin/acme-dns
lrwxrwxrwx 1 root root   34 Jun 23 18:19 ef5eefb7f10973319b10637f2dc26a7d0ccd1c.debug -> ../../usr/local/bin/acme-dns.debug

./usr/lib/debug/usr:
total 12K
drwxr-xr-x 3 root root 4.0K Jun 23 18:19 .
drwxr-xr-x 4 root root 4.0K Jun 23 18:19 ..
drwxr-xr-x 3 root root 4.0K Jun 23 18:19 local

./usr/lib/debug/usr/local:
total 12K
drwxr-xr-x 3 root root 4.0K Jun 23 18:19 .
drwxr-xr-x 3 root root 4.0K Jun 23 18:19 ..
drwxr-xr-x 2 root root 4.0K Jun 23 18:19 bin

./usr/lib/debug/usr/local/bin:
total 6.2M
drwxr-xr-x 2 root root 4.0K Jun 23 18:19 .
drwxr-xr-x 3 root root 4.0K Jun 23 18:19 ..
-r--r--r-- 1 root root 6.2M Oct 23  2019 acme-dns.debug

On Caddy, I’m running into an interesting problem–it isn’t building for arm, but it is for arm64 (and of course for x86_64). I’m building that under Fedora 34, and acme-dns under NS 7 (something in Caddy’s build process requires a newer version of git than is available under CentOS 7), so there’s a bit of difference between the respective build systems.

Edit:
And while it isn’t quite as elegant as running a single rpmbuild -ba command and having packages built for all desired architectures, it’s pretty straightforward to do something like this:

#!/usr/bin/bash

for arch in armv7hl aarch64 x86_64; do
	rpmbuild -bb --target $arch acme-dns.spec
done

rpmbuild -bs acme-dns.spec