ARM development: next steps

arm

(Giacomo Sanchietti) #1

We would like to setup a shared roadmap on the nethserver-arm project.

Purpose

The main aim of the roadmap it’s having an ARM beta when NethServer 7.6.x will be out.
We would like to release an official image for Raspberry 3 (ARM 32) along with a base system for ARM 64 (aarch64).
It should be something like an ARM spin with a list of supported (or at least tested) modules. Some of them may have less feasible use-cases, still offered keeping the Fun-Part in mind.

Development strategy

As long as the code is simple, we can merge ARM specific features inside the core.
If more modifications are needed, we can create an ad-hoc package for ARM installations (something like nethserver-arm-box) to tune specific parameters.
The last resource is the creation of a branch inside git repositories, but we should avoid it because it highly increase the maintenance effort.

Package building

Just stick on manual compilation for now, directly into a Raspberry or using QEMU.
In the meanwhile, we should try to setup an ARM build on Travis.
Otherwise Nethesis could provide a machine using Scaleway (thanks to @Amygos for pointing it out!)

Some Travis/ARM resources:

Repositories

The best solution is to have ARM repository officially hosted inside the main mirror (packages.nethserver.org). We need a little time to improve the upload process and deploy noarch packages for all architectures.

In the meanwhile, @mrmarkuz mirror is working great. We will let you know as soon as @davidep has little time to work on the infra.

The yum repository will have different comps for each architecture, so we are going to merge the ARM comps inside the main repository

Documentation

What is the best place to store the ARM documentation:

Issue tracker

After a brief chat, we decided to move the current issue tracker (https://github.com/markVnl/nethserver-arm-dev/) under NethServer organization (https://github.com/NethServer/arm-dev/).

These are just some ideas, share yours!


Getting a odroid HC1 booting with a Centos-kernel
Testers needed nethserver-arm img
Development Updates 11 December 2018
(Giacomo Sanchietti) #2

(Giacomo Sanchietti) #3

(Giacomo Sanchietti) #4

(Mark Verlinde) #5

Thank you @nethesis (and @giacomo) for pushing the nethserver-arm effort to the next level.
It is really appreciated :clap:

I’ll guess we have to collect the “success story’s” of installs on various systems in the future too
(cc// @LayLow)


(Alessio Fattorini) #6

You’re welcome. It deserves more advertising :slight_smile:


(Mark Verlinde) #7

Meanwhile the first merges in the core-code are a fact! :cake::cake::cake:

Tomorrow going find some time to test :relieved:

I’v been thinking and working on our aarch64 ambition; get some insight on the project risks (which is basically my job, but than for hardware). And identified 3 topics:

  • Default boot method for aarch64 in centos/rhel environment is UEFI boot. IMO we sould adopt this too and enhance it with u-boot EFI for SBC’s. Despite it’s against rhel’s policy.
  • There seem to be no SCL packages, other than devtoolsets, for aarch64 => PHP5.6 and PHP7.x will be hard to get.
  • Because of the smaller audience testing might be less elaborate

Already did some reached on the U-boot/UEFI marriage and am able to load GRUB2 with u-boot and boot a kernel with the linux EFI stub. Unfortunately it gets stuck at loading systemd (PID1) :rage:
(And to be frank I do know J@ck Sh!t about UEFI…)


(Mark Verlinde) #8

We have a prove of concept to boot Centos aarch64 on a SBC over u-boot > UEFI > grub2 :grinning:

As usual still a lot of ironing before us; it is still a bit “hacked” :

  • grub2 config is not written to the appropriate efi directory
  • centos uboot-images-armv8 carry a fedora patch causing u-boot to look in EFIi/fedora for grubaa64.efi
  • can not persuade the appliance-creator to flag the (fat) efi partition as boot

And my Odroid C2 is in need of a kernel > 4.16; Just a with a small change in the kernel-config of the fedora (FC28) was necessary.

Successfully booted an Odroid C2, rpi3 and rpi3+ in 64 bit.
NOTE: this does not change my position on the Raspberry PI’s. They are better off with the 32 bit kernel, it’s just a (academic) prove of concept. (And very funny!)

Here is the image which must be regard as experimental and will not be maintained:
https://drive.google.com/open?id=1RIWYxPadxsf_M-igFxPaP8FmwKA3Jcki

login: root
passwd: centos

On a RPI 3(+) it takes a loooong time to boot, and the hard-beat (green led) does not work;
be patient and if you can hook up a monitor (even this gets blacked out for quite some time…)

For U-boot based boards you must take care of a (recent) u-boot with efi capabilities,
(note for the odroid c2 you need mainline u-boot the one offert by HK nor Centos work, i grabbed mine from a alarm package)

commit of kickstart used to create image


(Mark Verlinde) #9

Turns out i made a stupid mistake creating the img with a kernel not suitable for the C2. Corrected now although it would not be maintained.
Furthermore the heartbeat LED comes to live…; modprobe ledtrig-heartbeat did it.
(Link does does not change)

Pretty close to a complete (armhfp matching) aarch64 repository;
evebox, nethserver-dc and nextcloud are still missing.
For the latter I propose to wait and see developments on PHP upstream can bring us.

EDIT aarch64 repo is complete, untested though!

afbeelding


(Giacomo Sanchietti) #10

The 7.6 is behind the corner (far more early than expected), so we need to decide how to continue the work.

My guess is that CentOS 7.6 will be out later this week, and I’d like to have a stable NS 7.6 release maximum in a couple of weeks.
Do you think we need to provide the ARM spin along with the official release? Or can we postpone it on a second time? Let’s say another couple of weeks.

IMO we are still missing a couple of things:

  • support for RPM uploads on packages.nethserver.org (or do you prefer a new machine?)
  • check if modifications are needed on mirrorlist
  • test builds on Travis (I’m not confident on this)
  • document the whole thing (including the list all supported modules)

Am I missing something?

@mark_nl, what is the best way to test the spin without a real hardware? I’d like to have an ARM 7.5, upgrade it from CR (http://mirror.centos.org/altarch/7/cr/) and check what is missing.

So guys, what should be done to reach our goal?


(Davide Principi) #11

I remember I considered the multi-arch during the initial development of the automated repository maintenance scripts. But I need to get my hands on it again: on the top of my list.


#12

i think that much will depend on how much it will engage the release of 7.6 on x86_64.
it would be better to have stable repositories from first release, so, a lot will depend on when packages.nethserver.org (or a stable alternative) will be available, and I would not see anything wrong if the release for ARM was postponed for a few weeks, it would make us do things right

i think scaleway could be a good start for aarch64, for armhfp i I have not tried it yet but it should be tested: https://arrfab.net/posts/2017/Sep/29/using-centos-7-armhfp-vm-on-centos-7-aarch64/
or also not tested a bare metal arm v7 always on scaleway(?)


(Mark Verlinde) #13

To me the first decision to be made seems to be whether the arm repositories will live beside (A) x86_64 or (like centos odes) in (B) altarch i.e:
(A) http://packages.nethserver.org/nethserver/7.6.1810/base/armhfp/Packages/
(B) http://packages.nethserver.org/nethserver/altarch/7.6.1810/base/armhfp/Packages/

(A) beside x86_64:
pro:

  • Simple and straight forward, probably the easiest to adopt current automation
  • Opens the possibility to merge arm in to nethserver release because repo urls are the same

con:

  • Final (point)releases must be released at the same time because “7” is linked to the latest point release.

(B) in altarch
pro:

  • Point releases can lag behind a bit

con:

  • !pro(A); does not have the pro’s of (A)

I tend to favor in altarch (B) because of the decoupling, especially in this early stage.
Also because:

I tend to agree with @dz00te and lag behind a little bit with arm:
at the release of x86_64 alpha we populate the arm repos, at x86_64 beta we release arm alpha and so on…
This mainly because of lack of man power; to me its appears to be more efficient to test on arm known to work packages instead of aiming at a moving (86_64 test) target.

See above, although it is possible to sync the arm testing repository with planned updates for early birds. As said I tend to prefer to wait until the x86_64 repositories are populated.

Cannot really help with this. But am not confident too: most solutions described are based on qemu-static and this is not a stable build environment. However noarch’s with some arm specific requirements (is this a contradiction? :rofl:) like nethserver-arm-release or nethserver-nextcloud can be build on x86_64

I can make a start with this, but written by me, suffering from a mild-form of Dyslexia, it’s mot going to be really good.

Do not have an answer yet :sweat: On my todo/idea list is to find a convenient way to make snapshots for testing employments, PXE boot on real hardware? ZFS?
For armhfp we have a community with appropriate hardware (raspberry pi) to help. For aarch64 it’s a bigger challenge…

(IMHO)

  • Agree on the repo location
  • Create means to populate this repo (automated or in the initial stage by hand)
  • Agree on the (intal) supported modules

Grtz Mark


(Davide Principi) #14

Great recap Mark!

I’m for option A, and I’m already working to repositories for 7.6 with this new feature.

It becomes an issue after the first “stable” release (I.e. with 7.7).

One possible solution could be based on the NsReleaseLock feature we developed for 7.5.


(Mark Verlinde) #15

Yes this will work with option A, Lets go for it!


(Robert Moskowitz) #16

Hello,

I am kind of back. Still a couple weeks in the sling, so working a little slow. Getting an Kenisngton Optical Track ball (Expert I think) mae a significant improvement; I can use it left-handed.

Also got an Odroid-HC1. I have Fedora29 running on it with rootfs on the sata drive. I am on the Odroid forum getting help on options to also move the boot partition over. It may only be a matter of setting the right env in their special bootloader piece. Centos is waiting on a new image build from Pablo; he had a bad build for 419 and is working on a fix.

So I am looking soon at working with Nethserver again. I will be working on a AD on a Cubietruck, so that should not be too challenging. (other than working with your AD setup and finally moving from XP to Win7 clients!).

The Odroid is for my mailserver and here is the challenge. I MUST support multiple domains in different DNS domains. I use Postfixadmin for this. It works great. How do I start a dialog to see what it would take to integrate Postfixadmin into the Nethserver mail environment?


(Davide Principi) #17

Open a new #feature topic! Anyway why another web admin interface?

The server manager has an Email module with a lot of features too…

http://docs.nethserver.org/en/v7/mail.html


(Robert Moskowitz) #18

Postfixadmin is provides all the hooks into postfix and an sql DB to support multiple domains. Your ldap/pop server, ldap web client, and spam/virus agent all have components to work with the multi-domain database.

For example. I can have joe@foo.com and joe@bar.com and they are different accounts and different users. They each log into the same ldap/pop server with their specific account.

From what I have seen of the Nethserver mail system, this is not supported.


(Mark Verlinde) #19

I have been working on 7.6 and think most (1) dependencies for arm32 bit are met. Coming day’s aarch64 will follow.

(1) most because rspamd 1.8.2 does not play along: it won’t compile, 1.8.1 and 1.8.3 does :tired_face:

Having thought about the above led to the insight it’s almost mandatory to merge arm in nethserver-release to avoid mistakenly install the noarch package targeted @x86_64. An other solution would be to define the arch of the release package.

AFAIK to make a merge possible we need:

  • a package which provides “epel” for arm32
  • a package which provides php72-scl for arm32 and rm64, although these packages may live in “base” (like rh-php56)
  • just a few conditionals in /usr/sbin/nethserver-install, athough it does not seem to be impossible to workaround this in the end.

my question is how do we get the arm packages on packges.nethserver.org ? Or should i proceed with building a 7.6 repo and image based on the current (@mrmarkuz) location?

//cc. @davidep


(Davide Principi) #20

I’m preparing the ARM repositories on packages.nethserver.org, allowing the developers to upload as usual with upload-rpms.

Please review the following proposal:

A noarch package would be cloned to x86_64, armhfp and aarch64 ¹, whilst for arch-dependant packages the rule is

  • x86_64 goes to x86_64 repo
  • armv7hl goes to armhfp repo
  • aarch64 goes to aarch64 repo

As the upload permissions are quite limited, when uploading an ARM package the repository name has to be prefixed with arm-. For example:

  • arm-base
  • arm-updates
  • arm-testing
  • arm-nethforge
  • arm-nethforge-testing
  • ²

Questions:

  1. do we need a blacklist to inihibit the automatic copy? IIRC we said: yes
  2. (sorry if already answered) do we need our arm-epel or just forward to the “official” https://armv7.dev.centos.org/repodir/epel-pass-1/ ?

I hope to finish my task on packages today. If you’re blocked by my work proceed with the current repository!