LSB and binfmt_misc issues

NethServer Version: 7.9.2009
Module: kernel.3.10.0-1160.15.2.el7.x86_64
Moved a vm from one proxmox machine to another and can not load br0. it shows in the web interface but not under ip add. if I try to reload it get the return

ifup br0
add bridge failed: Package not installed

the messages log shows

>     Jul 31 05:48:47 srv systemd: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 5274 (reader#3)
    > Jul 31 05:48:47 srv systemd: Mounting Arbitrary Executable File Formats File System...
    > Jul 31 05:48:47 srv systemd: proc-sys-fs-binfmt_misc.mount mount process exited, code=exited status=32
    > Jul 31 05:48:47 srv systemd: Failed to mount Arbitrary Executable File Formats File System.
    > Jul 31 05:48:47 srv systemd: Unit proc-sys-fs-binfmt_misc.mount entered failed state.
    > Jul 31 05:48:57 srv systemd: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 5271 (reader#0)
    > Jul 31 05:48:57 srv systemd: Mounting Arbitrary Executable File Formats File System...
    > Jul 31 05:48:57 srv systemd: proc-sys-fs-binfmt_misc.mount mount process exited, code=exited status=32
    > Jul 31 05:48:57 srv systemd: Failed to mount Arbitrary Executable File Formats File System.
    > Jul 31 05:48:57 srv systemd: Unit proc-sys-fs-binfmt_misc.mount entered failed state.
    > Jul 31 05:49:07 srv systemd: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 5276 (reader#4)
    > Jul 31 05:49:07 srv systemd: Mounting Arbitrary Executable File Formats File System...
    > Jul 31 05:49:07 srv systemd: proc-sys-fs-binfmt_misc.mount mount process exited, code=exited status=32
    > Jul 31 05:49:07 srv systemd: Failed to mount Arbitrary Executable File Formats File System.
    > Jul 31 05:49:07 srv systemd: Unit proc-sys-fs-binfmt_misc.mount entered failed state.

with the boot log showing
Failed to start LSB: Bring up/down networking.

Mount returns
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9740
So it does work but not when called.

systemctl list-units --failed

systemctl list-units --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● proc-sys-fs-binfmt_misc.mount loaded failed failed Arbitrary Executable File Formats File System
● var-lib-nfs-rpc_pipefs.mount loaded failed failed RPC Pipe File System
● auth-rpcgss-module.service loaded failed failed Kernel Module supporting RPCSEC_GSS
● dnsmasq.service loaded failed failed DNS caching server.
● ipsec.service loaded failed failed Internet Key Exchange (IKE) Protocol Daemon for IPsec
● network.service loaded failed failed LSB: Bring up/down networking
● nsdc.service loaded failed failed NethServer Domain Controller container
● openvpn@host-to-net.service loaded failed failed OpenVPN Robust And Highly Flexible Tunneling Application On host/to/net
● openvpn@shed2office.service loaded failed failed OpenVPN Robust And Highly Flexible Tunneling Application On shed2office
● shorewall.service loaded failed failed Shorewall IPv4 firewall

Is it some stale network setting that is throwing it? I do think it starts with LSB and the rest follows.
TIA

Any more details from:

systemctl status network.service -l

@compsos

Hi

Would be interesting to know which version of Proxmox you’re running on. Proxmox had, besides the upgrade 6x -> 7.0.10 sereral updates since…

If a hw component doesn’t start in a VM, it’s usually the hypervisor… Both Proxmox are identically configured as far as networking is concerned?

My 2 cents
Andy

Thank you for the replies. @dnutan Have shutdown that VM just to build a new VM and see if it works. currently doing 175 updates. NetworkManager was not active, but still nmcli does not show the bridge but does show the eth1. The interface for the bridge.
@Andy_Wismer The source is proxmox 6.4.8 and the new one 7.0.10 both with very similar hardware.
I will finish the current install and activate AD and see what happens.

Between 6 and 7 there are changes involving renaming of the NIC, and issues with the MAC Address of vmbr0 and others…

I have moved a similar system last week without the issue. But I do hear you.
I have tried changing the macaddress to see if that would solve it.

Check this bit here about the MAC Address:

https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0#Check_Linux_Network_Bridge_MAC

@Andy_Wismer Will try that. We were not using ifupdown2 but I did check if the proxmox restore was using the same macaddress.
The test unit install fine. So you maybe right that the restore has brought something extra.

Easiest way would be to upgrade the source Proxmox to 7.x, then you have no issues migrating… :slight_smile:

Thanks @Andy_Wismer I will try that. have just killed both interfaces and try to reassign. might work.
I have to leave the console now but will get back to it a bit later, thanks.

1 Like

Good luck!

@Andy_Wismer You maybe close to the mark.
root@pve:~# udevadm test-builtin net_id /sys/class/net/enp
/sys/class/net/enp4s0 /sys/class/net/enp5s0
root@pve:~# udevadm test-builtin net_id /sys/class/net/enp4s0
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Parsed configuration file /usr/lib/systemd/network/73-usb-net-by-mac.link
Created link configuration context.
Using default interface naming scheme ‘v247’.
ID_NET_NAMING_SCHEME=v247
ID_NET_NAME_MAC=enx086266a05bfb
ID_OUI_FROM_DATABASE=ASUSTek COMPUTER INC.
ID_NET_NAME_PATH=enp4s0
Unload module index
Unloaded link configuration context.
root@pve:~# udevadm test-builtin net_id /sys/class/net/enp5s0
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Parsed configuration file /usr/lib/systemd/network/73-usb-net-by-mac.link
Created link configuration context.
Using default interface naming scheme ‘v247’.
ID_NET_NAMING_SCHEME=v247
ID_NET_NAME_MAC=enx086266a05bfc
ID_OUI_FROM_DATABASE=ASUSTek COMPUTER INC.
ID_NET_NAME_PATH=enp5s0
Unload module index
Unloaded link configuration context.
Now how to fix it. Might go back to the test and try and find what is different.

Most likely, this NIC was called enp0 or something before, now it’s name changed.

The NIC name is referred to by the vmbr0 (Or whatever Bridge you might be using in Proxmox) - so double check that…

Check locally with ip -c link, and verify the correct name for the NIC is used in the vmbr configuration:

This Proxmox did NOT change it’s NIC:

But I needed to add in the lines with HWADDRESS in the config:

The correct MAC Addresses can be copy / pasted from ip -c link

I did have 30+ client Proxmox to upgrade, and had 2 which actually changed the NIC name.
That means no access from LAN, kinda crappy, when you’re doing the whole upgrade via VPN.

Luck and planning was on my side: Those two were members of a Cluster with a seperate storage and cluster network… So login with ssh to Proxmox2 (the working one), using that as jumphost and ssh in to Proxmox1 (the one working without LAN) and fixed the entry for vmbr0.
with ifupdown2, that worked right away… :slight_smile:

As a small note: Since Proxmox 7.0.10, there’s been over 40 updates, mostly “library” files…
So make sure you’ve got ALL !

2 Likes

It seems to be locked in a loop trying to fix br0 to be bridged with eth1
and this is constant when trying add and remove packages
systemd: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 16918 (reader#4)
Jul 31 12:35:12 srv systemd: Mounting Arbitrary Executable File Formats File System…
Jul 31 12:35:12 srv mount: mount: unknown filesystem type ‘binfmt_misc’
Jul 31 12:35:12 srv systemd: proc-sys-fs-binfmt_misc.mount mount process exited, code=exited status=32
Jul 31 12:35:12 srv collectd[16901]: statvfs(/proc/sys/fs/binfmt_misc) failed: No such device
Jul 31 12:35:12 srv systemd: Failed to mount Arbitrary Executable File Formats File System.
Jul 31 12:35:12 srv systemd: Unit proc-sys-fs-binfmt_misc.mount entered failed state.

Do not get this error on the test VM.

@Andy_Wismer No luck with the proxmox network side as it was enp4s0 and enp5s0 with both bridges pointing to the same nics. I have bought some time by having a copy of the proxmox VM image to play with.
Watching the console whilst trying to reinstall the AD had lots of the binfmt_misc errors and did not find a solution.
This VM has a few vpn 's configured to it and suspect they may be causing grief. Applied a back-config file to the test VM and got almost double the interfaces. vm & tap’s and of coarse they know about the working on site network config.
One issue which was interesting is the resolv.conf file had the nameserver as 127.0.0.1 which is fine in a working network but I had to add a nameserver that it could see. Which during reinstalling or removing packages the system returned it to std and then the routine fell over. Not even making the file read only stopped it updating the file and the party.

1 Like

@compsos

Really strange…
Maybe a hardware defect?
Maybe try booting with a LIVE system eg Ubuntu, just to verify the hardware is working as expected…

My 2 cents
Andy

(Still to early on a Sunday morning… More coffee needed!)

Last Monday, a client of mine in Germany had a lightning hit in his mains power…
The Power & Internet went out. Seems the wall socket where DSL comes in had blown, needed to be replaced.

One interesting thing: The client had two Mac Minis, both with dual monitors (One his, the other his wife). Both Mac Minis are still working. His is reduced to single screen (Thunderbolt Interface / Mini Display Port blown).
His wife’s Mac Mini had both Thunderbolts / Mini Display Ports blown, it works, but not even per remote is anything visible (No screen output), only via ssh / network everything seems working…
ARD can’t even reboot the box, ssh can…

Ah well, sh*t happens, needs replacement hardware (Servers had UPS, but not the workstations - it’s a private home…)

It is 2 individual machines. So built a new Proxmox 7 and transferred a vm image file. Should have been straight forward!! Maybe will have to construct a clean NS and use backup routines to load data.

@compsos

Migrating a VM (Specifically NethServer) with AD and a few VPNs is usually not an issue with Proxmox - especially not if using KVM CPUs and VirtIO networking.
Also other VMs, including OPNsense/PFsense migrate without issues…

-> Using KVM-CPUs, it is even possible to migrate a Windows Box from Intel to AMD or vica-versa - without Windows even noticing! No reregistration, or blue screen due to different CPUs!

@Andy_Wismer
Firstly my apologies for disappearing. I had to get this system back in place and go to a job 8 hours drive away.
I got it going by building a new NS7 and bring back the config files and the data manually. It did drop the AD again but was able to create a bridge and repair from there.
I can not prove it but suspect it was between Proxmox real and virtual interfaces (bridges) and the config of NS. AD did need to be rebuilt back on site when the network was active and plugged in. Of coarse on my bench the IP assignment was different. The existing OpenVPN road warrior accounts “download” files were incomplete. Lacked the key and cert, only had the CA. Deleted the definition and re-added the same user and it corrected. All issues were related to network and AD.
If i get the opportunity might try to duplicate and see if it would fail again.
Thanks for the help.

@compsos

Hi

Sh*t happens… :slight_smile:
Good you were able to restore your system and get it back up and running.

I can confirm that NethServer (AD) and Proxmox have no issues - I have this combo running at over 20 sites, and have migrated NethServer often enough from one to another Proxmox - even live (Migration).

My 2 cents
Andy