Best methods to setup a hypervisor from command line?

mark_nl · October 13, 2019, 1:51pm

After the latest test-cycle for NethServer-arm 7.7.1908 I started dreaming of a test environment in wich you can run and snapshot armhfp (32bit) and aarch64 (64bit) virtual machines. Full emulated-virtualization on x86_64 is to slow to be practicable.
However after some reading it may be feasible to run aarch64 hypervisor. No emulation for arm32 is needed as armv8 (which includes aarch64) is backward compatible with armv7.

Moreover a first exploration proves it is worthwhile to pursuit. (1)

Problem is i’m a complete noob is these matters. Still trying to grasp how kvm, kvm-qemu, libvirt and virtsh work together in my first prove of concept

Has somebody setup hyporvisor / VM-enviorment before from scratch? (I mean from the command-line using general availible packges) - if so:
Which dirsto? ATM in considering debian/ubuntu or arch linux arm because of matured arm support. - and:
Which tools/packages ?
what is the preferred disk format, ATM using qcow2 (ZFS is out of the question on a tiny arm SBC… could LVM2 be a solution?

Any info/ directions / further-reading are welcome !

mark_nl · October 13, 2019, 1:54pm

(1) prove of concept:

The rpi4 runs a (experimental) unofficial Ubuntu image.
(https://jamesachambers.com/raspberry-pi-ubuntu-server-18-04-2-installation-guide/)
(https://github.com/TheRemote/Ubuntu-Server-raspi4-unofficial)

EDIT:

Played around with my - i do not completely understand how it works - VirtualMachine.
It works like a charm ! Excuses for my excitement but running a arm 32bit VM with Nethserver installed running a DC in a container on a arm64 bit Raspberry PI … is cool.
Has no use case other than testing , still amazingly cool.

robb · October 13, 2019, 2:42pm

There is a lot to be explored. I found this and it just looks awesome: https://blog.alexellis.io/build-your-own-bare-metal-arm-cluster/
Imagine to have an rpi-cluster running where you can host your VM’s…

Over-the-top-example: https://hackaday.com/2018/01/24/firing-up-750-raspberry-pis/ based on BitScope rpi cluster: http://cluster.bitscope.com/

dnutan · October 13, 2019, 6:17pm

I can barely grasp all the virtualization and ARM stuff but check if this is of some use to you:

Using CentOS 7 armhfp VM on CentOS 7 aarch64

https://www.yoctoproject.org/

mark_nl · October 13, 2019, 6:26pm

Thank you @dnutan

came across yocto too @

However this seems to be focused (as many posts) on containers and for testing NS we need VM’s
But will investigate it further!

mark_nl · May 19, 2020, 4:16pm

necrobump

Going through a testing cycle again at the 7.8 point (arm) release, really want to have a better test environment next time!

The must have basic functionality is to be able to snapshot, at least “kick” back to original install before testing a package.

Prove of concept was with libvirt,
Any other suggestions before diving in this again?

stephdl · May 19, 2020, 5:04pm

Obviously I back your effort, find a way to emulate hardware to test rpm, is really fast and quick for testing and development.

I must find time to help :S

mark_nl · May 20, 2020, 4:38pm

followed this guide on my experimental rpi4-el8-aarch64 setup;

Downloaded the official CentOS 7 aarch64 iso and created an VM.
Miraculously it booted the CD and after a looong (45 min) but typical centos GUI installation

It booted…

And this is… well … remarkable : this kernel on itself does not run on a RPI or whatever SBC

Unfortunately no snap shots from the cockpit-webinterface.

However it is encouraging: if this runs we get our nethserver-arm stuff running for sure!

Andy_Wismer · May 20, 2020, 4:58pm

Hi

QEMU can actually “emulate” CPUs, AFAIK even a PPC based CPU would be possible. Albeit REALLY slow, to slow for certain timing operations (timeouts, etc). How much of that would work within KVM, which uses QEMU, is beyond my knowledge.

Maybe even aarch64…

Andy

mark_nl · May 20, 2020, 5:08pm

I’m aiming at aarch64 (armv8) because it is backwards compatible with armv7, and the emulation overhead is quite low.

Tried to emulate on x86 but as you mentioned it is to slow.

So next step is getting armhfp (32 bit) VM up.

and also found this, maybe more feature rich as cockpit-machines

davidep · May 20, 2020, 7:40pm

I still use virt-manager also for remote libvirt hosts and I’m happy with it!

mark_nl · May 20, 2020, 7:54pm

Prefer to stay headless, but on a necessity.

Armhfp runs now on my try, however have to export the kernel out side of the qcow2.
initialize the VM with:

virt-install  --name centos7_armhfp  --memory 2048  \
--boot kernel=/var/lib/libvirt/armhfp-boot/vmlinuz-5.4.28-200.el7.armv7hl+lpae,\
initrd=/var/lib/libvirt/initramfs-5.4.28-200.el7.armv7hl+lpae.img,\
kernel_args="console=ttyAMA0 rw root=/dev/sda2"  \
--disk /var/lib/libvirt/images/Centos-Qemu-lpae-armhfp.qcow2  \
--import  --arch armv7l  --machine virt

mark_nl · May 20, 2020, 7:58pm

I’m now figuring out to get grub on the mbr of the amhfp qcow2-image,
assuming his can load the the kernel inside the vm

davidep · May 21, 2020, 6:43am

With virt-manager you are surely headless. You install virt-manager where you install the browser (i.e. on your laptop). It then connects the hypervisor host with SSH.

To run commands directly on the hypervisor host console, there is virsh for start/stop/snapshot operations. However it is a bit difficult to edit those xml files directly for more advanced tasks like creating and configuring a new VM.

mark_nl · May 21, 2020, 7:58am

Thanx for the direction!

getting somewhere:

davidep · May 21, 2020, 8:38am

I’m happy it works for you!

Another command you’d love on the hypervisor host is

# virsh console <vmname>

It connects to the VM console in text-only. Good for SSH remote terminal connections.

But but but… to make it work you have to remove the existing “Console” device and add a new one like this

note “target type: virtio”

In the end my recipe is: set up the VM with virtmgr (“heavy”), then manage it with virsh (“light”).

mark_nl · May 21, 2020, 6:06pm

I’m a hardware guy, a (virtual ) serial-console works too - needs a argument in the kernel command line but that is common stuff for the hardware guy’s -

Basically with the direction of @davidep i’v got what i need: can clone and snapshot.
The virt-manager makes it supper easy to point to those external kernels and initramfs’s, it is not worth perusing booting the kernel inside the image.

Now to a “production” state: make rpi4 el8 less experimental and installing the lot fresh on a (usb3) UAS attached ssd.