kvm and libvirtd networking cheat sheet

Step 1: Verify your CPU support for Intel VT or AMD-V Virtualization extensions. In some systems, this is disabled on BIOS and you may need to enable it.

cat /proc/cpuinfo | egrep "vmx|svm" You can also do the same with lscpu command

lscpu | grep Virtualization Virtualization: VT-x

I was not seeing any output on "grep -E '(vmx|svm)' /proc/cpuinfo" so I suspected that the fusion settings on my CentOS8 host did not support hardware virtualization.

I was right:

I was now getting the responses I was looking for. For example:

modprobe -a kvm_intel
ls -l /sys/module/kvm/
ls -l /sys/module/kvm_intel/ 
cat /sys/module/kvm_intel/parameters/nested

Step 2: ummmmm??? What happened to step 2 ?

Step 3: Install KVM / QEMU on RHEL/ CentOS 8 KVM packages are distributed on RHEL 8 via AppStream repository. Install KVM on your RHEL 8 server by running the following commands:

• sudo dnf upgrade # like sudo yum update but better and newer.
• sudo dnf install @virt

After installation, verify that Kernel modules are loaded

$ lsmod | grep kvm
kvm_intel 233472 0
kvm 737280 1 kvm_intel

Other than using the above "dumpxml" command to create a xml file wherever you want to put it, kvm itself stores its production xml from some directory. Investigate which one…

sudo virsh setmaxmem vm1 --size 16777216 
sudo virsh setmaxmem vm1 --size 16777216 --live  # or --current
sudo virsh setmem    vm1 --size 8388608  --config
sudo virsh setmem    vm1 --size 8M  --config
sudo virsh setmem    vm1 --size 8G  --config

or use virsh edit vm1

when on vm1: free -m

Good to know specifics when sizing a VM and the difference between KiB and KB. KiB is the larger, while KB is a power of 10

1 KB = 1000 bytes  (MAC OSx and Linux)
1 KiB = 1024 bytes (universal)   2^10

Megabytes
1 MiB = 1,048,576 bytes  2^20

Gigabytes
1 GB =  1,000,000,000 bytes
1 GiB = 1,073,741,824 bytes   2^30
      ; almost 74 Megabytes larger.

Terabytes
1 TiB = 1,099,511,627,776 bytes  2^40  

Petabytes
PiB = 1,125,899,906,842,624   2^50

Exabytes
EiB = 1,152,921,504,606,847,000  2^60

Zettabytes  ZiB  2^70

Yottabytes  YiB   2^80

There are two steps involved in creating and attaching a new storage device to Linux KVM guest VM:

1. First, create a virtual disk image
2. Attach the virtual disk image to the VM

Details from the two steps:

To use the virt-clone command line to clone a virtual machine,

1. make sure the virtual machine is shutdown
2. run virt-clone

 → virt-clone 
--connect=qemu:///system 
--original pangaea 
--name australinea 
--file /var/lib/libvirt/images/austalinea.qcow2

If your virtual machine target name will match the file name, you can shorten the above to

1. run virt-clone (short-cut)

→ virt-clone 
--connect=qemu:///system 
--original pangaea 
--name australinea 
--auto-clone

If you’re connection to remote KVM/QEMU Host machine, put url before /system It will look something like:

   → virt-clone
--connect=qemu://192.168.1.30/system
--original pangaea
--name australinea
--file /var/lib/libvirt/images/australinea.qcow2

• pangaea: Name of VM from which you are cloning.
• australinea: Name given to resulting VM after cloning
• australinea.qcow2:! Image saved that australinea boots from.

Check to confirm that australinea.qcow2 file is successfully stored in /var/lib/libvirt/images folder.

virt-sysprep "resets" or "unconfigures" a virtual machine so that clones can be made from it. The "unconfiguring" just strips any machine specific customizations, like ssh keys, static ip addresses, etc, so that a clone will coexist with the source of the clone.

You can run qemu-img info /var/lib/libvirt/images/vm2.qcow2 to get insight into the kvm image. Good when investigating an error with that image.

1. shutdown the vm
2. destroy the VM (and undefine it
3. remove the disk image

> virsh net-destroy default

> virsh net-undefine default

> service libvirtd restart

> ifconfig

sudo virsh list –all resultining in nothting, no vms active or inactive. This may have changed. I can see all the VMs "shut off" and all the VMs "running"

virsh console vm1 virsh console –domain vm1 –safe

virsh help console

My issue is that the virtual serial console does NOT seem to have a login prompt. I can change that too. On the guest VM… sudo systemctl enable serial-getty@ttyS0.service sudo systemctl start serial-getty@ttyS0.service

To exit the virsh console session (not a trivial task), use CTRL-SHIFT ] Is that the same as C-} ????

Nope for me it was actually NOT shifted, i.e. just plain C-]

Can also try CTRL-SHIFT 5 (

Is that the same as C-% ??

virt-manager

Once you can virsh console into a vm, say vm2, then you have to also enable that console tty to have a login.

Old way: On the virtual machine, add ‘console=ttyS0‘ at the end of the kernel lines in the /boot/grub2/grub.cfg file:

But grub.cfg should NOT be edited by hand. So the new way:

edit the /etc/default/grub file, add ‘console=ttyS0‘ to the GRUB_CMDLINELINUX variable and execute ‘# grub2-mkconfig -o /boot/grub2/grub.cfg‘.

Now, reboot the virtual machine:

Connected to domain vm.example.com Escape character is ^]

Red Hat Enterprise Linux Server 7.0 (Maipo) Kernel 3.10.0-121.el7.x86₆₄ on an x86₆₄

vm login:

I had deleted the guest VM from VMM completely. I had a backup so I cp /backup/full/pangaea.qcow2 /var/lib/libvirt/images That was actually a gzip of the image so I had to run: cd /var/lib/libvirt/images gunzip pangaea.qcow2

That resulted in a proper .qcow2 images in the right location, but before I could run it via libvirt, I had to use VMM to import it. vi

1. create a new VM
2. import existing disk image
3. browse to /var/lib/libvirt/images and select the pangaea.qcow2 file
4. pick a name for the vm that is different from the image. I used "pan" This I found after twice trying to create the "new" vm as "pangaea" That did not work
5. This booted and all the settings of "pangaea" were restored. Only VMM sees the "domain" as "pan" and not "pangaea".

On a terminal session on the KVM host, type virsh net-list virsh net-dhcp-leases <networkname>

To delete a KVM guest use: virsh list virsh shutdown VM virsh undefine VM

KVM is not a clear case as it could be categorized as either one. The KVM kernel module turns Linux kernel into a type 1 bare-metal hypervisor, while the overall system could be categorized to type 2 because the host OS is still fully functional and the other VM's are standard Linux processes from its perspective.

• What is libvirt doing with iptables?

By default, libvirt provides a virtual network named 'default' which acts as a NAT router for virtual machines, routing traffic to the network connected to your host machine. This functionality uses iptables.

For more info, see: https://wiki.libvirt.org/page/VirtualNetworking in particular the section on NAT forwarding.

- How can I make libvirt stop using iptables? WARNING: Any VMs already configured to use these virtual networks will need to be edited: simply removing the <interface> devices via 'virsh edit' should be sufficient. Not doing this step will cause starting to fail, and even after the editing step the VMs will not have network access.

You can remove all libvirt virtual networks on the machine:

Use 'virsh net-list --all' to see a list of all virtual networks Use 'virsh net-destroy $net-name' to shutdown each running network Use 'virsh net-undefine $net-name' to permanently remove the network

• Why doesn't libvirt just auto configure a regular network bridge?

While this would be nice, it is difficult/impossible to do in a safe way that won't hit a lot of trouble with non trivial networking configurations. A static bridge is also not compatible with a laptop mode of networking, switching between wireless and wired. Static bridges also do not play well with NetworkManager as of this writing (Feb 2010).

You can find more info about the motivation virtual networks here: http://www.gnome.org/~markmc/virtual-networking.html

How do I manually configure a network bridge? See: Setting up a Host Bridge

How do I get my VM to use an existing network bridge? See: Configuring Guests to use a Host Bridge

How do I forward incoming connections to a guest that is connected via a NATed virtual network? = See: Forwarding Incoming Connections

This section for info purposes for now. (I did not use this for CentOS kvm configuration and setup.) This outlines how to setup briding using standard network initscripts and systemctl.

Using NetworkManager directly If your distro was released some time after 2015 and uses NetworkManager, it likely supports bridging natively. See these instructions for creating a bridge directly with NetworkManager:

Using nm-connection-editor UI:happyassassin.net

Using the command line:lukas sapletai blog on libvirt with bridge

(for older distros. Otherwise leave it running. It is good.) If your distro was released before 2015, the NetworkManager version likely does not handle bridging, so it is necessary to use "classic" network initscripts for the bridge, and to explicitly mark them as independent from NetworkManager (the "NM_CONTROLLED=no" lines in the scripts below).

If desired, you can also completely disable NetworkManager:

Creating network initscripts In the /etc/sysconfig/network-scripts directory it is neccessary to create 2 config files. The first (ifcfg-eth0) defines your physical network interface, and says that it will be part of a bridge:

   # cat > ifcfg-eth0 <<EOF
DEVICE=eth0
HWADDR=00:16:76:D6:C9:45
ONBOOT=yes
BRIDGE=br0
NM_CONTROLLED=no
EOF

Obviously change the HWADDR to match your actual NIC's address. You may also wish to configure the device's MTU here using e.g. MTU=9000.

The second config file (ifcfg-br0) defines the bridge device:

   # cat > ifcfg-br0 <<EOF
DEVICE=br0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0
NM_CONTROLLED=no
EOF

WARNING: The line TYPE=Bridge is case-sensitive - it must have uppercase 'B' and lower case 'ridge'

After changing this restart networking (or simply reboot)

The final step is to disable netfilter on the bridge:

   # cat >> /etc/sysctl.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
EOF
# sysctl -p /etc/sysctl.conf

It is recommended to do this for performance and security reasons. See Fedora bug #512206. Alternatively you can configure iptables to allow all traffic to be forwarded across the bridge:

You should now have a "shared physical device", to which guests can be attached and have full LAN access

# brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.000000000000       yes
br0             8000.000e0cb30550       yes             eth0

Note how this bridge is completely independant of the virbr0. Do NOT attempt to attach a physical device to 'virbr0' - this is only for NAT connectivity

for client information in ?dconf?

are in ~/.gconf/apps/virt-manager/

The steps here utilize the kvm utility qemu-img, from the host linux server. Shutdown the guest VM, in my example it is australinea. Then modify the KVM settings to allocate more disk space to the guest VM.

Get info on the qcow2 disk image with the command:

• qemu-img info /var/lib/libvirt/images/australinea.qcow2

zintis@c8host ~[1004]$
sudo qemu-img info /var/lib/libvirt/images/australinea.qcow2 
image: /var/lib/libvirt/images/australinea.qcow2
file format: qcow2
virtual size: 5 GiB (5368709120 bytes)
disk size: 3.91 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: true
    refcount bits: 16
    corrupt: false
zintis@c8host ~[1005]$

Notice that we have a virtual size of 5G, but a disk size of only 3.91G virtual size is the size of the virutal disk when creating or expanding the disk. The maximum size here is 5G. The disk size is the current size of the qcow2 file. How much is actually used by the host to store this vm.

We will be resizing the virtual size, adding 2 GB of space:

• qemu-img resize /var/lib/libvirt/images/australinea.qcow2 +2G

sudo qemu-img resize /var/lib/libvirt/images/australinea.qcow2 +2G
Image resized.
zintis@c8host ~[1006]$

Check the new virtual size is recognized by using the info command again:

• qemu-img info /var/lib/libvirt/images/australinea.qcow2

sudo qemu-img info /var/lib/libvirt/images/australinea.qcow2 

image: /var/lib/libvirt/images/australinea.qcow2
file format: qcow2
virtual size: 7 GiB (7516192768 bytes)
disk size: 3.91 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: true
    refcount bits: 16
    corrupt: false
zintis@c8host ~[1007]$

You should now see the additional space as far as qemu-img is concerned.

Ensure that there is no snapshots for this guest, because the next command won't work if you have snapshots:

• virsh snapshot-list australinea

 sudo virsh snapshot-list australinea   
 Name   Creation Time   State
-------------------------------

zintis@c8host ~[1008]$

I had none.

Finally, list the fdisk now visible to the guest:

• sudo fdisk -l /var/lib/libvirt/images/australinea.qcow2

sudo fdisk -l /var/lib/libvirt/images/australinea.qcow2 
Disk /var/lib/libvirt/images/australinea.qcow2: 3.9 GiB, 4197646336 bytes, 8198528 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
zintis@c8host ~[1009]$

Notice that fdisk still shows the original size, 3.9 GiB in my case.