Sometimes the path to learn something means using very different tools along the way. In my case, I have been learning more on the developing virtual network world along with some of the new DevOps toolsets popping up.
As part of this I have started using KVM as a hypervisor on an Ubuntu 11.10 platform in a portion of my home lab. I have learned quickly that getting something simple done in vSphere can be a bit of a chore in the KVM world. But on the flipside, KVM has been a fun learning experience in understanding virtualization in a more raw format.
One of these challenges I have decided to share is a simple one. I was wanting to play with the Dell-created DevOps deployment tool: Crowbar. Crowbar is an wrapper for OpsCode Chef Server. While a pretty slick little utility to research in the cloud deployment and automation space; one glaring problem is it is designed to run on Dell PowerEdge servers. Since I don’t have PowerEdge servers lying around anymore I needed to run this in a virtual machine. This in itself isn’t a huge problem as a virtual machine can pretty much match most of the logical hardware pieces needed. But, the one problem I ran into was that Crowbar out of the box likes to have a couple interfaces with the ability to tag VLANs itself. In a bare-metal world connecting 802.1q trunk ports to a server is pretty common. And even in the VMware vSphere world you can create trunk portgroups for a guest VM using documented methods(See VGT). But the problem I ran into is how do I pass a trunk port through in KVM/QEMU when using the default bridge-utils setup (not using ovswitch in this case)?
It turns out that quite a lot of articles out there on KVM explicitly state that this isn’t possible (link, link). Seems like the default behavior strips tagging between bridge connected interfaces which defeats any attempt to trunk to a guest VM. So it seemed it was not possible. And at one point after testing myself I was convinced it was not possible too. But, thanks to a teammate who somehow got it working in his lab I decided to press on further and see if there was a way to configure bridge-utils in Ubuntu 11.10 to pass VLAN tags frames in a way that behaves like a trunk port to its guest VM.
And it turns out that not only is it possible but it is also not that hard after all.
My baseline Ubuntu is the 11.10 Server edition with KVM and bridge-utils installed. Here the list of packages installed via apt-get after a clean OS install:
- openssh-server (SSH)
- ubuntu-desktop (for using virt-manager)
- gtk2-engines-pixbuf (see above)
- vlan (need vlan ability on host)
- bridge-utils (base bridge for KVM guests)
- qemu-kvm (the KVM/QEMU packages for Ubuntu 11.10)
- kvm-pxe (needed for PXE and to stop annoying error messages)
- libvirt-bin (Libvirt API for managing my KVM)
- virtinst (Handy python tool for creating guests)
- virt-manager (Cool GUI interface for managing KVM guests)
- virt-top (‘top’ like tool for guests)
And for those that want to past a nice clean line into a shell window
apt-get install openssh-server ubuntu-desktop gtk2-engines-pixbuf vlan bridge-utils qemu-kvm kvm-pxe libvirt-bin virtinst virt-manager virt-top
After installing packages above we need to do a couple steps to configure our host system bridging and networking.
First let’s activate our VLAN module now:
And we need to drop it into our modules file for loading next boot:
echo “8021q” >> /etc/modules
By default our Libvirt will come configured with a default network that will create a virtual bridge called “virbr0” which is setup for NAT to internal guests. In this case we want to use our own custom bridge and remove this virtual one. You can skip this if you want both.
Stop the network if active:
virsh net-destroy default
Undefine (remove) the network from the libvirt config:
virsh net-undefine default
Restart our libvirt service:
service libvirt-bin restart
By default Ubuntu 11.10 will come with some rules setup to allow iptables/ebtables to do filtering on packets passed on bridge interfaces. You have two choices with how to deal with this. You can either setup nice clean rules in iptables/ebtables that allow traffic (including VLAN tagged) through or you can stop them from filtering in the first place. The latter is much easier to do and what I am showing below. If you want to do the former then hit Google up for articles on how to enable filtering with iptables/ebtables and bridge-utils.
First let’s add our bridge module in earlier in the boot process:
echo “bridge” >> /etc/modules
Now open /etc/sysctl.conf up in VIM or Nano and add the following lines to the end:
net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0 net.bridge.bridge-nf-filter-pppoe-tagged = 0 net.bridge.bridge-nf-filter-vlan-tagged = 0
These lines disable any filtering of frames on the bridge devices.
Let’s enable the new settings with this command:
And we add this command to make sure these settings are applied last when our host server boots. Add the following to /etc/rc.local above the ‘exit 0’ line:
/sbin/sysctl -p /etc/sysctl.conf
Now in my setup I am using a host with two network interfaces both of which are presented trunk ports from the attached switch. The first interface (Eth0) will be used for Guest VM traffic and the second (Eth1) will be used for host server management (think: vSphere Service Console). While we will manually create the networking config in a second; we need to have a startup config present so we don’t have to do this every time we boot up. Here are the contents of my /etc/network/interfaces:
# Loopback auto lo iface lo inet loopback # eth0 used by br0 so we put it in manual mode # and add ‘up/down’ commands to enable it auto eth0 iface eth0 inet manual up /sbin/ifconfig eth0 up || /bin/true down /sbin/ifconfig eth0 down || /bin/true # eth1 is used by host for management so we leave dhcp # enabled. This can be replaced with static config also auto eth1 iface eth1 inet dhcp # We need to make sure our eth0 interface is 8021q enabled # We do this by adding an arbitrary VLAN 0 subinterface to it # In manual mode so it should never actually enable by init # vlan-raw-device is required to map when in manual mode auto eth0.0 iface eth0.0 inet manual vlan-raw-device eth0 # We create our bridge interface here # Since eth0 can’t be used as a host interface I am using br0 # as one itself. By enabling dhcp it will pull an IP and behave # just like eth0 for the host itself. You can also put into manual # mode but must uncomment the ‘up’ line to make sure it comes up auto br0 iface br0 inet dhcp # ‘dhcp’ can be ‘manual’ for no IP or ‘static’ for a static IP bridge_ports eth0 # Add our host VLAN-enabled interface to bridge bridge_stp off # disable STP pre-up /sbin/ifconfig eth0 up || /bin/true # make sure eth0 is started #up /sbin/ifconfig br0 up || /bin/true # uncomment if br0 in manual mode # You can also have a host interface on the bridge for a specific VLAN. # This is optional and needed if using single interface host. This is not # required to make VLAN trunking work. # My host will get a VLAN 100 network DHCP address for a br0.100 subinterface auto br0.100 iface br0.100 inet dhcp vlan-raw-device br0 # Nicholas Weaver # last mod: 11/28/2011
Once you customize the file above and update your interfaces file you can actually just run ‘/etc/init.d/networking restart’ to apply the changes. But let’s walk through the manual config also to see the steps:
# Clear any IP address on eth0 as it won’t work # and routing will be hosed once the bridge owns the port ip address flush eth0 # We need to enable VLANs on our source interface (eth0) vconfig add eth0 0 # Create our bridge brctl addbr br0 # Add our interface to our bridge brctl addif br0 eth0 # Get an IP address for eth0 on the br0 (optional, see above) dhclient br0 # Create a VLAN subinterface for host VLAN traffic on eth0 (optional, just an example) vconfig add br0 100 # Get DHCP address for br0.100 from the VLAN 100 DHCP server dhclient br0.100
The commands above result in a single bridge (br0) attached to an interface (eth0) that is enabled for VLAN tagging. At this point we can create a guest VM and test. I recommend using the ‘ubuntu-vm-builder’ package available via apt-get or ‘virt-install’ which we already installed. What is important is that with our current config we are no longer using the default network that comes with Librvirt (virbr0). So some changes need to be configured in the guest domain XML in Libvirt to bind virtual interfaces in the guest to our newly created bridge. Here is an example of how to modify your guest XML:
As you can see above the interface type is changed to ‘bridge’ and the source bridge is pointed at ‘br0’. This means when Libvirt starts our KVM Guest in QEMU it will place the Vnet0 interface from our guest on the Br0 bridge. We can then create subinterfaces inside the guest for Eth0 (which is Vnet0 on the host) and tag traffic for transmission our our VLAN-enabled Br0 & Eth0. In my case I created the default Eth0 and an Eth0.100 and was able to communicate on both the native VLAN and VLAN 100 from within the guest.
Now one last important note for those running some distros underneath KVM. I had issues with VLAN tagging being stripped by the guest VM before it was submitted to the underlying bridge even though VLAN tagging was enabled in the guest. Turns our that the config above automatically chooses Virtio as the interface type used. And in my case my Crowbar VM was build for Ubuntu 10.10 and was not correctly installing the driver for the Virtio network device it was presented. My solution was to switch to the more compatible E1000 device type (also slower) and tagging started working like a charm. Here is an example config with ‘model type’ specified:
So if you are doing as I am and testing the Crowbar VM inside KVM you will need to use E1000 to get the tagging working.
Couple of last cool notes. The entire config above was actually done inside VMware Workstation 8. It does work with VLAN tagging between VMs on the same Host Network. So in my lab I am using VMW –> KVM –> Guest VMs for some great Inception-style networking.
Also, it is possible to configure the all the above as a Libvirt defined network in newer versions of Libvirt. But by default the libvirt-bin package installed via apt-get for Ubuntu 11.10 is 0.9.2 which is missing some of the options we need.
I have more posts coming on some of the stuff I am playing with soon. So stay tuned.