In a virtual environment the network is the blood system. Without a proper network configuration, like with a real servers, you can’t access your applications. The difference to a normal server is that we run several virtual servers inside of an physical server. Each virtual server has its own network setup to access the outside world and that the outside world can access this particular virtual server. To provide this you need to configure the XenServer environment.
Here in this post series we will discuss the network concept of XenServer. This is my first time a dive into it. Please correct me when I understood something wrong. To be honest I will use this series to learn about the network of XenServer. This part 1 will look into the basic concept and naming conventions. Further parts will talk about specific topics like Redundancies, Management and Storage LAN and Virtual Machine networks. At the end I will talk about networks in a Resource Pool and what future version will provide us.
Types of Networks
XenServer knows 3 types of networks in latest 5.6 version:
Internal Network: A network with no link to the outside world. Only virtual machines connected to this network can communicate with each other
External Network: A network with a link to the outside network via a physical NIC that is installed in the server
Bonded Network: A network that has a link to the outside network via a bonded network interface that combines 2 ore more physical NICs. NICs are configured in an active/active configuration and provides redundancies and better throughput
The bonded network is actually not a special type of network with different network functionalities. This just changes the underlying physical uplink configuration for redundancy and performance reasons.
When we are talking about setting up networks and configure them we need to understand how they are named. XenServer knows 3 types of objects, the physical network interface, the virtual one and the network itself.
PIF: This is the object name for a physical network interface. XenServer is based on linux. This means each physical network interface will get the normal linux name ethX. XenServer will create a PIF object and assigned to each ethX device. Each PIF will get his unique ID, uuid, to give him an internal identifier.
VIF: This is the virtual interface card of a virtual machine. A VIF will get a uuid to give him an internal unique identifier. A VIF is connected to a network, which can be either internal or external. You can configure QoS parameters for each VID.
Network: A network is actually a virtual Ethernet switch with bridging functionality. The switch will get a uniqie uuid and name. The basic virtual switch that you will get with a standard XenServer installation acts like a simple physical switch and allows to connect virtual machines to it via their VIF. A virtual switch can provide an uplink via a PIF to create an external network.
For each physical NIC an external network is created and also a network device with the name xenbrX. This is an encapsulation to represent the virtual switch to the Linux kernel and use his bridging functionality. Internal networks and VLAN networks are creating a xapiX device. You will not see the internal network device in the ifconfig, but the VLAN network with a physical NIC assignment you will see.
The virtual switch implemented in XenServer acts as an Layer 2 switch. This means it operates on the Data Link Layer from the OSI reference model. This said the communication between the objects is based on MAC addresses rather than IP addresses. The virtual switch does not provide routing functionality.
XenServer is implemented in the way that a virtual machine, that is connected to only one virtual switch can only communicate with interfaces that are also connected to this virtual switch. A communication between virtual switches is not possible. If a virtual machine has more than one VIF, connected to different virtual switches, routing between internal networks is possible. If a virtual switch is connected to a PIF, the virtual machines can communicate with other servers outside of the XenServer host and also receive packets.
The virtual switch is offering bridging and works on Layer 2, there is no IP address required for the switch to work as virtual switch for virtual machines. The IP configuration is configured inside of virtual machines. The virtual switch transports the data packages based on the MAC addresses. If the switch is connected to a PIF the switch get the MAC address from the physical interface, but does not have an IP address. The management (Primary) or a storage interface (Storage) requires an IP address. In this case XenServer will assign an IP address to the PIF/network. This said the network will get an IP Address assigned to it, a network can have only 1 IP address.
The virtual switch will learn the MAC addresses through packages that are sent and received from connected virtual machines. Through this process the switch learns on which port which MAC address he will find and send data frames directly to them after the learning. If we have an uplink via a PIF, the switch will send a frame through the uplink if he can’t find a matching receiver directly connected to it. The PIF is connected to a physical switch, the port the PIF is connected should be configured as trunk port. This will mark this port to be a receiver for more than one MAC addresses. The physical switch will learn, that this MAC addresses belong to this port. The PIF itself needs to be configured to accept data frames not just for his MAC address. This mode is called promiscuous, the configuration is done by XenServer automatically.
Complication are arising when you live migrate a virtual machine to a different host. This host has a different virtual switch and the PIF is connected to a different port on the same or different physical switch. Before the virtual machine is fully migrated a gratuitous ARP packet is sent so that each external device is able to learn and refreshes the MAC address cache. After that the migrated virtual machine can communicate with outside world, but more important can receive data packets.
The virtual switch itself does not offer advanced options like most physical switches do. The virtual switch is a pretty simple bridging switch. There is only a basic QoS functionality that you can assign to each individual VIF and limit the bandwidth. If you look for more functionality with a full multi layer stack, you need to look at the Open vSwitch project.
We spoke a little bit about VLAN earlier. Creating a VLAN network is easy and must be connected to a PIF. The physical switch port from the physical NIC must be configured for these VLAN.
Creating such a network is doing actually two things. First it creates a ethX.Y Linux network device. EthX for the selected physical NIC and the Y for the VLAN number you select. The ethX.Y device has the same MAC address as the ethX device. On top of that a new network device is created to represent the virtual switch with the name xapiX.
Virtual machines connected to a VLAN configured switch are not aware that they are connected to a VLAN network. The virtual switch is tagging and untagging packets automatically and fully transparent for the virtual machine.
The management (Primary) interface does not support VLAN enabled network. To use a VLAN network we need the support of the physical switch to tag the packet for us. If we want to have the management traffic and a VLAN virtual machine network over the same interface the physical switch needs to be configured for the management interface as the native VLAN and the others as allowed VLANs. This will path through the configured VLANs and will tag each untagged package with the native VLAN.
You can define a dedicated storage interface and assign it to a VLAN configured network. In a later post we will discuss the storage network in more detail.
Network Interface Cards in Unmanaged Mode
Every NIC that is installed in your server will get a PIF representation. On top of a PIF XenServer creates a Network, which we know is a virtual switch. Virtual machines can be connected to these networks. This is the normal behaviour. But this means also, that if you want to connect to a shared storage for example you can’t prevent that the same network card can be used for virtual machines. The same situation we have for the primary management interface.
To prevent a physical interface to show up as a network, we need to put it into unmanaged mode. After we have done that the NIC does not have a PIF representation and no network assigned to it. Unfortunately we cannot do that within XenCenter. First you need to run the CLI command xe pif-forget and than configure the network card ethX manually inside of the console.
The basic idea is that you want to separate the virtual machine traffic from any management traffic, especially the storage network. Putting a network card into the unmanaged mode supports this separation and is from my point of view recommended if you use shared storage. I would like to that with the primary management interface as well. I’ve tried it and I was able to manage XenServer via XenCenter but the “Management Interface” section under the Network tab was not working anymore. Not sure at the moment if XenMotion is working as well.
So we reached the end of the first part. In the next part I will talk about the bonded interface.