1. Start an interactive Red Hat Enterprise Linux installation from the Red Hat Enterprise Linux Installation CD-ROM, DVD or PXE.
2. You must enter a valid installation number when prompted to receive access to the virtualization and other Advanced Platform packages.
3. Complete the other steps up to the package selection step.
   
   Select the Virtual Host server role to install a platform for virtualized guests. Alternatively, select the Customize Now radio button to specify individual packages.
4. Select the Virtualization package group. This selects the KVM hypervisor, virt-manager, libvirt and virt-viewer for installation.
   

5. Customize the packages (if required)

   Customize the Virtualization group if you require other virtualization packages.
   
   Press the Close button then the Next button to continue the installation.

Procedure 5.1. Adding the Virtualization entitlement with RHN

1. Log in to RHN using your RHN username and password.
2. Select the systems you want to install virtualization on.
3. In the System Properties section the present systems entitlements are listed next to the Entitlements header. Use the (Edit These Properties) link to change your entitlements.
4. Select the Virtualization checkbox.

Procedure 6.1. Creating a virtualized guest with virt-manager

1. Open virt-manager

   Start virt-manager. Launch the Virtual Machine Manager application from the Applications menu and System Tools submenu. Alternatively, run the virt-manager command as root.

2. Optional: Open a remote hypervisor

   Refer to Section 31.5, “Adding a remote connection”
   Select the hypervisor and press the Connect button to connect to the remote hypervisor.

3. Create a new guest

   The virt-manager window allows you to create a new virtual machine. Click the New button (Figure 6.1, “Virtual Machine Manager window”) to open the New VM wizard.
   

   Figure 6.1. Virtual Machine Manager window

   
   

4. New VM wizard

   The New VM wizard breaks down the guest creation process into five steps:
   1. Naming the guest and choosing the installation type
   2. Locating and configuring the installation media
   3. Configuring memory and CPU options
   4. Configuring the guest's storage
   5. Configuring networking, hypervisor type, architecture, and other hardware settings
   Ensure that virt-manager can access the installation media (whether locally or over the network).

5. Specify name and installation type

   The guest creation process starts with the selection of a name and installation type. Virtual machine names can have underscores (_), periods (.), and hyphens (-).
   

   Figure 6.2. Step 1

   
   
   Type in a virtual machine name and choose an installation type:

   Local install media (ISO image or CDROM)

   This method uses a CD-ROM, DVD, or image of an installation disk (e.g. .iso).

   Network Install (HTTP, FTP, or NFS)

   Network installing involves the use of a mirrored Red Hat Enterprise Linux or Fedora installation tree to install a guest. The installation tree must be accessible through either HTTP, FTP, or NFS.

   Network Boot (PXE)

   This method uses a Preboot eXecution Environment (PXE) server to install the guest. Setting up a PXE server is covered in the Deployment Guide. To install via network boot, the guest must have a routable IP address or shared network device. For information on the required networking configuration for PXE installation, refer to Chapter 10, Network Configuration.

   Import existing disk image

   This method allows you to create a new guest and import a disk image (containing a pre-installed, bootable operating system) to it.

   Click Forward to continue.

6. Configure installation

   Next, configure the OS type and Version of the installation. Except for network booting, this step also requires further configuration (depending on your chosen installation method). When using local install media or importing an existing disk image, you need to specify the location of the installation media or disk image.
   

   Figure 6.3. Local install media (configuration)

   
   
   

   Figure 6.4. Import existing disk image (configuration)

   
   

   Important

   It is recommend that you use the default directory for virtual machine images, /var/lib/libvirt/images/. If you are using a different location, make sure it is added to your SELinux policy and relabeled before you continue with the installation. Refer to Section 16.2, “SELinux and virtualization” for details on how to do this.
   When performing a network install, you need to specify the URL of the installation tree. You can also specify the URL of any kickstart files you want to use, along with any kernel options you want to pass during installation.
   

   Figure 6.5. Network Install (configuration)

   
   
   Click Forward to continue.

7. Configure CPU and memory

   The next step involves configuring the number of CPUs and amount of memory to allocate to the virtual machine. The wizard shows the number of CPUs and amount of memory you can allocate; configure these settings and click Forward.
   

   Figure 6.6. Configuring CPU and Memory

   
   

8. Configure storage

   Assign a physical storage device (Block device) or a file-based image (File). File-based images should be stored in /var/lib/libvirt/images/ to satisfy default SELinux permissions.
   

   Figure 6.7. Configuring virtual storage

   
   
   If you chose to import an existing disk image during the first step, virt-manager will skip this step.
   Assign sufficient space for your virtualized guest and any applications the guest requires, then click Forward to continue.

9. Final configuration

   Verify the settings of the virtual machine and click Finish when you are satisfied; doing so will create the guest with default networking settings, virtualization type, and architecture.
   

   Figure 6.8. Verifying the configuration

   
   
   If you prefer to further configure the virtual machine's hardware first, check the Customize configuration before install box first before clicking Finish. Doing so will open another wizard Figure 6.9, “Virtual hardware configuration” that will allow you to add, remove, and configure the virtual machine's hardware settings.
   

   Figure 6.9. Virtual hardware configuration

   
   
   After configuring the virtual machine's hardware, click Apply. virt-manager will then create the guest with your specified hardware settings.

1. Create a new bridge

   1. Create a new network script file in the /etc/sysconfig/network-scripts/ directory. This example creates a file named ifcfg-installation which makes a bridge named installation.

      # cd /etc/sysconfig/network-scripts/
      # vim ifcfg-installation
      DEVICE=installation
      TYPE=Bridge
      BOOTPROTO=dhcp
      ONBOOT=yes
      

      Warning

      The line, TYPE=Bridge, is case-sensitive. It must have uppercase 'B' and lower case 'ridge'.
   2. Start the new bridge by restarting the network service. The ifup installation command can start the individual bridge but it is safer to test the entire network restarts properly.

      # service network restart
      

   3. There are no interfaces added to the new bridge yet. Use the brctl show command to view details about network bridges on the system.

      # brctl show
      bridge name     bridge id               STP enabled     interfaces
      installation    8000.000000000000       no
      virbr0          8000.000000000000       yes
      

      The virbr0 bridge is the default bridge used by libvirt for Network Address Translation (NAT) on the default Ethernet device.

2. Add an interface to the new bridge

   Edit the configuration file for the interface. Add the BRIDGE parameter to the configuration file with the name of the bridge created in the previous steps.

   # Intel Corporation Gigabit Network Connection
   DEVICE=eth1
   BRIDGE=installation
   BOOTPROTO=dhcp
   HWADDR=00:13:20:F7:6E:8E
   ONBOOT=yes
   

   After editing the configuration file, restart networking or reboot.

   # service network restart
   

   Verify the interface is attached with the brctl show command:

   # brctl show
   bridge name     bridge id               STP enabled     interfaces
   installation    8000.001320f76e8e       no              eth1
   virbr0          8000.000000000000       yes
   

3. Security configuration

   Configure iptables to allow all traffic to be forwarded across the bridge.

   # iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT
   # service iptables save
   # service iptables restart
   

   Disable iptables on bridges

   Alternatively, prevent bridged traffic from being processed by iptables rules. In /etc/sysctl.conf append the following lines:

   net.bridge.bridge-nf-call-ip6tables = 0
   net.bridge.bridge-nf-call-iptables = 0
   net.bridge.bridge-nf-call-arptables = 0
   

   Reload the kernel parameters configured with sysctl.

   # sysctl -p /etc/sysctl.conf
   

4. Restart libvirt before the installation

   Restart the libvirt daemon.

   # service libvirtd reload
   

1. Select PXE

   Select PXE as the installation method.
   

2. Select the bridge

   Select Shared physical device and select the bridge created in the previous procedure.
   

3. Start the installation

   The installation is ready to start.
   

Procedure 7.1. Creating a Red Hat Enterprise Linux 6 guest with virt-manager

1. Optional: Preparation

   Prepare the storage environment for the virtualized guest. For more information on preparing storage, refer to Part V, “Virtualization storage topics”.

   Note

   Various storage types may be used for storing virtualized guests. However, for a guest to be able to use migration features the guest must be created on networked storage.
   Red Hat Enterprise Linux 6 requires at least 1GB of storage space. However, Red Hat recommends at least 5GB of storage space for a Red Hat Enterprise Linux 6 installation and for the procedures in this guide.

2. Open virt-manager and start the wizard

   Open virt-manager by executing the virt-manager command as root or opening Applications -> System Tools -> Virtual Machine Manager.
   

   Figure 7.1. The main virt-manager window

   
   
   Press the create new virtualized guest button (see figure Figure 7.2, “The create new virtualized guest button”) to start the new virtualized guest wizard.
   

   Figure 7.2. The create new virtualized guest button

   
   
   The Create a new virtual machine window opens.

3. Name the virtualized guest

   Guest names can contain letters, numbers and the following characters: '_', '.' and '-'. Guest names must be unique for migration.
   Choose the Local install media (ISO image or CDROM) radio button.
   

   Figure 7.3. The Create a new virtual machine window - Step 1

   
   
   Press Forward to continue.

4. Select the installation media

   Select the installation ISO image location or a DVD drive with the installation disc inside. This example uses an ISO file image of the Red Hat Enterprise Linux 6.0 installation DVD image.
   

   Figure 7.4. The Locate ISO media volume window

   
   

   Image files and SELinux

   For ISO image files and guest storage images, the recommended directory to use is the /var/lib/libvirt/images/ directory. Any other location may require additional configuration for SELinux, refer to Section 16.2, “SELinux and virtualization” for details.
   Select the operating system type and version which match the installation media you have selected.
   

   Figure 7.5. The Create a new virtual machine window - Step 2

   
   
   Press Forward to continue.

5. Set RAM and virtual CPUs

   Choose appropriate values for the virtualized CPUs and RAM allocation. These values affect the host's and guest's performance. Memory and virtualized CPUs can be overcommitted, for more information on overcommitting refer to Chapter 20, Overcommitting with KVM.
   Virtualized guests require sufficient physical memory (RAM) to run efficiently and effectively. Red Hat supports a minimum of 512MB of RAM for a virtualized guest. Red Hat recommends at least 1024MB of RAM for each logical core.
   Assign sufficient virtual CPUs for the virtualized guest. If the guest runs a multithreaded application, assign the number of virtualized CPUs the guest will require to run efficiently.
   You cannot assign more virtual CPUs than there are physical processors (or hyper-threads) available on the host system. The number of virtual CPUs available is noted in the Up to X available field.
   

   Figure 7.6. The Create a new virtual machine window - Step 3

   
   
   Press Forward to continue.

6. Storage

   Enable and assign storage for the Red Hat Enterprise Linux 6 guest. Assign at least 5GB for a desktop installation or at least 1GB for a minimal installation.

   Migration

   Live and offline migrations require guests to be installed on shared network storage. For information on setting up shared storage for guests refer to Part V, “Virtualization storage topics”.

   1. With the default local storage

      Select the Create a disk image on the computer's hard drive radio button to create a file-based image in the default storage pool, the /var/lib/libvirt/images/ directory. Enter the size of the disk image to be created. If the Allocate entire disk now check box is selected, a disk image of the size specified will be created immediately. If not, the disk image will grow as it becomes filled.
      

      Figure 7.7. The Create a new virtual machine window - Step 4

      
      

   2. With a storage pool

      Select Select managed or other existing storage to use a storage pool.
      

      Figure 7.8. The Locate or create storage volume window

      
      
      1. Press the browse button to open the storage pool browser.
      2. Select a storage pool from the Storage Pools list.
      3. Optional: Press the New Volume button to create a new storage volume. Enter the name of the new storage volume.
      4. Press the Choose Volume button to select the volume for the virtualized guest.
      

      Figure 7.9. The Create a new virtual machine window - Step 4

      
      
   Press Forward to continue.

7. Verify and finish

   Verify there were no errors made during the wizard and everything appears as expected.
   Select the Customize configuration before install check box to change the guest's storage or network devices, to use the para-virtualized drivers or, to add additional devices.
   Press the Advanced options down arrow to inspect and modify advanced options. For a standard Red Hat Enterprise Linux 6 none of these options require modification.
   

   Figure 7.10. The Create a new virtual machine window - Step 5

   
   
   Press Finish to continue into the Red Hat Enterprise Linux installation sequence. For more information on installing Red Hat Enterprise Linux 6 refer to the Red Hat Enterprise Linux 6 Installation Guide.

Procedure 7.2. Creating a Red Hat Enterprise Linux 6 guest with virt-manager

1. Optional: Preparation

   Prepare the storage environment for the virtualized guest. For more information on preparing storage, refer to Part V, “Virtualization storage topics”.

   Note

   Various storage types may be used for storing virtualized guests. However, for a guest to be able to use migration features the guest must be created on networked storage.
   Red Hat Enterprise Linux 6 requires at least 1GB of storage space. However, Red Hat recommends at least 5GB of storage space for a Red Hat Enterprise Linux 6 installation and for the procedures in this guide.

2. Open virt-manager and start the wizard

   Open virt-manager by executing the virt-manager command as root or opening Applications -> System Tools -> Virtual Machine Manager.
   

   Figure 7.11. The main virt-manager window

   
   
   Press the create new virtualized guest button (see figure Figure 7.12, “The create new virtualized guest button”) to start the new virtualized guest wizard.
   

   Figure 7.12. The create new virtualized guest button

   
   
   The Create a new virtual machine window opens.

3. Name the virtualized guest

   Guest names can contain letters, numbers and the following characters: '_', '.' and '-'. Guest names must be unique for migration.
   Choose the installation method from the list of radio buttons.
   

   Figure 7.13. The Create a new virtual machine window - Step 1

   
   
   Press Forward to continue.

Procedure 7.3. Creating a Red Hat Enterprise Linux 6 guest with virt-manager

1. Optional: Preparation

   Prepare the storage environment for the virtualized guest. For more information on preparing storage, refer to Part V, “Virtualization storage topics”.

   Note

   Various storage types may be used for storing virtualized guests. However, for a guest to be able to use migration features the guest must be created on networked storage.
   Red Hat Enterprise Linux 6 requires at least 1GB of storage space. However, Red Hat recommends at least 5GB of storage space for a Red Hat Enterprise Linux 6 installation and for the procedures in this guide.

2. Open virt-manager and start the wizard

   Open virt-manager by executing the virt-manager command as root or opening Applications -> System Tools -> Virtual Machine Manager.
   

   Figure 7.14. The main virt-manager window

   
   
   Press the create new virtualized guest button (see figure Figure 7.15, “The create new virtualized guest button”) to start the new virtualized guest wizard.
   

   Figure 7.15. The create new virtualized guest button

   
   
   The Create a new virtual machine window opens.

3. Name the virtualized guest

   Guest names can contain letters, numbers and the following characters: '_', '.' and '-'. Guest names must be unique for migration.
   Choose the installation method from the list of radio buttons.
   

   Figure 7.16. The Create a new virtual machine window - Step 1

   
   
   Press Forward to continue.

Procedure 8.1. Creating a para-virtualized Red Hat Enterprise Linux 6 guest with virt-manager

1. Open virt-manager

   Start virt-manager. Launch the Virtual Machine Manager application from the Applications menu and System Tools submenu. Alternatively, run the virt-manager command as root.

2. Select the hypervisor

   Select the hypervisor. Note that presently the KVM hypervisor is named qemu.
   Connect to a hypervisor if you have not already done so. Open the File menu and select the Add Connection... option. Refer to Section 31.5, “Adding a remote connection”.
   Once a hypervisor connection is selected the New button becomes available. Press the New button.

3. Start the new virtual machine wizard

   Pressing the New button starts the virtual machine creation wizard.

   

   Press Forward to continue.

4. Name the virtual machine

   Provide a name for your virtualized guest. The following punctuation and whitespace characters are permitted for '_', '.' and '-' characters.

   

   Press Forward to continue.

5. Choose a virtualization method

   Select Xen para-virtualized as the virtualization method.
   
   Press Forward to continue.

6. Select the installation method

   Red Hat Enterprise Linux can be installed using one of the following methods:
   • local install media, either an ISO image or physical optical media.
   • Select Network install tree if you have the installation tree for Red Hat Enterprise Linux hosted somewhere on your network via HTTP, FTP or NFS.
   • PXE can be used if you have a PXE server configured for booting Red Hat Enterprise Linux installation media. Configuring a sever to PXE boot a Red Hat Enterprise Linux installation is not covered by this guide. However, most of the installation steps are the same after the media boots.
   Set OS Type to Linux and OS Variant to Red Hat Enterprise Linux 5 as shown in the screenshot.

   

   Press Forward to continue.

7. Locate installation media

   Select ISO image location or CD-ROM or DVD device. This example uses an ISO file image of the Red Hat Enterprise Linux installation DVD.
   1. Press the Browse button.
   2. Search to the location of the ISO file and select the ISO image. Press Open to confirm your selection.
   3. The file is selected and ready to install.

      

      Press Forward to continue.

   Image files and SELinux

   For ISO image files and guest storage images it is recommended to use the /var/lib/libvirt/images/ directory. Any other location may require additional configuration for SELinux, refer to Section 16.2, “SELinux and virtualization” for details.

8. Storage setup

   Assign a physical storage device (Block device) or a file-based image (File). File-based images should be stored in the /var/lib/libvirt/images/ directory to satisfy default SELinux permissions. Assign sufficient space for your virtualized guest and any applications the guest requires.

   

   Press Forward to continue.

   Migration

   Live and offline migrations require guests to be installed on shared network storage. For information on setting up shared storage for guests refer to Part V, “Virtualization storage topics”.

9. Network setup

   Select either Virtual network or Shared physical device.
   The virtual network option uses Network Address Translation (NAT) to share the default network device with the virtualized guest. Use the virtual network option for wireless networks.
   The shared physical device option uses a network bond to give the virtualized guest full access to a network device.

   

   Press Forward to continue.

10. Memory and CPU allocation

    The Memory and CPU Allocation window displays. Choose appropriate values for the virtualized CPUs and RAM allocation. These values affect the host's and guest's performance.
    Virtualized guests require sufficient physical memory (RAM) to run efficiently and effectively. Choose a memory value which suits your guest operating system and application requirements. Remember, guests use physical RAM. Running too many guests or leaving insufficient memory for the host system results in significant usage of virtual memory and swapping. Virtual memory is significantly slower which causes degraded system performance and responsiveness. Ensure you allocate sufficient memory for all guests and the host to operate effectively.
    Assign sufficient virtual CPUs for the virtualized guest. If the guest runs a multithreaded application, assign the number of virtualized CPUs the guest will require to run efficiently. Do not assign more virtual CPUs than there are physical processors (or hyper-threads) available on the host system. It is possible to over allocate virtual processors, however, over allocating has a significant, negative effect on guest and host performance due to processor context switching overheads.
    

    Press Forward to continue.

11. Verify and start guest installation

    Verify the configuration.

    

    Press Finish to start the guest installation procedure.

12. Installing Red Hat Enterprise Linux

    Complete the Red Hat Enterprise Linux installation sequence. The installation sequence is covered by the Red Hat Enterprise Linux 6 Installation Guide. Refer to Red Hat Documentation for the Red Hat Enterprise Linux 6 Installation Guide.

1. Change to the network scripts directory

   Change to the /etc/sysconfig/network-scripts directory:

   # cd /etc/sysconfig/network-scripts
   

2. Modify a network interface to make a bridge

   Edit the network script for the network device you are adding to the bridge. In this example, /etc/sysconfig/network-scripts/ifcfg-eth0 is used. This file defines eth0, the physical network interface which is set as part of a bridge:

   DEVICE=eth0
   # change the hardware address to match the hardware address your NIC uses
   HWADDR=00:16:76:D6:C9:45
   ONBOOT=yes
   BRIDGE=br0
   

   Tip

   You can configure the device's Maximum Transfer Unit (MTU) by appending an MTU variable to the end of the configuration file.

   MTU=9000
   

3. Create the bridge script

   Create a new network script in the /etc/sysconfig/network-scripts directory called ifcfg-br0 or similar. The br0 is the name of the bridge, this can be anything as long as the name of the file is the same as the DEVICE parameter, and that it matches the bridge name used in step 2.

   DEVICE=br0
   TYPE=Bridge
   BOOTPROTO=dhcp
   ONBOOT=yes
   DELAY=0
   

   Warning

   The line, TYPE=Bridge, is case-sensitive. It must have uppercase 'B' and lower case 'ridge'.

4. Restart the network

   After configuring, restart networking or reboot.

   # service network restart
   

5. Configure iptables

   Configure iptables to allow all traffic to be forwarded across the bridge.

   # iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT
   # service iptables save
   # service iptables restart
   

   Disable iptables on bridges

   Alternatively, prevent bridged traffic from being processed by iptables rules. In /etc/sysctl.conf append the following lines:

   net.bridge.bridge-nf-call-ip6tables = 0
   net.bridge.bridge-nf-call-iptables = 0
   net.bridge.bridge-nf-call-arptables = 0
   

   Reload the kernel parameters configured with sysctl.

   # sysctl -p /etc/sysctl.conf
   

6. Restart the libvirt service

   Restart the libvirt service with the service command.

   # service libvirtd reload
   

7. Verify the bridge

   Verify the new bridge is available with the bridge control command (brctl).

   # brctl show
   bridge name     bridge id               STP enabled     interfaces
   virbr0          8000.000000000000       yes
   br0             8000.000e0cb30550       no              eth0
   

1. Download the drivers

   The virtio-win package contains the para-virtualized block and network drivers for all supported Windows guests.
   Download the virtio-win package with the yum command.

   # yum install virtio-win
   

   The drivers are also from Microsoft (windowsservercatalog.com). Note that the Red Hat Enterprise Virtualization Hypervisor and Red Hat Enterprise Linux are created on the same code base so the drivers for the same version (for example, Red Hat Enterprise Virtualization Hypervisor 2.2 and Red Hat Enterprise Linux 5.5) are supported for both environments.
   The virtio-win package installs a CD-ROM image, virtio-win.iso, in the /usr/share/virtio-win/ directory.

2. Install the para-virtualized drivers

   It is recommended to install the drivers on the guest before attaching or modifying a device to use the para-virtualized drivers.
   For block devices storing root file systems or other block devices required for booting the guest, the drivers must be installed before the device is modified. If the drivers are not installed on the guest and the driver is set to the virtio driver the guest will not boot.

1. Run the virsh edit <guestname> command to edit the XML configuration file for your device. For example, virsh edit guest1. The configuration files are located in /etc/libvirt/qemu.
2. Below is a file-based block device using the virtualized IDE driver. This is a typical entry for a virtualized guest not using the para-virtualized drivers.

   <disk type='file' device='disk'>
      <source file='/var/lib/libvirt/images/disk1.img'/>
      <target dev='hda' bus='ide'/>
   </disk>
   

3. Change the entry to use the para-virtualized device by modifying the bus= entry to virtio. Note that if the disk was previously IDE it will have a target similar to hda, hdb, or hdc and so on. When changing to bus=virtio the target needs to be changed to vda, vdb, or vdc accordingly.

   <disk type='file' device='disk'>
      <source file='/var/lib/libvirt/images/disk1.img'/>
      <target dev='vda' bus='virtio'/>
   </disk>
   

4. Remove the address tag inside the disk tags. This must be done for this procedure to work. Libvirt will regenerate the address tag appropriately the next time the guest is started.

Procedure 11.3. Starting the new device wizard

1. Open the virtualized guest by double clicking on the name of the guest in virt-manager.
2. Open the Information tab by pressing the i information button.
   

   Figure 11.1. The information tab button

   
   
3. In the information tab, press the Add Hardware button.
4. In the Adding Virtual Hardware tab select Storage or Network for the type of device. The storage and network device wizards are covered in procedures Procedure 11.4, “Adding a storage device using the para-virtualized storage driver” and Procedure 11.5, “Adding a network device using the para-virtualized network driver”

Procedure 11.4. Adding a storage device using the para-virtualized storage driver

1. Select hardware type

   Select Storage as the Hardware type.
   
   Press Forward to continue.

2. Select the storage device and driver

   Create a new disk image or select a storage pool volume.
   Set the Device type to Virtio Disk to use the para-virtualized drivers.
   
   Press Forward to continue.

3. Finish the procedure

   Confirm the details for the new device are correct.
   
   Press Finish to complete the procedure.

Procedure 11.5. Adding a network device using the para-virtualized network driver

1. Select hardware type

   Select Network as the Hardware type.
   
   Press Forward to continue.

2. Select the network device and driver

   Select the network device from the Host device list.
   Create a custom MAC address or use the one provided.
   Set the Device model to virtio to use the para-virtualized drivers.
   
   Press Forward to continue.

3. Finish the procedure

   Confirm the details for the new device are correct.
   
   Press Finish to complete the procedure.

1. Identify the device

   Identify the PCI device designated for passthrough to the guest. The virsh nodedev-list command lists all devices attached to the system. The --tree option is useful for identifying devices attached to the PCI device (for example, disk controllers and USB controllers).

   # virsh nodedev-list --tree

   For a list of only PCI devices, run the following command:

   # virsh nodedev-list | grep pci

   In the output from this command, each PCI device is identified by a string, as shown in the following example output:

   pci_0000_00_00_0
   pci_0000_00_02_0
   pci_0000_00_02_1
   pci_0000_00_03_0
   pci_0000_00_03_2
   pci_0000_00_03_3
   pci_0000_00_19_0
   pci_0000_00_1a_0
   pci_0000_00_1a_1
   pci_0000_00_1a_2
   pci_0000_00_1a_7
   pci_0000_00_1b_0
   pci_0000_00_1c_0
   

   Tip: determining the PCI device

   Comparing lspci output to lspci -n (which turns off name resolution) output can assist in deriving which device has which device identifier code.
   Record the PCI device number; the number is needed in other steps.
2. Information on the domain, bus and function are available from output of the virsh nodedev-dumpxml command:

   # virsh nodedev-dumpxml pci_8086_3a6c
   <device>
     <name>pci_8086_3a6c</name>
     <parent>computer</parent>
     <capability type='pci'>
       <domain>0</domain>
       <bus>0</bus>
       <slot>26</slot>
       <function>7</function>
       <id='0x3a6c'>82801JD/DO (ICH10 Family) USB2 EHCI Controller #2</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
     </capability>
   </device>

3. Detach the device from the system. Attached devices cannot be used and may cause various errors if connected to a guest without detaching first.

   # virsh nodedev-dettach pci_8086_3a6c 
   Device pci_8086_3a6c dettached

4. Convert slot and function values to hexadecimal values (from decimal) to get the PCI bus addresses. Append "0x" to the beginning of the output to tell the computer that the value is a hexadecimal number.
   For example, if bus = 0, slot = 26 and function = 7 run the following:

   $ printf %x 0
   0
   $ printf %x 26
   1a
   $ printf %x 7
   7

   The values to use:

   bus='0x00'
   slot='0x1a'
   function='0x7'

5. Run virsh edit (or virsh attach device) and added a device entry in the <devices> section to attach the PCI device to the guest.

   # virsh edit win2k3
   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
         <address domain='0x0000' bus='0x00' slot='0x1a' function='0x7'/>
     </source>
   </hostdev>

6. Once the guest system is configured to use the PCI address, the host system must be configured to stop using the device. The ehci driver is loaded by default for the USB PCI controller.

   $ readlink /sys/bus/pci/devices/0000\:00\:1a.7/driver
   ../../../bus/pci/drivers/ehci_hcd

7. Detach the device:

   $ virsh nodedev-dettach pci_8086_3a6c

8. Verify it is now under the control of pci_stub:

   $ readlink /sys/bus/pci/devices/0000\:00\:1d.7/driver
   ../../../bus/pci/drivers/pci-stub

9. Set a sebool to allow the management of the PCI device from the guest:

   $ setsebool -P virt_manage_sysfs 1

10. Start the guest system :

    # virsh start win2k3

1. Identify the device

   Identify the PCI device designated for passthrough to the guest. The virsh nodedev-list command lists all devices attached to the system. The --tree option is useful for identifying devices attached to the PCI device (for example, disk controllers and USB controllers).

   # virsh nodedev-list --tree

   For a list of only PCI devices, run the following command:

   # virsh nodedev-list | grep pci

   In the output from this command, each PCI device is identified by a string, as shown in the following example output:

   pci_0000_00_00_0
   pci_0000_00_02_0
   pci_0000_00_02_1
   pci_0000_00_03_0
   pci_0000_00_03_2
   pci_0000_00_03_3
   pci_0000_00_19_0
   pci_0000_00_1a_0
   pci_0000_00_1a_1
   pci_0000_00_1a_2
   pci_0000_00_1a_7
   pci_0000_00_1b_0
   pci_0000_00_1c_0
   

   Tip: determining the PCI device

   Comparing lspci output to lspci -n (which turns off name resolution) output can assist in deriving which device has which device identifier code.
   Record the PCI device number; the number is needed in other steps.

2. Detach the PCI device

   Detach the device from the system.

   # virsh nodedev-dettach pci_8086_3a6c 
   Device pci_8086_3a6c dettached

3. Open the hardware settings

   Open the virtual machine and click the Add Hardware button to add a new device to the guest.
   

4. Add the new device

   Select Physical Host Device from the Hardware type list. Click Forward to continue.
   

5. Select a PCI device

   Select an unused PCI device. Note that selecting PCI devices presently in use on the host causes errors. In this example a PCI to USB interface device is used.
   

6. Confirm the new device

   Click the Finish button to confirm the device setup and add the device to the guest.
   

1. Identify the PCI device

   Identify the PCI device designated for passthrough to the guest. The virsh nodedev-list command lists all devices attached to the system. The --tree option is useful for identifying devices attached to the PCI device (for example, disk controllers and USB controllers).

   # virsh nodedev-list --tree

   For a list of only PCI devices, run the following command:

   # virsh nodedev-list | grep pci

   In the output from this command, each PCI device is identified by a string, as shown in the following example output:

   pci_0000_00_00_0
   pci_0000_00_02_0
   pci_0000_00_02_1
   pci_0000_00_03_0
   pci_0000_00_03_2
   pci_0000_00_03_3
   pci_0000_00_19_0
   pci_0000_00_1a_0
   pci_0000_00_1a_1
   pci_0000_00_1a_2
   pci_0000_00_1a_7
   pci_0000_00_1b_0
   pci_0000_00_1c_0
   

   Tip: determining the PCI device

   Comparing lspci output to lspci -n (which turns off name resolution) output can assist in deriving which device has which device identifier code.

2. Add the device

   Use the PCI identifier output from the virsh nodedev command as the value for the --host-device parameter.

   # virt-install \
    -n hostdev-test -r 1024 --vcpus 2 \
    --os-variant fedora11 -v \
    -l http://download.fedoraproject.org/pub/fedora/linux/development/x86_64/os \
    -x 'console=ttyS0 vnc' --nonetworks --nographics  \
    --disk pool=default,size=8 \
    --debug --host-device=pci_8086_10bd 

3. Complete the installation

   Complete the guest installation. The PCI device should be attached to the guest.

Procedure 13.1. Attach an SR-IOV network device

1. Enable Intel VT-d in BIOS and in the kernel

   Skip this step if Intel VT-d is already enabled and working.
   Enable Intel VT-D in BIOS if it is not enabled already. Refer to Procedure 12.1, “Preparing an Intel system for PCI passthrough” for procedural help on enabling Intel VT-d in BIOS and the kernel.

2. Verify support

   Verify if the PCI device with SR-IOV capabilities are detected. This example lists an Intel 82576 network interface card which supports SR-IOV. Use the lspci command to verify if the device was detected.

   # lspci
   03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   

   Note that the output has been modified to remove all other devices.

3. Start the SR-IOV kernel modules

   If the device is supported the driver kernel module should be loaded automatically by the kernel. Optional parameters can be passed to the module using the modprobe command. The Intel 82576 network interface card uses the igb driver kernel module.

   # modprobe igb [<option>=<VAL1>,<VAL2>,]
   # lsmod |grep igb
   igb    87592  0
   dca    6708    1 igb
   

4. Activate Virtual Functions

   The max_vfs parameter of the igb module allocates the maximum number of Virtual Functions. The max_vfs parameter causes the driver to spawn, up to the value of the parameter in, Virtual Functions. For this particular card the valid range is 0 to 7.
   Remove the module to change the variable.

   # modprobe -r igb

   Restart the module with the max_vfs set to 1 or any number of Virtual Functions up to the maximum supported by your device.

   # modprobe igb max_vfs=7
   

5. Make the Virtual Functions persistent

   The modprobe command /etc/modprobe.d/igb.conf options igb max_vfs=7

6. Inspect the new Virtual Functions

   Using the lspci command, list the newly added Virtual Functions attached to the Intel 82576 network device.

   # lspci | grep 82576
   0b:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   0b:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   0b:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   

   The identifier for the PCI device is found with the -n parameter of the lspci command. The Physical Functions corresponds to 0b:00.0 and 0b:00.1. All the Virtual Functions have Virtual Function in the description.

7. Verify devices exist with virsh

   The libvirt service must recognize the device before adding a device to a guest. libvirt uses a similar notation to the lspci output. All punctuation characters, ; and ., in lspci output are changed to underscores (_).
   Use the virsh nodedev-list command and the grep command to filter the Intel 82576 network device from the list of available host devices. 0b is the filter for the Intel 82576 network devices in this example. This may vary for your system and may result in additional devices.

   # virsh nodedev-list | grep 0b
   pci_0000_0b_00_0
   pci_0000_0b_00_1
   pci_0000_0b_10_0
   pci_0000_0b_10_1
   pci_0000_0b_10_2
   pci_0000_0b_10_3
   pci_0000_0b_10_4
   pci_0000_0b_10_5
   pci_0000_0b_10_6
   pci_0000_0b_11_7
   pci_0000_0b_11_1
   pci_0000_0b_11_2
   pci_0000_0b_11_3
   pci_0000_0b_11_4
   pci_0000_0b_11_5
   

   The serial numbers for the Virtual Functions and Physical Functions should be in the list.

8. Get device details with virsh

   The pci_0000_0b_00_0 is one of the Physical Functions and pci_0000_0b_10_0 is the first corresponding Virtual Function for that Physical Function. Use the virsh nodedev-dumpxml command to get advanced output for both devices.

   <device>
       <name>pci_0000_0b_00_0</name>
       <parent>pci_0000_00_01_0</parent>
       <driver>
         <name>igb</name>
       </driver>
       <capability type='pci'>
         <domain>0</domain>
         <bus>11</bus>
         <slot>0</slot>
         <function>0</function>
         <product id='0x10c9'>Intel Corporation</product>
         <vendor id='0x8086'>82576 Gigabit Network Connection</vendor>
       </capability>
     </device>
   # virsh nodedev-dumpxml pci_0000_0b_10_0
     <device>
       <name>pci_0000_0b_10_0</name>
       <parent>pci_0000_00_01_0</parent>
       <driver>
         <name>igbvf</name>
       </driver>
       <capability type='pci'>
         <domain>0</domain>
         <bus>11</bus>
         <slot>16</slot>
         <function>0</function>
         <product id='0x10ca'>Intel Corporation</product>
         <vendor id='0x8086'>82576 Virtual Function</vendor>
       </capability>
     </device>
   

   This example adds the Virtual Function pci_0000_0b_10_0 to the guest in Step 10. Note the bus, slot and function parameters of the Virtual Function, these are required for adding the device.

9. Detach the Virtual Functions

   Devices attached to a host cannot be attached to guests. Red Hat Enterprise Linux automatically attaches new devices to the host. Detach the Virtual Function from the host so that the Virtual Function can be used by the guest. Detaching the Physical Function causes errors, only detach the required Virtual Functions.

   # virsh nodedev-dettach pci_0000_0b_10_0
   Device pci_0000_0b_10_0 dettached
   

10. Add the Virtual Function to the guest

    1. Shut down the guest.
    2. Use the output from the virsh nodedev-dumpxml pci_8086_10ca_0 command to calculate the values for the configuration file. Convert slot and function values to hexadecimal values (from decimal) to get the PCI bus addresses. Append "0x" to the beginning of the output to tell the computer that the value is a hexadecimal number.
       The example device has the following values: bus = 3, slot = 16 and function = 1. Use the printf utility to convert decimal values to hexadecimal values.

       $ printf %x 3
       3
       $ printf %x 16
       10
       $ printf %x 1
       1

       This example would use the following values in the configuration file:

       bus='0x03'
       slot='0x10'
       function='0x01'

    3. Open the XML configuration file with the virsh edit command. This example edits a guest named MyGuest.

       # virsh edit MyGuest

    4. The default text editor will open the libvirt configuration file for the guest. Add the new device to the devices section of the XML configuration file.

       <hostdev mode='subsystem' type='pci'>
             <source>
               <address bus='0x03' slot='0x10' function='0x01'/>
             </source>
       </hostdev>
       

    5. Save the configuration.

11. Restart

    Restart the guest to complete the installation.

    # virsh start MyGuest

Procedure 16.1. Creating and mounting a logical volume on a virtualized guest with SELinux enabled

1. Create a logical volume. This example creates a 5 gigabyte logical volume named NewVolumeName on the volume group named volumegroup.

   # lvcreate -n NewVolumeName -L 5G volumegroup

2. Format the NewVolumeName logical volume with a file system that supports extended attributes, such as ext3.

   # mke2fs -j /dev/volumegroup/NewVolumeName

3. Create a new directory for mounting the new logical volume. This directory can be anywhere on your file system. It is advised not to put it in important system directories (/etc, /var, /sys) or in home directories (/home or /root). This example uses a directory called /virtstorage

   # mkdir /virtstorage

4. Mount the logical volume.

   # mount /dev/volumegroup/NewVolumeName /virtstorage

5. Set the correct SELinux type for the libvirt image folder.

   # semanage fcontext -a -t virt_image_t "/virtstorage(/.*)?"
   

   If the targeted policy is used (targeted is the default policy) the command appends a line to the /etc/selinux/targeted/contexts/files/file_contexts.local file which makes the change persistent. The appended line may resemble this:

   /virtstorage(/.*)?    system_u:object_r:virt_image_t:s0
   

6. Run the command to change the type of the mount point (/virtstorage) and all files under it to virt_image_t (the restorecon and setfiles commands read the files in /etc/selinux/targeted/contexts/files/).

   # restorecon -R -v /virtstorage
   

1. Export your libvirt image directory

   Add the default image directory to the /etc/exports file:

   /var/lib/libvirt/images *.example.com(rw,no_root_squash,async)
   

   Change the hosts parameter as required for your environment.

2. Start NFS

   1. Install the NFS packages if they are not yet installed:

      # yum install nfs
      

   2. Open the ports for NFS in iptables and add NFS to the /etc/hosts.allow file.
   3. Start the NFS service:

      # service nfs start
      

3. Mount the shared storage on the destination

   On the destination system, mount the /var/lib/libvirt/images directory:

   # mount sourceURL:/var/lib/libvirt/images /var/lib/libvirt/images
   

   Locations must be the same on source and destination

   Whichever directory is chosen for the guests must exactly the same on host and guest. This applies to all types of shared storage. The directory must be the same or the migration will fail.

1. Verify the guest is running

   From the source system, test1.example.com, verify RHEL4test is running:

   [root@test1 ~]# virsh list
   Id Name                 State
   ----------------------------------
    10 RHEL4                running
   

2. Migrate the guest

   Execute the following command to live migrate the guest to the destination, test2.example.com. Append /system to the end of the destination URL to tell libvirt that you need full access.

   # virsh migrate --live RHEL4test qemu+ssh://test2.example.com/system

   Once the command is entered you will be prompted for the root password of the destination system.

3. Wait

   The migration may take some time depending on load and the size of the guest. virsh only reports errors. The guest continues to run on the source host until fully migrated.

4. Verify the guest has arrived at the destination host

   From the destination system, test2.example.com, verify RHEL4test is running:

   [root@test2 ~]# virsh list
   Id Name                 State
   ----------------------------------
    10 RHEL4                running
   

1. Connect to the source and the target hosts

   Connect to the source and target hosts. On the File menu, click Add Connection, the Add Connection window appears.
   Enter the following details:
   • Hypervisor: Select QEMU.
   • Connection: Select the connection type.
   • Hostname: Enter the hostname.
   Press the Connect button.
   

   Figure 18.1. Add Connection

   
   
   virt-manager now displays the newly connected host in the list of available hosts.
   

   Figure 18.2. Connected Host

   
   

2. Add a storage pool to both hosts

   Both hosts must be connected to the same storage pool. Create the storage pool on both hosts using the same network storage device. Using a storage pool ensures both servers have identical storage configurations. This procedure uses a NFS server.

   1. Open the storage tab

      On the Edit menu, click Host Details, the Host Details window appears.
      Click the Storage tab.
      

      Figure 18.3. Storage tab

      
      

   2. Add a storage pool with the same NFS to the source and target hosts.

      Add a new storage pool. In the lower left corner of the window, click the + button. The Add a New Storage Pool window appears.
      Enter the following details:
      • Name: Enter the name of the storage pool.
      • Type: Select netfs: Network Exported Directory.
      

      Figure 18.4. Add a new Storage Pool

      
      
      Press Forward to continue.

   3. Specify storage pool details

      Enter the following details:
      • Format: Select the storage type. This must be NFS or iSCSI for live migrations.
      • Host Name: Enter the IP address or fully-qualified domain name of the storage server.
      

      Figure 18.5. Storage pool details

      
      
      Press the Finish button to add the storage pool.

   4. Verify the new storage pool was added sucessfully

      The new storage pool should be visible in the Storage tab.
      

      Figure 18.6. New storage pool in the storage tab

      
      
   Complete these steps on both hosts before proceeding.

3. Optional: Add a volume to the storage pool

   Add a volume to the storage pool or create a new virtualized guest on the storage pool. If your storage pool already has virtualized guests, you can skip this step.
   1. Create a new volume in the shared storage pool, click New Volume.
      Enter the details, then click Create Volume.
      

      Figure 18.7. Add a storage volume

      
      

   2. Create a new virtualized guest on the new volume

      Create a new virtualized guest that uses the new volume. For information on creating virtualized guests, refer to Part II, “Installation”.
      

      Figure 18.8. New virtualized guest

      
      
      The Virtual Machine window appears.
      

      Figure 18.9. Virtual Machine window

      
      

4. Migrate the virtualized guest

   From the main virt-manager screen, right-click on the virtual machine and select Migrate.... The Migrate the virtual machine window appears.
   

   Figure 18.10. Migrate the virtual machine

   
   
   Select the destination host from the list.
   Select Migrate offline to disable live migration and do an offline migration.
   Select advanced options if required. For a standard migration, no of these settings should be modified.
   Press Migrate to confirm and migrate the virtualized guest.
5. A status bar tracks the progress of the migration. Once the migration is complete the virtualized guest will appear in the list of virtualized guests on the destination.
   

   Figure 18.11. Completed migration

   
   

1. Optional: Changing user

   Change user, if required. This example uses the local root user for remotely managing the other hosts and the local host.

   $ su -

2. Generating the SSH key pair

   Generate a public key pair on the machine virt-manager is used. This example uses the default key location, in the ~/.ssh/ directory.

   $ ssh-keygen -t rsa

3. Copying the keys to the remote hosts

   Remote login without a password, or with a passphrase, requires an SSH key to be distributed to the systems being managed. Use the ssh-copy-id command to copy the key to root user at the system address provided (in the example, root@example.com).

   $ ssh-copy-id -i ~/.ssh/id_rsa.pub root@example.com
   root@example.com's password:
   

   Now try logging into the machine, with the ssh root@example.com command and check in the .ssh/authorized_keys file to make sure unexpected keys have not been added.
   Repeat for other systems, as required.

4. Optional: Add the passphrase to the ssh-agent

   Add the passphrase for the SSH key to the ssh-agent, if required. On the local host, use the following command to add the passphrase (if there was one) to enable password-less login.

   # ssh-add ~/.ssh/id_rsa.pub

1. Start virt-manager.
2. Open the File->Add Connection menu.
3. Input values for the hypervisor type, the connection, Connection->Remote tunnel over SSH, and enter the desired hostname, then click connection.

1. Run the following command to verify the CPU virtualization extensions are available:

   $ grep -E 'svm|vmx' /proc/cpuinfo
   

2. Analyze the output.
   • The following output contains a vmx entry indicating an Intel processor with the Intel VT extensions:

     flags   : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush 
     	dts acpi mmx fxsr sse sse2 ss ht  tm syscall lm constant_tsc pni monitor ds_cpl
     	vmx est tm2 cx16 xtpr lahf_lm
     

   • The following output contains an svm entry indicating an AMD processor with the AMD-V extensions:

     flags   :  fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush
     	mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cx16
     	lahf_lm cmp_legacy svm cr8legacy ts fid vid ttp tm stc
     

   If any output is received, the processor has the hardware virtualization extensions. However in some circumstances manufacturers disable the virtualization extensions in BIOS.
   The "flags:" output content may appear multiple times, once for each hyperthread, core or CPU on the system.
   The virtualization extensions may be disabled in the BIOS. If the extensions do not appear or full virtualization does not work refer to Procedure 34.1, “Enabling virtualization extensions in BIOS”.

3. For users of the KVM hypervisor

   If the kvm package is installed. I As an additional check, verify that the kvm modules are loaded in the kernel:

   # lsmod | grep kvm
   

   If the output includes kvm_intel or kvm_amd then the kvm hardware virtualization modules are loaded and your system meets requirements. sudo

1. Extract from the virsh capabilities output.

   <topology>
     <cells num='2'>
       <cell id='0'>
       <cpus num='4'>
         <cpu id='0'/>
         <cpu id='1'/>
         <cpu id='2'/>
         <cpu id='3'/>
       </cpus>
     </cell>
     <cell id='1'>
       <cpus num='4'>
         <cpu id='4'/>
         <cpu id='5'/>
         <cpu id='6'/>
         <cpu id='7'/>
       </cpus>
     </cell>
     </cells>
   </topology>

2. Observe that the node 1, <cell id='1'>, has physical CPUs 4 to 7.
3. The guest can be locked to a set of CPUs by appending the cpuset attribute to the configuration file.
   1. While the guest is offline, open the configuration file with virsh edit.
   2. Locate where the guest's virtual CPU count is specified. Find the vcpus element.

      <vcpus>4</vcpus>

      The guest in this example has four CPUs.
   3. Add a cpuset attribute with the CPU numbers for the relevant NUMA cell.

      <vcpus cpuset='4-7'>4</vcpus>

4. Save the configuration file and restart the guest.

Procedure 24.1. Workaround for Red Hat Enterprise Linux 6

1. Install the acpid package

   The acpid service listen and processes ACPI requests.
   Log into the guest and install the acpid package on the guest:

   # yum install acpid

2. Enable the acpid service

   Set the acpid service to start during the guest boot sequence and start the service:

   # chkconfig acpid on
   # service acpid start