Hello World,

I have recently received questions/requests about nested virtual machines within Proxmox VE. I’ve been asked to confirm that Proxmox VE 2.0 was offering nesting capabilities and how you would configure your hypervisor in order to achieve this. For people how does not know, nesting consist of virtualizing an hypervisor such as vmware or proxmox ve and have the capability to run a virtual machine inside the virtualized hypervisor.

In a previous post, we already explained how you could virtualize a Proxmox VE server.  The purpose of the article was simply to demonstrate the installation process when using Hyper-V as hypervisor solution. In the article, we mentioned that it was possible to run OpenVz container but no kvm virtual machines.

Since then, things have changed.  Newer kvm versions offer you the possibility to run nested virtual machines (whatever the cpu type you are using – in the past, there was a better support on AMD Processor).  For training and learning purposes, we have installed and deployed some virtual Proxmox VE hosts.  We never needed to have kvm virtual machines running inside these virtualized hosts.  The purpose of such implemenation was to become familiar with the Proxmox ve product and the web management interface.

This post will try to demonstrate how you can virtualize Proxmox VE and having KVM nested virtual machines running.  Howerver, note that  we do not recommend to use this technique for production environments. Use this at your own risk and do not be suprise if you have really poor performance.    

This having been said…Time to nest virtual machines.

Nesting KVM Virtual Machines

We have tested this procedure using different virtualization software solution.  We had the procedure working on the following virtual products

• Hyper-V (I’ve tested on Hyper-v 3.0)
• ESXi 5.x
• VirtualBox
• Proxmox VE 2.0

Important Note :

This procedure is quite to simple to achieve as long as you have the latest version of Proxmox VE 2.0.  I did not try with previous versions of Proxmox VE.

Nested KVM inside Proxmox VE using Proxmox VE as Hypervisor

The process is quite simple !  We will show you how you can easily test the functionality.  In this scenario, we have Proxmox VE 2.x software on a bare metal computer.  We have then created a virtual machine (kvm) and installed the Proxmox VE software on it (i.e. Virtualized Proxmox VE). Finally, after connecting to the web interface of the Virtualized PVE, we have created a kvm virtual machine.  The following drawing illustrate our scenario

Click on picture for better resolution

The following screenshot shows that we have created indeed a virtual machine that’s currently running Proxmox VE 2.0 software.

 

Click on picture for better resolution

The following screenshot confirm that we were running a virtualized Proxmox VE instance.

 

Click on picture for better resolution

Using you favorite web browser, you can connect to the web interface of the virtualized Proxmox VE server.  Within the web interface, you will see that nothing is blocking you from creating a KVM virtual machine.  After you have completed the create VM Wizard, you will see within your web interface that the nested virtual machine has been created and listed within the console.

 

Click on picture for better resolution

At this stage, if you try to boot up the nested virtual machine, you will see it booting and then shutdown almost immediately. If you look in the syslog (within the web interface, click on the PVE node), you will see a error messsage stating that Accelerator Not Found

Note:

If you have started your nested virtual machine without disabling the kvm support, you might get disconnected from the web interface of the virtualized PVE. If this is the case, you need to shutdown the virtualized PVE and restart it and make the setting change on the nested virtual machine

In order to have the nested kvm machine running, you will need to perform a small change in the kvm configuration.  To perform this change,perform the following actions

Locate the virtual machine, click on it. In the right pane, click on the Options tab.  Click on the KVM Virtualization Hardware support setting

 

Click on picture for better resolution

Click on the Edit button and deselect the checkbox.

Click on picture for better resolution

The setting has to show the value no.

 

Click on picture for better resolution

When this change has been done, you can start up your nested virtual machine.  If everything goes fine, you would see your nested virtual machine booting up and you can proceed with the installation of the OS.

Nested KVM inside Proxmox VE using other Hypervisor than Proxmox VE

If you have installed ESXi as your main physical Hypervisor,  you can create the Proxmox VE virtual machine and have a nested kvm virtual machine running as long as you configure the KVM Virtualization hardware support as described above.  I had configure my ESXi 5 with Hardware virtualization support enabled (by issuing the command echo ‘vhv.allow – “True”  >>/etc/vmware/config).  I do not know is this is needed or not…

If you have installed Hyper-V 3.0; simply follow the installation instruction found here. Again, ensure that the KVM Virtualization hardware support on the nested virtual machine is configured correctly (i.e. Set to No) and you should be able to have it started.

If you have installed VirtualBox as virtualization solution, you can also virtualize the Proxmox VE and having nested kvm virtual machine running as long as you disable the KVM virtualization hardware support within the virtual Proxmox VE.  The Proxmox VE virtual machine should be configured as required within the virtualBox software.

Final Words

As you can see, nesting virtual machines using Proxmox VE as Hypervisor is really simple.  You do not have to edit any configuration files. You simply create your virtual Proxmox VE instance. Then, you create your nested virtual machine. Finally, you will need to ensure that the hardware virtualization is disabled on the nested virtual machine.   The surprising thing is that this recipice works on Hyper-V, Esxi and VirtualBox. 

The KVM version included in the Proxmox VE 2.0 offers indeed support for nestedvirtual machines.  However, do not expect to have super fast nested virtual machines.  Performance are generally not that good…. We do not recommend this for production environments. You can try to have nested virtual machine but do not expect to have blazing performance.  Use this if you need to study or give training about Proxmox VE and if you do not have other choices.

Till next Time

See ya