wiki:TipAndDoc/VM/NestedVMM

Version 18 (modified by mitty, 11 years ago) (diff)

--

  1. VMware
    1. H/W 仮想化支援
      1. VMware Workstation on Intel VT
      2. virtualize Intel VT / AMD-V
  2. KVM
    1. KVM on KVM
      1. Debian 6.0 with kernel backports
      2. nested=1

VMware

  • VMware製品には、H/W仮想化支援が無くても動作するものがあるため、適切に設定すればNested VMは簡単に実現可能
    • きちんと設定されていないと次のように表示される
      VMware Server on VMware Workstation
  • ​VMware Communities: Running Nested VMs
    • VMware製品各種での設定がまとめられている
    • ​http://communities.vmware.com/docs/DOC-8970/version/30 VMware Workstation 7以前の情報

      Running 32-bit nested VMs with binary translation under ESX(i) 4.x, ESXi 5.0, Workstation 7.x, Player 3.x or Fusion 3.x is the only officially supported configuration. Other configurations may work, but are entirely unsupported.

      Guest Hypervisors Under VMware Hypervisors

      Enabling Intel VT-x for the Outer Guest

      ESX 3.0, Workstation 6.0 and Fusion 1.0
      monitor_control.vt32 = TRUE
      Workstation 6.5 (or later), Fusion 2.0 (or later) and ESX 4.0 (or later)
      monitor.virtual_exec = hardware

      Note that this is the default for any 64-bit guest OS type on Intel hardware, so you do not have to edit the configuration file to select hardware-assisted virtualization if you have specified a 64-bit guest OS.

      Enabling AMD-V for the Outer Guest

      Family 0FH Rev F and later AMD64 processors provide hardware support for AMD-V, but current VMware products only support AMD-V on Family 10H or later AMD64 processors (Barcelona/Shanghai/Istanbul or Phenom/Phenom II). You can use AMD-V on Family 0FH Rev F and Rev G processors with ESX 3.0 and Workstation 6.0. However, you need a Family 10H or later processor to use AMD-V with any later products.

      ESX 3.0 and Workstation 6.0
      monitor_control.enable_svm = TRUE
      ESX 3.5
      monitor.virtual_mmu = hardware
      Workstation 6.5 (or later) and ESX(i) 4.0 (or later)
      monitor.virtual_exec = hardware

      Networking in Nested Guests

      see ​vmware:287 (Linux hosts) or ​vmware:1004099 (ESX hosts). No special configuration is required for Windows hosts.

      VMware Products as Guest Hypervisors

      • ESXi 5.0, you will be able to run 64-bit nested guests
      • VMWare products prior to ESXi 5.0 => you will be restricted to running only 32-bit nested guests using binary translation under any VMware product as the guest hypervisor.
      ESX(i) as a Guest Hypervisor
      • Workstation 7.0/Player 3.0/Fusion 3.0 => select "ESX Server 4"
      • ESX(i) 4.1 => manually edit the VM configuration file and select guestOS="vmkernel"
      • ESXi 5.0 => select "ESX Server 4" or "ESX Server 5"
      Other VMware Products as Guest Hypervisors

      If you plan to run VMware Tools in your inner VM, you will need an additional configuration option to restrict the backdoor interface to the host hypervisor. For guest OS types other than "ESX Server 4" or "ESX Server 5", you should add the following line to the configuration file for the outer guest: 

      monitor_control.restrict_backdoor = TRUE

      You should not install VMware Tools in the outer guest.

      Virtualizing HV (VT-x or AMD-V)

      With ESXi 5.0 as the host hypervisor, it is possible to virtualize the hardware-assisted virtualization capabilities of the physical processor. This feature is used internally within VMware for testing purposes, but it is not recommended for production systems. This feature is also not recommended for systems without hardware-assisted MMU virtualization (EPT or RVI), because of its abysmal performance without that hardware support.

      This feature is disabled by default. To permit the use of this feature, the ESXi administrator must add the following configuration option to the /etc/vmware/config file on the physical host: 

      vhv.allow = TRUE

      Once allowed by the ESXi administrator, virtualized HV will be enabled by default for all hardware version 8 VMs with guest OS type "ESX Server 4" and "ESX Server 5." To enable virtualized HV for other guests, add the following lines to the outer guest's configuration file: 

      cpuid.1.ecx="----:----:----:----:----:----:--h-:----"
cpuid.80000001.ecx.amd="----:----:----:----:----:----:----:-h--"
cpuid.8000000a.eax.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.ebx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.edx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"

      With virtual HV enabled in the outer guest, you should be able to run any guest hypervisor that requires hardware-assisted virtualization. In particular, this means that you will be able to run 64-bit nested guests under VMware guest hypervisors.

      KVM as a Guest Hypervisor

      KVM performs relatively poorly in a nested VM using virtualized VT-x. This situation is much improved with Linux kernel versions 3.0 and greater.

      VMware Guest Hypervisors Under Foreign Hypervisors

      If we detect that a VMware product is running under a foreign hypervisor, we will refuse to power on a nested VM. To bypass this constraint, add the following option to your nested VM configuration file: 

      vmx.allowNested = TRUE

H/W 仮想化支援

  • 実用的な実行速度で動かすためのハードウェア条件
    • Intel VT-x
    • AMD-v with Barcelona or Phenom or later
      • AMD-V is supported on Rev F and later Family 0FH AMD64 processors, but current VMware products only support AMD-V on Family 10H or later AMD64 processors (Barcelona or Phenom). Thus, you can use AMD-V on Family 0FH Rev F and Rev G processors with Workstation 6.0. However, you need a Family 10H processor to use AMD-V with ESX 3.5 and Workstation 6.5 (or later).
      • 手持ちの「AMD Athlon 64 X2 5000+」では「MONITOR MODE: allowed modes : BT」となり、AMD-v使用不能。
  • vmxファイルに以下の2行を追加すると可能になる(非Intel-VT, 非AMD-V環境では未テスト) 
    monitor_control.vt32 = "TRUE"
monitor_control.restrict_backdoor = "TRUE"
  • IntelCPU+x64GuestだとVTが必須なので自動的に有効になる 
    Software virtualization is incompatible with long mode on this platform. Virtual execution will begin in software mode, but will automatically switch to hardware mode if the guest enters long mode.
    • それ以外の環境では、次の1行もあった方が良いかも知れない(入れ子内のVMguestの起動が非常に遅くなり得る) 
      monitor.virtual_exec = "hardware"

VMware Workstation on Intel VT

  • Ubuntu 10.04 lucid on VMware Workstation 7.1.3 with Windows 7 x64 Host
    • Linux ubuntu32 2.6.32-26-generic #48-Ubuntu SMP Wed Nov 24 09:00:03 UTC 2010 i686 GNU/Linux
    • model name : Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz
  • 仮想マシン設定 > ハードウェア > プロセッサ > 優先モード
    • 自動
      • vmware.log 
        Dec 12 17:27:11.858: vmx| MONITOR MODE: allowed modes          : BT32 HV
Dec 12 17:27:11.858: vmx| MONITOR MODE: user requested modes   : BT32 HV HWMMU
Dec 12 17:27:11.858: vmx| MONITOR MODE: guestOS preferred modes: HWMMU BT32 HV
Dec 12 17:27:11.858: vmx| MONITOR MODE: filtered list          : BT32 HV
Dec 12 17:27:11.858: vmx| HV Settings: virtual exec = 'dynamic'; virtual mmu = 'software'
      • /proc/cpuinfo 
        flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht nx lm constant_tsc arch_perfmon pebs bts tsc_reliable aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida
    • Intel VT-xまたはAMD-V
      • diff of Ubuntu32.vmx 
        -monitor.virtual_mmu = "automatic"
-monitor.virtual_exec = "automatic"
+monitor.virtual_mmu = "software"
+monitor.virtual_exec = "hardware"
      • vmware.log 
        Dec 12 17:53:28.574: vmx| MONITOR MODE: allowed modes          : BT32 HV
Dec 12 17:53:28.574: vmx| MONITOR MODE: user requested modes   : HV
Dec 12 17:53:28.574: vmx| MONITOR MODE: guestOS preferred modes: HWMMU BT32 HV
Dec 12 17:53:28.574: vmx| MONITOR MODE: filtered list          : HV
Dec 12 17:53:28.574: vmx| HV Settings: virtual exec = 'hardware'; virtual mmu = 'software'
      • /proc/cpuinfo 
        flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht nx lm constant_tsc arch_perfmon pebs bts tsc_reliable aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida
  • 元々が「自動」だからなのか、CPU flagsに変化はなかった。

virtualize Intel VT / AMD-V

  • Intel EPT / AMD RVI 対応のCPUが必須

KVM

  • KVMはH/W仮想化支援が必須

KVM on KVM

  • Intel EPT / AMD RVI 対応のCPUが必須

Debian 6.0 with kernel backports

  • 3.x系(3.1以降？)のkernelで無いと、Intel CPUには対応出来ないらしい
  • virt-managerが古いせいか、ゲストの設定ファイル(xml)を直接編集した後、ゲストが起動しなくなった
  • mitty@walken:~$ lsb_release -a 
    No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 6.0.4 (squeeze)
Release:        6.0.4
Codename:       squeeze
  • mitty@walken:~$ uname -a 
    Linux walken 3.2.0-0.bpo.2-amd64 #1 SMP Sun Mar 25 10:33:35 UTC 2012 x86_64 GNU/Linux
  • mitty@walken:~$ vmware -v 
    VMware Workstation 8.0.2 build-591240
    • VMwareとのゲスト同時稼働も問題ない模様
  • mitty@walken:~$ kvm --version 
    QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c) 2003-2008 Fabrice Bellard
  • mitty@walken:~$ virt-manager --version 
    0.8.4
    • CPU設定に細かい設定項目がない
    • see also virt-manager
  • mitty@walken:~$ cat /proc/cpuinfo 
    processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 30
model name      : Intel(R) Core(TM) i5 CPU         750  @ 2.67GHz
stepping        : 5
microcode       : 0x3
cpu MHz         : 1197.000
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid
bogomips        : 5319.48
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

(snip)

nested=1

  • mitty@walken:~$ modinfo kvm_intel |grep nested 
    parm:           nested:bool
  • mitty@walken:~$ cat /sys/module/kvm_intel/parameters/nested 
    N
  • mitty@walken:~$ sudo vim /etc/modprobe.d/kvm-nested.conf 
    options kvm_intel nested=1
  • mitty@walken:~$ sudo modprobe -r kvm_intel
  • mitty@walken:~$ sudo modprobe kvm_intel
  • mitty@walken:~$ cat /sys/module/kvm_intel/parameters/nested 
    Y
  • mitty@walken:~$ sudo vim /etc/libvirt/qemu/test.xml

    • test.xml

      old new  
      1414    <apic/> 
      1515    <pae/> 
      1616  </features> 
       17  <cpu match='exact'> 
       18    <model>Nehalem</model> 
       19    <vendor>Intel</vendor> 
       20    <feature policy='require' name='rdtscp'/> 
       21    <feature policy='require' name='ss'/> 
       22    <feature policy='require' name='hypervisor'/> 
       23    <feature policy='require' name='vme'/> 
       24    <feature policy='require' name='ds'/> 
       25    <feature policy='require' name='vmx'/> 
       26    <feature policy='require' name='acpi'/> 
       27  </cpu> 
      1728  <clock offset='utc'/> 
      1829  <on_poweroff>destroy</on_poweroff> 
      1930  <on_reboot>restart</on_reboot> 
    • Fedora16のLiveCDでvirt-managerを起動し、下記サイトを参考にGUI上から設定を行った場合に、xmlに書き込まれる差分をそのまま適用してみた
    • うまく起動しない

Attachments (23)

Download all attachments as: .zip