[[PageOutline]]

 * [[Image(nestedvmm.png, 33%)]]

 = VMware =
 * VMware製品には、H/W仮想化支援が無くても動作するものがあるため、適切に設定すればNested VMは簡単に実現可能
  * きちんと設定されていないと次のように表示される[[br]][[Image(vmsvonvmws.png, 33%)]]

 * [http://communities.vmware.com/docs/DOC-8970 VMware Communities: Running Nested VMs]
  * VMware製品各種での設定がまとめられている
  * http://communities.vmware.com/docs/DOC-8970/version/30 VMware Workstation 7以前の情報
  > Running 32-bit nested VMs with binary translation under ESX(i) 4.x, ESXi 5.0, Workstation 7.x, Player 3.x or Fusion 3.x is the only officially supported configuration. Other configurations may work, but are entirely unsupported.
  > === Guest Hypervisors Under VMware Hypervisors ===
  > ==== Enabling Intel VT-x for the Outer Guest ====
  > ===== ESX 3.0, Workstation 6.0 and Fusion 1.0 =====
  > {{{
  > monitor_control.vt32 = TRUE
  > }}}
  > ===== Workstation 6.5 (or later), Fusion 2.0 (or later) and ESX 4.0 (or later) =====
  > {{{
  > monitor.virtual_exec = hardware
  > }}}
  > Note that this is the default for any 64-bit guest OS type on Intel hardware, so you do not have to edit the configuration file to select hardware-assisted virtualization if you have specified a 64-bit guest OS.
  > ==== Enabling AMD-V for the Outer Guest ====
  > Family 0FH Rev F and later AMD64 processors provide hardware support for AMD-V, but current VMware products only support AMD-V on Family 10H or later AMD64 processors (Barcelona/Shanghai/Istanbul or Phenom/Phenom II). You can use AMD-V on Family 0FH Rev F and Rev G processors with ESX 3.0 and Workstation 6.0. However, you need a Family 10H or later processor to use AMD-V with any later products.
  > ===== ESX 3.0 and Workstation 6.0 =====
  > {{{
  > monitor_control.enable_svm = TRUE
  > }}}
  > ===== ESX 3.5 =====
  > {{{
  > monitor.virtual_mmu = hardware
  > }}}
  > ===== Workstation 6.5 (or later) and ESX(i) 4.0 (or later) =====
  > {{{
  > monitor.virtual_exec = hardware
  > }}}
  > ==== Networking in Nested Guests ====
  > see vmware:287 (Linux hosts) or vmware:1004099 (ESX hosts). No special configuration is required for Windows hosts.
  > ==== VMware Products as Guest Hypervisors ====
  > * ESXi 5.0, you will be able to run 64-bit nested guests 
  > * VMWare products prior to ESXi 5.0 => you will be restricted to running only 32-bit nested guests using binary translation under any VMware product as the guest hypervisor.
  > ===== ESX(i) as a Guest Hypervisor =====
  > * Workstation 7.0/Player 3.0/Fusion 3.0 => select "ESX Server 4"
  > * ESX(i) 4.1 => manually edit the VM configuration file and select guestOS="vmkernel"
  > * ESXi 5.0 => select "ESX Server 4" or "ESX Server 5"
  > ===== Other VMware Products as Guest Hypervisors =====
  > If you plan to run VMware Tools in your inner VM, you will need an additional configuration option to restrict the backdoor interface to the host hypervisor. For guest OS types other than "ESX Server 4" or "ESX Server 5", you should add the following line to the configuration file for the outer guest:
  > {{{
  > monitor_control.restrict_backdoor = TRUE
  > }}}
  > You should not install VMware Tools in the outer guest.
  > ==== Virtualizing HV (VT-x or AMD-V) ====
  > With ESXi 5.0 as the host hypervisor, it is possible to virtualize the hardware-assisted virtualization capabilities of the physical processor. This feature is used internally within VMware for testing purposes, but it is not recommended for production systems. This feature is also not recommended for systems without hardware-assisted MMU virtualization (EPT or RVI), because of its abysmal performance without that hardware support.
  > 
  > This feature is disabled by default. To permit the use of this feature, the ESXi administrator must add the following configuration option to the /etc/vmware/config file on the physical host:
  > {{{
  > vhv.allow = TRUE
  > }}}
  > Once allowed by the ESXi administrator, virtualized HV will be enabled by default for all hardware version 8 VMs with guest OS type "ESX Server 4" and "ESX Server 5." To enable virtualized HV for other guests, add the following lines to the outer guest's configuration file:
  > {{{
  > cpuid.1.ecx="----:----:----:----:----:----:--h-:----"
  > cpuid.80000001.ecx.amd="----:----:----:----:----:----:----:-h--"
  > cpuid.8000000a.eax.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
  > cpuid.8000000a.ebx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
  > cpuid.8000000a.edx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
  > }}}
  > With virtual HV enabled in the outer guest, you should be able to run any guest hypervisor that requires hardware-assisted virtualization. In particular, this means that you will be able to run 64-bit nested guests under VMware guest hypervisors.
  > ===== KVM as a Guest Hypervisor =====
  > KVM performs relatively poorly in a nested VM using virtualized VT-x. This situation is much improved with Linux kernel versions 3.0 and greater.
  > === VMware Guest Hypervisors Under Foreign Hypervisors ===
  > If we detect that a VMware product is running under a foreign hypervisor, we will refuse to power on a nested VM. To bypass this constraint, add the following option to your nested VM configuration file:
  > {{{
  > vmx.allowNested = TRUE
  > }}}

 == H/W 仮想化支援 ==
 * 実用的な実行速度で動かすためのハードウェア条件
   * Intel VT-x
   * AMD-v with Barcelona or Phenom or later
     * AMD-V is supported on Rev F and later Family 0FH AMD64 processors, but current VMware products only support AMD-V on Family 10H or later AMD64 processors (Barcelona or Phenom). Thus, you can use AMD-V on Family 0FH Rev F and Rev G processors with Workstation 6.0. However, you need a Family 10H processor to use AMD-V with ESX 3.5 and Workstation 6.5 (or later).
     * 手持ちの「AMD Athlon 64 X2 5000+」では「MONITOR MODE: allowed modes : BT」となり、AMD-v使用不能。

 * vmxファイルに以下の2行を追加すると可能になる(非Intel-VT, 非AMD-V環境では未テスト)
{{{
monitor_control.vt32 = "TRUE"
monitor_control.restrict_backdoor = "TRUE"
}}}
 * IntelCPU+x64GuestだとVTが必須なので自動的に有効になる
{{{
Software virtualization is incompatible with long mode on this platform. Virtual execution will begin in software mode, but will automatically switch to hardware mode if the guest enters long mode.
}}}
   * それ以外の環境では、次の1行もあった方が良いかも知れない(入れ子内のVMguestの起動が非常に遅くなり得る)
{{{
monitor.virtual_exec = "hardware"
}}}

 === VMware Workstation on Intel VT ===
 * Ubuntu 10.04 lucid on VMware Workstation 7.1.3 with Windows 7 x64 Host
  * Linux ubuntu32 2.6.32-26-generic !#48-Ubuntu SMP Wed Nov 24 09:00:03 UTC 2010 i686 GNU/Linux
  * model name      : Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
 * 仮想マシン設定 > ハードウェア > プロセッサ > 優先モード
  * 自動
   * vmware.log
{{{
Dec 12 17:27:11.858: vmx| MONITOR MODE: allowed modes          : BT32 HV
Dec 12 17:27:11.858: vmx| MONITOR MODE: user requested modes   : BT32 HV HWMMU
Dec 12 17:27:11.858: vmx| MONITOR MODE: guestOS preferred modes: HWMMU BT32 HV
Dec 12 17:27:11.858: vmx| MONITOR MODE: filtered list          : BT32 HV
Dec 12 17:27:11.858: vmx| HV Settings: virtual exec = 'dynamic'; virtual mmu = 'software'
}}}
   * /proc/cpuinfo
{{{
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht nx lm constant_tsc arch_perfmon pebs bts tsc_reliable aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida
}}}
  * Intel VT-xまたはAMD-V
   * diff of Ubuntu32.vmx
{{{
-monitor.virtual_mmu = "automatic"
-monitor.virtual_exec = "automatic"
+monitor.virtual_mmu = "software"
+monitor.virtual_exec = "hardware"
}}}
   * vmware.log
{{{
Dec 12 17:53:28.574: vmx| MONITOR MODE: allowed modes          : BT32 HV
Dec 12 17:53:28.574: vmx| MONITOR MODE: user requested modes   : HV
Dec 12 17:53:28.574: vmx| MONITOR MODE: guestOS preferred modes: HWMMU BT32 HV
Dec 12 17:53:28.574: vmx| MONITOR MODE: filtered list          : HV
Dec 12 17:53:28.574: vmx| HV Settings: virtual exec = 'hardware'; virtual mmu = 'software'
}}}
   * /proc/cpuinfo
{{{
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht nx lm constant_tsc arch_perfmon pebs bts tsc_reliable aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida
}}}

 * 元々が「自動」だからなのか、CPU flagsに変化はなかった。

 * [http://www.grc.com/securable.htm GRC | SecurAble: Determine Processor Security Features |  ][[br]][[Image(WindowsXPonVM.png, 33%)]]

 === virtualize Intel VT / AMD-V ===
 * Intel EPT / AMD RVI 対応のCPUが必須[[br]][[Image(vmware-virtualize.png,33%)]]

 * ゲストのDebian, Ubuntuなどで、「virsh -c !qemu:///system」が刺さって帰ってこない場合は、VMの設定ファイル(.vmx)に以下の文を設定
{{{
vcpu.hotadd = FALSE
}}}
  * [http://communities.vmware.com/docs/DOC-8970 VMware Communities: Running Nested VMs]
  * [http://sceneryandfish.withnotes.net/?p=812 Scenery and Fish » VMware WorkstationでNested KVM環境]

 * 使用したゲストOSはFedora 16 LiveCD
   * from http://ftp.tsukuba.wide.ad.jp/Linux/fedora/linux/releases/16/Live/x86_64/ Fedora-16-x86_64-Live-Desktop.iso
 * CPU flagsにvmxが表れる[[br]][[Image(vmware-vmx-enabled.png,33%)]]
 * VMware ゲストOS上でKVMが使用可能、さらにKVMゲストへ仮想化支援をネスト出来る[[br]][[Image(vmware-virt-manager-kvm.png,33%)]]
  * Fedora16だと仮想化支援を仮想化する設定はvirt-managerのGUI上から可能
  * see also [#KVMonKVM], source:/lab.git/TipAndDoc/kvm/test.xml
 * Core i5(物理マシン)上のVMwareゲスト(Fedora 16)上でKVMを動かし、その上でDebian Installerが仮想化支援有効で動作している[[br]][[Image(vmware-kvm-vmx-enabled.png,33%)]]
  * 一番内側のVMで使用したiso => http://ftp.tsukuba.wide.ad.jp/Linux/debian/dists/stable/main/installer-amd64/current/images/netboot/ mini.iso
 = KVM =
 * KVMはH/W仮想化支援が必須

 == KVM on KVM ==
 * Intel EPT / AMD RVI 対応のCPUが必須

 * [http://aikotobaha.blogspot.jp/2012/02/kvm-on-kvmnested-kvm.html OSSはアルミニウムの翼で飛ぶ: KVM on KVM(nested KVM)]
 * [http://baalzephon.dyndns.org/tech/index.php?Linux%2FKVM Linux/KVM - PukiWiki Plus!]
 * [http://d.hatena.ne.jp/pochio_pochiyama/20110228/p1 Ubuntu 10.10 で KVM on KVM (nested svm) - それなりに適当にやってます]

 * 「nested=1」は3.1以上のカーネルが必要
  * [https://lkml.org/lkml/2011/7/24/52 LKML: Avi Kivity: (GIT PULL) KVM updates for the Linux 3.1 merge window]
 === Debian 6.0 with kernel backports ===
 * 3.x系(3.1以降？)のkernelで無いと、Intel CPUには対応出来ないらしい
  * see wiki:TipAndDoc/Linux/Debian#backports
 * virt-managerが古いせいか、ゲストの設定ファイル(xml)を直接編集した後、ゲストが起動しなくなった

 * mitty@walken:~$ lsb_release -a
{{{
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 6.0.4 (squeeze)
Release:        6.0.4
Codename:       squeeze
}}}
 * mitty@walken:~$ uname -a
{{{
Linux walken 3.2.0-0.bpo.2-amd64 #1 SMP Sun Mar 25 10:33:35 UTC 2012 x86_64 GNU/Linux
}}}

 * mitty@walken:~$ vmware -v
{{{
VMware Workstation 8.0.2 build-591240
}}}
  * VMwareとのゲスト同時稼働も問題ない模様

 * mitty@walken:~$ kvm --version
{{{
QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c) 2003-2008 Fabrice Bellard
}}}

 * mitty@walken:~$ virt-manager --version
{{{
0.8.4
}}}
  * CPU設定に細かい設定項目がない[[br]][[Image(virt-manager-processor.png,33%)]]
  * see also [../virt-manager#useversion0.9.1onDebian6.0]

 * mitty@walken:~$ cat /proc/cpuinfo
{{{
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 30
model name      : Intel(R) Core(TM) i5 CPU         750  @ 2.67GHz
stepping        : 5
microcode       : 0x3
cpu MHz         : 1197.000
cache size      : 8192 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid
bogomips        : 5319.48
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

(snip)
}}}
 === nested=1 ===
 * mitty@walken:~$ modinfo kvm_intel |grep nested
{{{
parm:           nested:bool
}}}
 * mitty@walken:~$ cat /sys/module/kvm_intel/parameters/nested
{{{
N
}}}

 * mitty@walken:~$ sudo vim /etc/modprobe.d/kvm-nested.conf
{{{
options kvm_intel nested=1
}}}

 * mitty@walken:~$ sudo modprobe -r kvm_intel
 * mitty@walken:~$ sudo modprobe kvm_intel
 * mitty@walken:~$ cat /sys/module/kvm_intel/parameters/nested
{{{
Y
}}}

 * mitty@walken:~$ sudo vim /etc/libvirt/qemu/test.xml
{{{#!diff
--- test.xml	2012-04-17 12:36:32.464246092 +0900
+++ /etc/libvirt/qemu/test.xml	2012-04-17 14:11:45.439356087 +0900
@@ -14,6 +14,17 @@
     <apic/>
     <pae/>
   </features>
+  <cpu match='exact'>
+    <model>Nehalem</model>
+    <vendor>Intel</vendor>
+    <feature policy='require' name='rdtscp'/>
+    <feature policy='require' name='ss'/>
+    <feature policy='require' name='hypervisor'/>
+    <feature policy='require' name='vme'/>
+    <feature policy='require' name='ds'/>
+    <feature policy='require' name='vmx'/>
+    <feature policy='require' name='acpi'/>
+  </cpu>
   <clock offset='utc'/>
   <on_poweroff>destroy</on_poweroff>
   <on_reboot>restart</on_reboot>
}}}
  * Fedora16のLiveCDでvirt-managerを起動し、下記サイトを参考にGUI上から設定を行った場合に、xmlに書き込まれる差分をそのまま適用してみた
  * うまく起動しない[[br]][[Image(xml-error.png,33%)]]