wiki:TipAndDoc/VM/KVM

Version 58 (modified by mitty, 9 years ago) (diff)

--

  1. KVM
    1. Client
  2. Windows Guest
    1. drivers
      1. block device
    2. Windows 7 x64 with virtio
    3. Windows 10
      1. Heavy load on playing sound
    4. power efficiency
  3. virbr0
    1. iptables
      1. port forwarding
  4. Live Migration
  5. kvm-spice
    1. transfer rate
  6. start/shutdown log for VM
    1. autoshutdown
  7. passthrough
    1. Marvell controller
      1. Plextor M6e
      2. patch
  8. Clock
  9. QoS
  10. OVMF

KVM

  • KVM自体にはIntel-VT or AMD-V が必須 
    egrep '(vmx|svm)' --color=always /proc/cpuinfo
  • QEMUはVT/AMD-vが無くても動く(ただし非常に遅い)

Client

Windows Guest

drivers

  • デバイスマネージャ

block device

  • virtio-win-1.1.16.vfdを用いてVirtIO diskでセットアップ後、virtio-win-0.1-30.isoによってドライバを更新するとBSODを出して起動しなくなる。
    • OSセットアップ直後
    • ドライバ更新
    • BSOD

Windows 7 x64 with virtio

  • H/W
    • ASUS H97M-PLUS
    • Core i7-4790 3.60GHz
    • W3U1600HQ-8G (DDR3-1600 8GB x2)
    • RAID1 on WD60EFRX-68MYMN1 x2
  • S/W
    • ArchLinux 4.0.4-2-ARCH x86_64
    • qemu 2.3.0-2
    • seabios 1.8.1-1
    • libvirt 1.2.15-1
    • virt-manager 1.2.0-2
    • spice 0.12.5-1
    • virtio-win-0.1.104.iso

Windows 10

Heavy load on playing sound

  • ver.1511 (TH2)で音声を再生すると、ホストCPUに非常に高い負荷がかかる
    • ,
Windows 10 TH1 mp3 playingWindows 10 TH2 no soundWindows 10 TH2 mp3 playing

power efficiency

  • H/W: wiki:TipAndDoc/Hardware#ASUSH97M-PLUS
  • S/W
    • ArchLinux 4.2.5-1-ARCH x86_64
    • qemu 2.4.0.1-1
    • seabios 1.8.2-1
    • libvirt 1.2.20-3
    • virt-manager 1.2.1-1
    • spice 0.12.6-1
    • virtio-win 0.1.110.2-1
  • VM
    • CPU: 2core 2HT
    • Mem: 8GB
    • HDD: qcow2 60GB
  1. no VM (idle): 39W～40W
  2. Win 7 x64 SP1 (idle): 40W
  3. Win 7 x64 SP1 (Chrome x64 with 艦これ): 48W～50W
    • リモートデスクトップ接続
    • 音飛び(ブツブツ音)が若干入る(RDP 8にアップグレードするも効果なし)
  4. Win 10 x64 (idle): 40W
  5. Win 10 x64 (Chrome x64 with 艦これ): 49W～54W
    • リモートデスクトップ接続

virbr0

  • libvirtdによって自動で作られる「default」のNAT
    • ps aux | grep dns 
      nobody    2684  0.0  0.0  21264   792 ?        S    Feb07   0:01 dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file=  --listen-address 192.168.122.1 --except-interface lo --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-lease-max=253
    • lease file => /var/lib/misc/dnsmasq.leases

iptables

  • sudo iptables-save 
    # Generated by iptables-save v1.4.8 on Thu Feb 16 10:31:29 2012
*nat
:PREROUTING ACCEPT [743892:129452770]
:POSTROUTING ACCEPT [1179:112796]
:OUTPUT ACCEPT [1215:115607]
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE 
COMMIT
# Completed on Thu Feb 16 10:31:29 2012
# Generated by iptables-save v1.4.8 on Thu Feb 16 10:31:29 2012
*filter
:INPUT ACCEPT [4957020:3457557240]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3323525:4327643495]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT 
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 
COMMIT
# Completed on Thu Feb 16 10:31:29 2012

port forwarding

Live Migration

kvm-spice

  • Ubuntu 12.04でqemu-kvm-spiceをインストールすると、デフォルトのemulatorが/usr/bin/kvm-spiceになって以下の様な問題が起きる
  • kern.logにAppArmorからDENIEDとエラーが出る 
    Oct 13 22:45:08 HOSTNAME kernel: [507466.445098] type=1400 audit(1350135908.372:69): apparmor="DENIED" operation="open" parent=1 profile="libvirt-52d08545-a490-c890-654a-260caf35d27d" name="/proc/17685/auxv" pid=17685 comm="kvm-spice" requested_mask="r" denied_mask="r" fsuid=104 ouid=104
  • Video modelをcirrus以外にすると、画面が表示されなくなる(真っ黒のまま)

transfer rate

  • ネットワーク転送量がRDPに比較して極端に多くなるので注意
    • RDP: 100Kbps以下 (idle) ～ 10Mbps (動画再生)
    • SPICE: 100Kbps以下 (idle) ～ 100Mbps以上 (動画再生)、400Mbps近くなることもあった

start/shutdown log for VM

  • Ubuntu 12.04
  • /var/log/libvirt/qemu 以下に、VMごとのログが残される 
    -rw------- 1 root root  3940 Oct  8 00:24 quantal.log
-rw------- 1 root root 12394 Oct 13 23:13 test.log
  • /var/log/libvirt/qemu/xp-vm01.log 
    2012-10-28 12:46:55.328+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name xp-vm01 -uuid XXXXXXXX-YYYY-ZZZZ-WWWW-VVVVVVVVVVVV -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/xp-vm01.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -drive file=/var/lib/libvirt/images/xp-vm01.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive file=/home/mitty/virtio-win-0.1-30.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev tap,fd=16,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a8:d0:1d,bus=pci.0,addr=0x3 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -vga vmware -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
vmsvga_value_read: Bad register 2b
vmsvga_value_read: Bad register 2c
qemu: terminating on signal 15 from pid 1803
2012-10-28 12:51:56.141+0000: shutting down

autoshutdown

  • /var/log/libvirt/shutdownlog.log で確認できる 
    libvirt: libvirt-bin: entering pre-stop at Fri Oct 19 23:22:33 JST 2012
libvirt: libvirt-bin: attempting clean shutdown of xp-vm02 at Fri Oct 19 23:22:33 JST 2012
libvirt: destroying xp-vm02
libvirt: libvirt-bin: exiting pre-stop at Fri Oct 19 23:23:06 JST 2012
libvirt: libvirt-bin: entering pre-stop at Fri Oct 26 23:37:35 JST 2012
libvirt: libvirt-bin: exiting pre-stop at Fri Oct 26 23:37:35 JST 2012
libvirt: libvirt-bin: entering pre-stop at Sun Oct 28 21:45:17 JST 2012
libvirt: libvirt-bin: attempting clean shutdown of xp-vm02 at Sun Oct 28 21:45:17 JST 2012
libvirt: libvirt-bin: attempting clean shutdown of xp-vm01 at Sun Oct 28 21:45:17 JST 2012
libvirt: libvirt-bin: exiting pre-stop at Sun Oct 28 21:45:45 JST 2012
  • /etc/init/libvirt-bin.conf で制御されている模様

passthrough

  • QEMUの引数に追加されるものの例 
    01:00.0 Multimedia controller [0480]: Altera Corporation Device [1172:4c15] (rev 01)

    -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9

Marvell controller

Plextor M6e

  • PX-G128M6e
    • lspci -nnv 
      02:00.0 SATA controller [0106]: Lite-On IT Corp. / Plextor M6e PCI Express SSD [Marvell 88SS9183] [1c28:0122] (rev 14) (prog-if 01 [AHCI 1.0])
        Subsystem: Marvell Technology Group Ltd. Device [1b4b:9183]
  • root@archiso ~ # uname -a 
    Linux archiso 4.0.4-2-ARCH #1 SMP PREEMPT Fri May 22 03:05:23 UTC 2015 x86_64 GNU/Linux
  • root@archiso ~ # cat /proc/cmdline 
    BOOT_IMAGE=boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201506 initrd=boot/intel_ucode.img,boot/x86_64/archiso.img intel_iommu=on
  • root@archiso ~ # dmesg|grep -i -e DMAR -e IOMMU 
    [    0.000000] Command line: BOOT_IMAGE=boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201506 initrd=boot/intel_ucode.img,boot/x86_64/archiso.img intel_iommu=on
[    0.000000] ACPI: DMAR 0x00000000DAC6CED0 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Kernel command line: BOOT_IMAGE=boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201506 initrd=boot/intel_ucode.img,boot/x86_64/archiso.img intel_iommu=on
[    0.000000] Intel-IOMMU: enabled
[    0.086964] dmar: Host address width 39
[    0.086969] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[    0.086987] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.086990] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[    0.087000] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.087003] dmar: RMRR base: 0x000000dbe7b000 end: 0x000000dbe89fff
[    0.087006] dmar: RMRR base: 0x000000dd000000 end: 0x000000df1fffff
[    0.087011] IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    7.370829] DMAR: No ATSR found
[    7.371048] IOMMU: dmar0 using Queued invalidation
[    7.371051] IOMMU: dmar1 using Queued invalidation
[    7.371056] IOMMU: Setting RMRR:
[    7.371083] IOMMU: Setting identity map for device 0000:00:02.0 [0xdd000000 - 0xdf1fffff]
[    7.371527] IOMMU: Setting identity map for device 0000:00:14.0 [0xdbe7b000 - 0xdbe89fff]
[    7.371577] IOMMU: Setting identity map for device 0000:00:1a.0 [0xdbe7b000 - 0xdbe89fff]
[    7.371616] IOMMU: Setting identity map for device 0000:00:1d.0 [0xdbe7b000 - 0xdbe89fff]
[    7.371644] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    7.371661] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    7.673577] [drm] DMAR active, disabling use of stolen memory
[    7.673740] dmar: DRHD: handling fault status reg 2
[    7.673744] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[    7.986473] dmar: DRHD: handling fault status reg 3
[    7.986493] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[    9.337664] dmar: DRHD: handling fault status reg 2
[    9.337683] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 74ff240000
               DMAR:[fault reason 05] PTE Write access is not set
[   12.985456] dmar: DRHD: handling fault status reg 2
[   12.985519] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   13.298243] dmar: DRHD: handling fault status reg 3
[   13.298307] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   18.297124] dmar: DRHD: handling fault status reg 2
[   18.297152] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   18.609924] dmar: DRHD: handling fault status reg 3
[   18.609971] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   23.608862] dmar: DRHD: handling fault status reg 2
[   23.608924] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
  • PX-G128M6eを取り外すと以下のようになる 
    [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=0f566cb2-aec8-4122-a80a-f2e5f48e4426 rw intel_iommu=on
[    0.000000] ACPI: DMAR 0x00000000DAC6CED0 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=0f566cb2-aec8-4122-a80a-f2e5f48e4426 rw intel_iommu=on
[    0.000000] Intel-IOMMU: enabled
[    0.107383] dmar: Host address width 39
[    0.107388] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[    0.107406] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.107409] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[    0.107419] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.107422] dmar: RMRR base: 0x000000dbe7b000 end: 0x000000dbe89fff
[    0.107425] dmar: RMRR base: 0x000000dd000000 end: 0x000000df1fffff
[    0.107430] IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    0.677861] DMAR: No ATSR found
[    0.678093] IOMMU: dmar0 using Queued invalidation
[    0.678096] IOMMU: dmar1 using Queued invalidation
[    0.678100] IOMMU: Setting RMRR:
[    0.678127] IOMMU: Setting identity map for device 0000:00:02.0 [0xdd000000 - 0xdf1fffff]
[    0.678608] IOMMU: Setting identity map for device 0000:00:14.0 [0xdbe7b000 - 0xdbe89fff]
[    0.678655] IOMMU: Setting identity map for device 0000:00:1a.0 [0xdbe7b000 - 0xdbe89fff]
[    0.678697] IOMMU: Setting identity map for device 0000:00:1d.0 [0xdbe7b000 - 0xdbe89fff]
[    0.678725] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.678743] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    4.326005] [drm] DMAR active, disabling use of stolen memory

patch

  • ​LKML: Tim Sander: (PATCH) 3.19-rc7: add quirk for 1c28:0122 (rev 14) SATA controller
  • linux-stable-v4.0.5.patch 
    diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 85f247e..a78b225 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3595,6 +3595,10 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_TTI, 0x0642,
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_JMICRON,
 			 PCI_DEVICE_ID_JMICRON_JMB388_ESD,
 			 quirk_dma_func1_alias);
+/* https://bugzilla.kernel.org/show_bug.cgi?id=42679#c94 */
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_LITE_ON,
+			 PCI_DEVICE_ID_PLEXTOR_M6E,
+			 quirk_dma_func1_alias);
 
 /*
  * Some devices DMA with the wrong devfn, not just the wrong function.
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index e63c02a..1607b20 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2487,6 +2487,9 @@
 
 #define PCI_VENDOR_ID_ASMEDIA		0x1b21
 
+#define PCI_VENDOR_ID_LITE_ON		0x1c28
+#define PCI_DEVICE_ID_PLEXTOR_M6E	0x0122
+
 #define PCI_VENDOR_ID_CIRCUITCO		0x1cc8
 #define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD	0x0001

Clock

QoS

  • ​libvirt: Domain XML format
  • ex) 
        <interface type='network'>
      <mac address='--:--:--:--:--:--'/>
      <source network='network'/>
      <bandwidth>
        <inbound average='1024' peak='1024' burst='1024'/>
        <outbound average='1024' peak='1024' burst='1024'/>
      </bandwidth>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    • virt-managerでは設定できないが、virt-managerでNICの設定を変更しても<bandwidth></bandwidth>の部分は維持されるので、virsh editで設定すればよい
  • Virtual Network設定で指定できるQoS機能は、ざっと試した限りでは反映されない感じだった

OVMF

  • mitty@kvmhost:~ (master)$ pacaur -S ovmf 
    ovmf /usr/
ovmf /usr/share/
ovmf /usr/share/licenses/
ovmf /usr/share/licenses/ovmf/
ovmf /usr/share/licenses/ovmf/License.txt
ovmf /usr/share/ovmf/
ovmf /usr/share/ovmf/ovmf_ia32.bin
ovmf /usr/share/ovmf/ovmf_x64.bin
  • mitty@kvmhost:~/works (master)$ wget ​https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20160316.b1616.g6fe9d9c.noarch.rpm
  • mitty@kvmhost:~/works (master)$ wget ​https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-ia32-0-20160316.b1616.g6fe9d9c.noarch.rpm
  • mitty@kvmhost:~/works (master)$ rpmextract.sh edk2.git-ovmf-x64-0-20160316.b1616.g6fe9d9c.noarch.rpm
  • mitty@kvmhost:~/works (master)$ rpmextract.sh edk2.git-ovmf-ia32-0-20160316.b1616.g6fe9d9c.noarch.rpm
  • mitty@kvmhost:~/works (master)$ sudo cp -R --preserve=timestamp usr/share/* /usr/share/
  • mitty@kvmhost:~ (master)$ sudo vim /etc/libvirt/qemu.conf

    • etc/libvirt/qemu.conf

      a b group="78" 
      515515#   "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", 
      516516#   "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd" 
      517517#] 
       518nvram = [ 
       519   "/usr/share/edk2.git/ovmf-ia32/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-ia32/OVMF_VARS-pure-efi.fd", 
       520   "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" 
       521] 
      518522 
      519523# The backend to use for handling stdout/stderr output from 
      520524# QEMU processes. 
  • EFI自体は起動するが、OSのブートができない (archlinux-2016.03.01-dual.iso, Win10_1511_Japanese_x64.isoでテスト)

Attachments (17)

Download all attachments as: .zip