wiki:TipAndDoc/VM/KVM

Version 60 (modified by mitty, 8 years ago) (diff)

--

  1. KVM
    1. Client
  2. Windows Guest
    1. drivers
      1. block device
    2. Windows 7 x64 with virtio
    3. Windows 10
      1. Heavy load on playing sound
        1. Heavy load problem is fixed
    4. power efficiency
  3. virbr0
    1. iptables
      1. port forwarding
  4. Live Migration
  5. kvm-spice
    1. transfer rate
  6. start/shutdown log for VM
    1. autoshutdown
  7. passthrough
    1. Marvell controller
      1. Plextor M6e
      2. patch
  8. Clock
  9. QoS
  10. OVMF

KVM

  • KVM自体にはIntel-VT or AMD-V が必須 
    egrep '(vmx|svm)' --color=always /proc/cpuinfo
  • QEMUはVT/AMD-vが無くても動く(ただし非常に遅い)

Client

Windows Guest

drivers

  • デバイスマネージャ

block device

  • virtio-win-1.1.16.vfdを用いてVirtIO diskでセットアップ後、virtio-win-0.1-30.isoによってドライバを更新するとBSODを出して起動しなくなる。
    • OSセットアップ直後
    • ドライバ更新
    • BSOD

Windows 7 x64 with virtio

  • H/W
    • ASUS H97M-PLUS
    • Core i7-4790 3.60GHz
    • W3U1600HQ-8G (DDR3-1600 8GB x2)
    • RAID1 on WD60EFRX-68MYMN1 x2
  • S/W
    • ArchLinux 4.0.4-2-ARCH x86_64
    • qemu 2.3.0-2
    • seabios 1.8.1-1
    • libvirt 1.2.15-1
    • virt-manager 1.2.0-2
    • spice 0.12.5-1
    • virtio-win-0.1.104.iso

Windows 10

Heavy load on playing sound

  • ver.1511 (TH2)で音声を再生すると、ホストCPUに非常に高い負荷がかかる 解消された模様(後述)
    • ,
Windows 10 TH1 mp3 playingWindows 10 TH2 no soundWindows 10 TH2 mp3 playing

Heavy load problem is fixed

  • 以下の環境下では発生しなくなっていた
  • ホスト
    • ArchLinux 4.6.2-1
    • qemu 2.6.0-1
    • seabios 1.9.2-1
    • libvirt 1.3.5-1
    • virt-manager 1.3.2-5
    • spice 0.12.7-1
    • virtio-win 0.1.118.1-1
  • ゲスト

power efficiency

  • H/W: wiki:TipAndDoc/Hardware#ASUSH97M-PLUS
  • S/W
    • ArchLinux 4.2.5-1-ARCH x86_64
    • qemu 2.4.0.1-1
    • seabios 1.8.2-1
    • libvirt 1.2.20-3
    • virt-manager 1.2.1-1
    • spice 0.12.6-1
    • virtio-win 0.1.110.2-1
  • VM
    • CPU: 2core 2HT
    • Mem: 8GB
    • HDD: qcow2 60GB
  1. no VM (idle): 39W～40W
  2. Win 7 x64 SP1 (idle): 40W
  3. Win 7 x64 SP1 (Chrome x64 with 艦これ): 48W～50W
    • リモートデスクトップ接続
    • 音飛び(ブツブツ音)が若干入る(RDP 8にアップグレードするも効果なし)
  4. Win 10 x64 (idle): 40W
  5. Win 10 x64 (Chrome x64 with 艦これ): 49W～54W
    • リモートデスクトップ接続

virbr0

  • libvirtdによって自動で作られる「default」のNAT
    • ps aux | grep dns 
      nobody    2684  0.0  0.0  21264   792 ?        S    Feb07   0:01 dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file=  --listen-address 192.168.122.1 --except-interface lo --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-lease-max=253
    • lease file => /var/lib/misc/dnsmasq.leases

iptables

  • sudo iptables-save 
    # Generated by iptables-save v1.4.8 on Thu Feb 16 10:31:29 2012
*nat
:PREROUTING ACCEPT [743892:129452770]
:POSTROUTING ACCEPT [1179:112796]
:OUTPUT ACCEPT [1215:115607]
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE 
COMMIT
# Completed on Thu Feb 16 10:31:29 2012
# Generated by iptables-save v1.4.8 on Thu Feb 16 10:31:29 2012
*filter
:INPUT ACCEPT [4957020:3457557240]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3323525:4327643495]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT 
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 
COMMIT
# Completed on Thu Feb 16 10:31:29 2012

port forwarding

Live Migration

kvm-spice

  • Ubuntu 12.04でqemu-kvm-spiceをインストールすると、デフォルトのemulatorが/usr/bin/kvm-spiceになって以下の様な問題が起きる
  • kern.logにAppArmorからDENIEDとエラーが出る 
    Oct 13 22:45:08 HOSTNAME kernel: [507466.445098] type=1400 audit(1350135908.372:69): apparmor="DENIED" operation="open" parent=1 profile="libvirt-52d08545-a490-c890-654a-260caf35d27d" name="/proc/17685/auxv" pid=17685 comm="kvm-spice" requested_mask="r" denied_mask="r" fsuid=104 ouid=104
  • Video modelをcirrus以外にすると、画面が表示されなくなる(真っ黒のまま)

transfer rate

  • ネットワーク転送量がRDPに比較して極端に多くなるので注意
    • RDP: 100Kbps以下 (idle) ～ 10Mbps (動画再生)
    • SPICE: 100Kbps以下 (idle) ～ 100Mbps以上 (動画再生)、400Mbps近くなることもあった

start/shutdown log for VM

  • Ubuntu 12.04
  • /var/log/libvirt/qemu 以下に、VMごとのログが残される 
    -rw------- 1 root root  3940 Oct  8 00:24 quantal.log
-rw------- 1 root root 12394 Oct 13 23:13 test.log
  • /var/log/libvirt/qemu/xp-vm01.log 
    2012-10-28 12:46:55.328+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name xp-vm01 -uuid XXXXXXXX-YYYY-ZZZZ-WWWW-VVVVVVVVVVVV -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/xp-vm01.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -drive file=/var/lib/libvirt/images/xp-vm01.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive file=/home/mitty/virtio-win-0.1-30.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev tap,fd=16,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a8:d0:1d,bus=pci.0,addr=0x3 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -vga vmware -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
vmsvga_value_read: Bad register 2b
vmsvga_value_read: Bad register 2c
qemu: terminating on signal 15 from pid 1803
2012-10-28 12:51:56.141+0000: shutting down

autoshutdown

  • /var/log/libvirt/shutdownlog.log で確認できる 
    libvirt: libvirt-bin: entering pre-stop at Fri Oct 19 23:22:33 JST 2012
libvirt: libvirt-bin: attempting clean shutdown of xp-vm02 at Fri Oct 19 23:22:33 JST 2012
libvirt: destroying xp-vm02
libvirt: libvirt-bin: exiting pre-stop at Fri Oct 19 23:23:06 JST 2012
libvirt: libvirt-bin: entering pre-stop at Fri Oct 26 23:37:35 JST 2012
libvirt: libvirt-bin: exiting pre-stop at Fri Oct 26 23:37:35 JST 2012
libvirt: libvirt-bin: entering pre-stop at Sun Oct 28 21:45:17 JST 2012
libvirt: libvirt-bin: attempting clean shutdown of xp-vm02 at Sun Oct 28 21:45:17 JST 2012
libvirt: libvirt-bin: attempting clean shutdown of xp-vm01 at Sun Oct 28 21:45:17 JST 2012
libvirt: libvirt-bin: exiting pre-stop at Sun Oct 28 21:45:45 JST 2012
  • /etc/init/libvirt-bin.conf で制御されている模様

passthrough

  • QEMUの引数に追加されるものの例 
    01:00.0 Multimedia controller [0480]: Altera Corporation Device [1172:4c15] (rev 01)

    -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9

Marvell controller

Plextor M6e

  • PX-G128M6e
    • lspci -nnv 
      02:00.0 SATA controller [0106]: Lite-On IT Corp. / Plextor M6e PCI Express SSD [Marvell 88SS9183] [1c28:0122] (rev 14) (prog-if 01 [AHCI 1.0])
        Subsystem: Marvell Technology Group Ltd. Device [1b4b:9183]
  • root@archiso ~ # uname -a 
    Linux archiso 4.0.4-2-ARCH #1 SMP PREEMPT Fri May 22 03:05:23 UTC 2015 x86_64 GNU/Linux
  • root@archiso ~ # cat /proc/cmdline 
    BOOT_IMAGE=boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201506 initrd=boot/intel_ucode.img,boot/x86_64/archiso.img intel_iommu=on
  • root@archiso ~ # dmesg|grep -i -e DMAR -e IOMMU 
    [    0.000000] Command line: BOOT_IMAGE=boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201506 initrd=boot/intel_ucode.img,boot/x86_64/archiso.img intel_iommu=on
[    0.000000] ACPI: DMAR 0x00000000DAC6CED0 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Kernel command line: BOOT_IMAGE=boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201506 initrd=boot/intel_ucode.img,boot/x86_64/archiso.img intel_iommu=on
[    0.000000] Intel-IOMMU: enabled
[    0.086964] dmar: Host address width 39
[    0.086969] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[    0.086987] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.086990] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[    0.087000] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.087003] dmar: RMRR base: 0x000000dbe7b000 end: 0x000000dbe89fff
[    0.087006] dmar: RMRR base: 0x000000dd000000 end: 0x000000df1fffff
[    0.087011] IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    7.370829] DMAR: No ATSR found
[    7.371048] IOMMU: dmar0 using Queued invalidation
[    7.371051] IOMMU: dmar1 using Queued invalidation
[    7.371056] IOMMU: Setting RMRR:
[    7.371083] IOMMU: Setting identity map for device 0000:00:02.0 [0xdd000000 - 0xdf1fffff]
[    7.371527] IOMMU: Setting identity map for device 0000:00:14.0 [0xdbe7b000 - 0xdbe89fff]
[    7.371577] IOMMU: Setting identity map for device 0000:00:1a.0 [0xdbe7b000 - 0xdbe89fff]
[    7.371616] IOMMU: Setting identity map for device 0000:00:1d.0 [0xdbe7b000 - 0xdbe89fff]
[    7.371644] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    7.371661] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    7.673577] [drm] DMAR active, disabling use of stolen memory
[    7.673740] dmar: DRHD: handling fault status reg 2
[    7.673744] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[    7.986473] dmar: DRHD: handling fault status reg 3
[    7.986493] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[    9.337664] dmar: DRHD: handling fault status reg 2
[    9.337683] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 74ff240000
               DMAR:[fault reason 05] PTE Write access is not set
[   12.985456] dmar: DRHD: handling fault status reg 2
[   12.985519] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   13.298243] dmar: DRHD: handling fault status reg 3
[   13.298307] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   18.297124] dmar: DRHD: handling fault status reg 2
[   18.297152] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   18.609924] dmar: DRHD: handling fault status reg 3
[   18.609971] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   23.608862] dmar: DRHD: handling fault status reg 2
[   23.608924] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
  • PX-G128M6eを取り外すと以下のようになる 
    [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=0f566cb2-aec8-4122-a80a-f2e5f48e4426 rw intel_iommu=on
[    0.000000] ACPI: DMAR 0x00000000DAC6CED0 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=0f566cb2-aec8-4122-a80a-f2e5f48e4426 rw intel_iommu=on
[    0.000000] Intel-IOMMU: enabled
[    0.107383] dmar: Host address width 39
[    0.107388] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[    0.107406] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.107409] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[    0.107419] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.107422] dmar: RMRR base: 0x000000dbe7b000 end: 0x000000dbe89fff
[    0.107425] dmar: RMRR base: 0x000000dd000000 end: 0x000000df1fffff
[    0.107430] IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    0.677861] DMAR: No ATSR found
[    0.678093] IOMMU: dmar0 using Queued invalidation
[    0.678096] IOMMU: dmar1 using Queued invalidation
[    0.678100] IOMMU: Setting RMRR:
[    0.678127] IOMMU: Setting identity map for device 0000:00:02.0 [0xdd000000 - 0xdf1fffff]
[    0.678608] IOMMU: Setting identity map for device 0000:00:14.0 [0xdbe7b000 - 0xdbe89fff]
[    0.678655] IOMMU: Setting identity map for device 0000:00:1a.0 [0xdbe7b000 - 0xdbe89fff]
[    0.678697] IOMMU: Setting identity map for device 0000:00:1d.0 [0xdbe7b000 - 0xdbe89fff]
[    0.678725] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.678743] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    4.326005] [drm] DMAR active, disabling use of stolen memory

patch

  • ​LKML: Tim Sander: (PATCH) 3.19-rc7: add quirk for 1c28:0122 (rev 14) SATA controller
  • linux-stable-v4.0.5.patch 
    diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 85f247e..a78b225 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3595,6 +3595,10 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_TTI, 0x0642,
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_JMICRON,
 			 PCI_DEVICE_ID_JMICRON_JMB388_ESD,
 			 quirk_dma_func1_alias);
+/* https://bugzilla.kernel.org/show_bug.cgi?id=42679#c94 */
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_LITE_ON,
+			 PCI_DEVICE_ID_PLEXTOR_M6E,
+			 quirk_dma_func1_alias);
 
 /*
  * Some devices DMA with the wrong devfn, not just the wrong function.
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index e63c02a..1607b20 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2487,6 +2487,9 @@
 
 #define PCI_VENDOR_ID_ASMEDIA		0x1b21
 
+#define PCI_VENDOR_ID_LITE_ON		0x1c28
+#define PCI_DEVICE_ID_PLEXTOR_M6E	0x0122
+
 #define PCI_VENDOR_ID_CIRCUITCO		0x1cc8
 #define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD	0x0001

Clock

QoS

  • ​libvirt: Domain XML format
  • ex) 
        <interface type='network'>
      <mac address='--:--:--:--:--:--'/>
      <source network='network'/>
      <bandwidth>
        <inbound average='1024' peak='1024' burst='1024'/>
        <outbound average='1024' peak='1024' burst='1024'/>
      </bandwidth>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    • virt-managerでは設定できないが、virt-managerでNICの設定を変更しても<bandwidth></bandwidth>の部分は維持されるので、virsh editで設定すればよい
  • Virtual Network設定で指定できるQoS機能は、ざっと試した限りでは反映されない感じだった

OVMF

  • mitty@kvmhost:~ (master)$ pacaur -S ovmf 
    ovmf /usr/
ovmf /usr/share/
ovmf /usr/share/licenses/
ovmf /usr/share/licenses/ovmf/
ovmf /usr/share/licenses/ovmf/License.txt
ovmf /usr/share/ovmf/
ovmf /usr/share/ovmf/ovmf_ia32.bin
ovmf /usr/share/ovmf/ovmf_x64.bin
  • mitty@kvmhost:~/works (master)$ wget ​https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20160316.b1616.g6fe9d9c.noarch.rpm
  • mitty@kvmhost:~/works (master)$ wget ​https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-ia32-0-20160316.b1616.g6fe9d9c.noarch.rpm
  • mitty@kvmhost:~/works (master)$ rpmextract.sh edk2.git-ovmf-x64-0-20160316.b1616.g6fe9d9c.noarch.rpm
  • mitty@kvmhost:~/works (master)$ rpmextract.sh edk2.git-ovmf-ia32-0-20160316.b1616.g6fe9d9c.noarch.rpm
  • mitty@kvmhost:~/works (master)$ sudo cp -R --preserve=timestamp usr/share/* /usr/share/
  • mitty@kvmhost:~ (master)$ sudo vim /etc/libvirt/qemu.conf

    • etc/libvirt/qemu.conf

      a b group="78" 
      515515#   "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", 
      516516#   "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd" 
      517517#] 
       518nvram = [ 
       519   "/usr/share/edk2.git/ovmf-ia32/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-ia32/OVMF_VARS-pure-efi.fd", 
       520   "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" 
       521] 
      518522 
      519523# The backend to use for handling stdout/stderr output from 
      520524# QEMU processes. 
  • EFI自体は起動するが、OSのブートができない (archlinux-2016.03.01-dual.iso, Win10_1511_Japanese_x64.isoでテスト)

Attachments (17)

Download all attachments as: .zip