wiki:TipAndDoc/VM/KVM

Version 36 (modified by mitty, 9 years ago) (diff)

--

  1. KVM
    1. Client
  2. Windows Guest
    1. drivers
      1. block device
  3. virbr0
    1. iptables
      1. port forwarding
  4. Live Migration
  5. kvm-spice
  6. start/shutdown log for VM
    1. autoshutdown
  7. passthrough
    1. Marvell controller
      1. PX-G128M6e

KVM

  • KVM自体にはIntel-VT or AMD-V が必須 
    egrep '(vmx|svm)' --color=always /proc/cpuinfo
  • QEMUはVT/AMD-vが無くても動く(ただし非常に遅い)

Client

Windows Guest

drivers

  • デバイスマネージャ

block device

  • virtio-win-1.1.16.vfdを用いてVirtIO diskでセットアップ後、virtio-win-0.1-30.isoによってドライバを更新するとBSODを出して起動しなくなる。
    • OSセットアップ直後
    • ドライバ更新
    • BSOD

virbr0

  • libvirtdによって自動で作られる「default」のNAT
    • ps aux | grep dns 
      nobody    2684  0.0  0.0  21264   792 ?        S    Feb07   0:01 dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file=  --listen-address 192.168.122.1 --except-interface lo --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-lease-max=253
    • lease file => /var/lib/misc/dnsmasq.leases

iptables

  • sudo iptables-save 
    # Generated by iptables-save v1.4.8 on Thu Feb 16 10:31:29 2012
*nat
:PREROUTING ACCEPT [743892:129452770]
:POSTROUTING ACCEPT [1179:112796]
:OUTPUT ACCEPT [1215:115607]
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE 
COMMIT
# Completed on Thu Feb 16 10:31:29 2012
# Generated by iptables-save v1.4.8 on Thu Feb 16 10:31:29 2012
*filter
:INPUT ACCEPT [4957020:3457557240]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3323525:4327643495]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT 
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 
COMMIT
# Completed on Thu Feb 16 10:31:29 2012

port forwarding

Live Migration

kvm-spice

  • Ubuntu 12.04でqemu-kvm-spiceをインストールすると、デフォルトのemulatorが/usr/bin/kvm-spiceになって以下の様な問題が起きる
  • kern.logにAppArmorからDENIEDとエラーが出る 
    Oct 13 22:45:08 HOSTNAME kernel: [507466.445098] type=1400 audit(1350135908.372:69): apparmor="DENIED" operation="open" parent=1 profile="libvirt-52d08545-a490-c890-654a-260caf35d27d" name="/proc/17685/auxv" pid=17685 comm="kvm-spice" requested_mask="r" denied_mask="r" fsuid=104 ouid=104
  • Video modelをcirrus以外にすると、画面が表示されなくなる(真っ黒のまま)

start/shutdown log for VM

  • Ubuntu 12.04
  • /var/log/libvirt/qemu 以下に、VMごとのログが残される 
    -rw------- 1 root root  3940 Oct  8 00:24 quantal.log
-rw------- 1 root root 12394 Oct 13 23:13 test.log
  • /var/log/libvirt/qemu/xp-vm01.log 
    2012-10-28 12:46:55.328+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name xp-vm01 -uuid XXXXXXXX-YYYY-ZZZZ-WWWW-VVVVVVVVVVVV -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/xp-vm01.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -drive file=/var/lib/libvirt/images/xp-vm01.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive file=/home/mitty/virtio-win-0.1-30.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev tap,fd=16,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a8:d0:1d,bus=pci.0,addr=0x3 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -vga vmware -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
vmsvga_value_read: Bad register 2b
vmsvga_value_read: Bad register 2c
qemu: terminating on signal 15 from pid 1803
2012-10-28 12:51:56.141+0000: shutting down

autoshutdown

  • /var/log/libvirt/shutdownlog.log で確認できる 
    libvirt: libvirt-bin: entering pre-stop at Fri Oct 19 23:22:33 JST 2012
libvirt: libvirt-bin: attempting clean shutdown of xp-vm02 at Fri Oct 19 23:22:33 JST 2012
libvirt: destroying xp-vm02
libvirt: libvirt-bin: exiting pre-stop at Fri Oct 19 23:23:06 JST 2012
libvirt: libvirt-bin: entering pre-stop at Fri Oct 26 23:37:35 JST 2012
libvirt: libvirt-bin: exiting pre-stop at Fri Oct 26 23:37:35 JST 2012
libvirt: libvirt-bin: entering pre-stop at Sun Oct 28 21:45:17 JST 2012
libvirt: libvirt-bin: attempting clean shutdown of xp-vm02 at Sun Oct 28 21:45:17 JST 2012
libvirt: libvirt-bin: attempting clean shutdown of xp-vm01 at Sun Oct 28 21:45:17 JST 2012
libvirt: libvirt-bin: exiting pre-stop at Sun Oct 28 21:45:45 JST 2012
  • /etc/init/libvirt-bin.conf で制御されている模様

passthrough

  • QEMUの引数に追加されるものの例 
    01:00.0 Multimedia controller [0480]: Altera Corporation Device [1172:4c15] (rev 01)

    -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9

Marvell controller

PX-G128M6e

  • lspci -nnv 
    02:00.0 SATA controller [0106]: Lite-On IT Corp. / Plextor M6e PCI Express SSD [Marvell 88SS9183] [1c28:0122] (rev 14) (prog-if 01 [AHCI 1.0])
        Subsystem: Marvell Technology Group Ltd. Device [1b4b:9183]
  • root@archiso ~ # uname -a 
    Linux archiso 4.0.4-2-ARCH #1 SMP PREEMPT Fri May 22 03:05:23 UTC 2015 x86_64 GNU/Linux
  • root@archiso ~ # cat /proc/cmdline 
    BOOT_IMAGE=boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201506 initrd=boot/intel_ucode.img,boot/x86_64/archiso.img intel_iommu=on
  • root@archiso ~ # dmesg|grep -i -e DMAR -e IOMMU 
    [    0.000000] Command line: BOOT_IMAGE=boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201506 initrd=boot/intel_ucode.img,boot/x86_64/archiso.img intel_iommu=on
[    0.000000] ACPI: DMAR 0x00000000DAC6CED0 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Kernel command line: BOOT_IMAGE=boot/x86_64/vmlinuz archisobasedir=arch archisolabel=ARCH_201506 initrd=boot/intel_ucode.img,boot/x86_64/archiso.img intel_iommu=on
[    0.000000] Intel-IOMMU: enabled
[    0.086964] dmar: Host address width 39
[    0.086969] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[    0.086987] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.086990] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[    0.087000] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.087003] dmar: RMRR base: 0x000000dbe7b000 end: 0x000000dbe89fff
[    0.087006] dmar: RMRR base: 0x000000dd000000 end: 0x000000df1fffff
[    0.087011] IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    7.370829] DMAR: No ATSR found
[    7.371048] IOMMU: dmar0 using Queued invalidation
[    7.371051] IOMMU: dmar1 using Queued invalidation
[    7.371056] IOMMU: Setting RMRR:
[    7.371083] IOMMU: Setting identity map for device 0000:00:02.0 [0xdd000000 - 0xdf1fffff]
[    7.371527] IOMMU: Setting identity map for device 0000:00:14.0 [0xdbe7b000 - 0xdbe89fff]
[    7.371577] IOMMU: Setting identity map for device 0000:00:1a.0 [0xdbe7b000 - 0xdbe89fff]
[    7.371616] IOMMU: Setting identity map for device 0000:00:1d.0 [0xdbe7b000 - 0xdbe89fff]
[    7.371644] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    7.371661] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    7.673577] [drm] DMAR active, disabling use of stolen memory
[    7.673740] dmar: DRHD: handling fault status reg 2
[    7.673744] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[    7.986473] dmar: DRHD: handling fault status reg 3
[    7.986493] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[    9.337664] dmar: DRHD: handling fault status reg 2
[    9.337683] dmar: DMAR:[DMA Write] Request device [00:02.0] fault addr 74ff240000
               DMAR:[fault reason 05] PTE Write access is not set
[   12.985456] dmar: DRHD: handling fault status reg 2
[   12.985519] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   13.298243] dmar: DRHD: handling fault status reg 3
[   13.298307] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   18.297124] dmar: DRHD: handling fault status reg 2
[   18.297152] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   18.609924] dmar: DRHD: handling fault status reg 3
[   18.609971] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
[   23.608862] dmar: DRHD: handling fault status reg 2
[   23.608924] dmar: DMAR:[DMA Write] Request device [02:00.1] fault addr fffe0000
               DMAR:[fault reason 02] Present bit in context entry is clear
  • PX-G128M6eを取り外すと以下のようになる 
    [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=0f566cb2-aec8-4122-a80a-f2e5f48e4426 rw intel_iommu=on
[    0.000000] ACPI: DMAR 0x00000000DAC6CED0 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=0f566cb2-aec8-4122-a80a-f2e5f48e4426 rw intel_iommu=on
[    0.000000] Intel-IOMMU: enabled
[    0.107383] dmar: Host address width 39
[    0.107388] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[    0.107406] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.107409] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[    0.107419] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.107422] dmar: RMRR base: 0x000000dbe7b000 end: 0x000000dbe89fff
[    0.107425] dmar: RMRR base: 0x000000dd000000 end: 0x000000df1fffff
[    0.107430] IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    0.677861] DMAR: No ATSR found
[    0.678093] IOMMU: dmar0 using Queued invalidation
[    0.678096] IOMMU: dmar1 using Queued invalidation
[    0.678100] IOMMU: Setting RMRR:
[    0.678127] IOMMU: Setting identity map for device 0000:00:02.0 [0xdd000000 - 0xdf1fffff]
[    0.678608] IOMMU: Setting identity map for device 0000:00:14.0 [0xdbe7b000 - 0xdbe89fff]
[    0.678655] IOMMU: Setting identity map for device 0000:00:1a.0 [0xdbe7b000 - 0xdbe89fff]
[    0.678697] IOMMU: Setting identity map for device 0000:00:1d.0 [0xdbe7b000 - 0xdbe89fff]
[    0.678725] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.678743] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    4.326005] [drm] DMAR active, disabling use of stolen memory

Attachments (17)

Download all attachments as: .zip