wiki:TipAndDoc/VM/NestedVMM

Version 22 (modified by mitty, 8 years ago) (diff)

--

/proc/cpuinfo

  • KVM nested environment
  • M/B ASUS H97M-PLUS
  • Core i7-4790 3.60GHz
  • Arch Linux 4.2.5-1-ARCH x86_64
  • /etc/libvirt/qemu/GUEST.xml
      <cpu mode='host-model'>
        <model fallback='allow'/>
      </cpu>
    
  • Host (L0)
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 60
    model name      : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
    stepping        : 3
    microcode       : 0x1c
    cpu MHz         : 3600.421
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 8
    core id         : 0
    cpu cores       : 4
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt
    bugs            :
    bogomips        : 7202.88
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 39 bits physical, 48 bits virtual
    power management:
    
  • L1 Guest
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 60
    model name      : Intel Core Processor (Haswell, no TSX)
    stepping        : 1
    microcode       : 0x1
    cpu MHz         : 3600.016
    cache size      : 4096 KB
    physical id     : 0
    siblings        : 1
    core id         : 0
    cpu cores       : 1
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm arat tpr_shadow vnmi flexpriority ept fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
    bugs            :
    bogomips        : 7202.03
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 40 bits physical, 48 bits virtual
    power management:
    
  • L2 Guest
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 60
    model name      : Intel Core Processor (Haswell, no TSX)
    stepping        : 1
    microcode       : 0x1
    cpu MHz         : 3600.016
    cache size      : 4096 KB
    physical id     : 0
    siblings        : 1
    core id         : 0
    cpu cores       : 1
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 13
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm arat tpr_shadow vnmi flexpriority ept fsgsbase bmi1 avx2 smep bmi2 erms xsaveopt
    bugs            :
    bogomips        : 7202.03
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 40 bits physical, 48 bits virtual
    power management:
    
  • CPU flags
    L0: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand            lahf_lm abm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt
    L1: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush          mmx fxsr sse sse2 ss           syscall nx pdpe1gb rdtscp lm constant_tsc                       rep_good nopl                                  eagerfpu pni pclmulqdq                       vmx             ssse3 fma cx16           pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm     arat                    tpr_shadow vnmi flexpriority ept      fsgsbase            bmi1 avx2 smep bmi2 erms invpcid xsaveopt
    L2: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush          mmx fxsr sse sse2 ss           syscall nx         rdtscp lm constant_tsc                       rep_good nopl                                  eagerfpu pni pclmulqdq                       vmx             ssse3 fma cx16           pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm     arat                    tpr_shadow vnmi flexpriority ept      fsgsbase            bmi1 avx2 smep bmi2 erms         xsaveopt
    

VMware

  • VMware製品には、H/W仮想化支援が無くても動作するものがあるため、適切に設定すればNested VMは簡単に実現可能
    • きちんと設定されていないと次のように表示される
      VMware Server on VMware Workstation
  • VMware Communities: Running Nested VMs
    • VMware製品各種での設定がまとめられている
    • http://communities.vmware.com/docs/DOC-8970/version/30 VMware Workstation 7以前の情報

      Running 32-bit nested VMs with binary translation under ESX(i) 4.x, ESXi 5.0, Workstation 7.x, Player 3.x or Fusion 3.x is the only officially supported configuration. Other configurations may work, but are entirely unsupported.

      Guest Hypervisors Under VMware Hypervisors

      Enabling Intel VT-x for the Outer Guest

      ESX 3.0, Workstation 6.0 and Fusion 1.0
      monitor_control.vt32 = TRUE
      
      Workstation 6.5 (or later), Fusion 2.0 (or later) and ESX 4.0 (or later)
      monitor.virtual_exec = hardware
      

      Note that this is the default for any 64-bit guest OS type on Intel hardware, so you do not have to edit the configuration file to select hardware-assisted virtualization if you have specified a 64-bit guest OS.

      Enabling AMD-V for the Outer Guest

      Family 0FH Rev F and later AMD64 processors provide hardware support for AMD-V, but current VMware products only support AMD-V on Family 10H or later AMD64 processors (Barcelona/Shanghai/Istanbul or Phenom/Phenom II). You can use AMD-V on Family 0FH Rev F and Rev G processors with ESX 3.0 and Workstation 6.0. However, you need a Family 10H or later processor to use AMD-V with any later products.

      ESX 3.0 and Workstation 6.0
      monitor_control.enable_svm = TRUE
      
      ESX 3.5
      monitor.virtual_mmu = hardware
      
      Workstation 6.5 (or later) and ESX(i) 4.0 (or later)
      monitor.virtual_exec = hardware
      

      Networking in Nested Guests

      see vmware:287 (Linux hosts) or vmware:1004099 (ESX hosts). No special configuration is required for Windows hosts.

      VMware Products as Guest Hypervisors

      • ESXi 5.0, you will be able to run 64-bit nested guests
      • VMWare products prior to ESXi 5.0 => you will be restricted to running only 32-bit nested guests using binary translation under any VMware product as the guest hypervisor.
      ESX(i) as a Guest Hypervisor
      • Workstation 7.0/Player 3.0/Fusion 3.0 => select "ESX Server 4"
      • ESX(i) 4.1 => manually edit the VM configuration file and select guestOS="vmkernel"
      • ESXi 5.0 => select "ESX Server 4" or "ESX Server 5"
      Other VMware Products as Guest Hypervisors

      If you plan to run VMware Tools in your inner VM, you will need an additional configuration option to restrict the backdoor interface to the host hypervisor. For guest OS types other than "ESX Server 4" or "ESX Server 5", you should add the following line to the configuration file for the outer guest:

      monitor_control.restrict_backdoor = TRUE
      

      You should not install VMware Tools in the outer guest.

      Virtualizing HV (VT-x or AMD-V)

      With ESXi 5.0 as the host hypervisor, it is possible to virtualize the hardware-assisted virtualization capabilities of the physical processor. This feature is used internally within VMware for testing purposes, but it is not recommended for production systems. This feature is also not recommended for systems without hardware-assisted MMU virtualization (EPT or RVI), because of its abysmal performance without that hardware support.

      This feature is disabled by default. To permit the use of this feature, the ESXi administrator must add the following configuration option to the /etc/vmware/config file on the physical host:

      vhv.allow = TRUE
      

      Once allowed by the ESXi administrator, virtualized HV will be enabled by default for all hardware version 8 VMs with guest OS type "ESX Server 4" and "ESX Server 5." To enable virtualized HV for other guests, add the following lines to the outer guest's configuration file:

      cpuid.1.ecx="----:----:----:----:----:----:--h-:----"
      cpuid.80000001.ecx.amd="----:----:----:----:----:----:----:-h--"
      cpuid.8000000a.eax.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
      cpuid.8000000a.ebx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
      cpuid.8000000a.edx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
      

      With virtual HV enabled in the outer guest, you should be able to run any guest hypervisor that requires hardware-assisted virtualization. In particular, this means that you will be able to run 64-bit nested guests under VMware guest hypervisors.

      KVM as a Guest Hypervisor

      KVM performs relatively poorly in a nested VM using virtualized VT-x. This situation is much improved with Linux kernel versions 3.0 and greater.

      VMware Guest Hypervisors Under Foreign Hypervisors

      If we detect that a VMware product is running under a foreign hypervisor, we will refuse to power on a nested VM. To bypass this constraint, add the following option to your nested VM configuration file:

      vmx.allowNested = TRUE
      

H/W 仮想化支援

  • 実用的な実行速度で動かすためのハードウェア条件
    • Intel VT-x
    • AMD-v with Barcelona or Phenom or later
      • AMD-V is supported on Rev F and later Family 0FH AMD64 processors, but current VMware products only support AMD-V on Family 10H or later AMD64 processors (Barcelona or Phenom). Thus, you can use AMD-V on Family 0FH Rev F and Rev G processors with Workstation 6.0. However, you need a Family 10H processor to use AMD-V with ESX 3.5 and Workstation 6.5 (or later).
      • 手持ちの「AMD Athlon 64 X2 5000+」では「MONITOR MODE: allowed modes : BT」となり、AMD-v使用不能。
  • vmxファイルに以下の2行を追加すると可能になる(非Intel-VT, 非AMD-V環境では未テスト)
    monitor_control.vt32 = "TRUE"
    monitor_control.restrict_backdoor = "TRUE"
    
  • IntelCPU+x64GuestだとVTが必須なので自動的に有効になる
    Software virtualization is incompatible with long mode on this platform. Virtual execution will begin in software mode, but will automatically switch to hardware mode if the guest enters long mode.
    
    • それ以外の環境では、次の1行もあった方が良いかも知れない(入れ子内のVMguestの起動が非常に遅くなり得る)
      monitor.virtual_exec = "hardware"
      

VMware Workstation on Intel VT

  • Ubuntu 10.04 lucid on VMware Workstation 7.1.3 with Windows 7 x64 Host
    • Linux ubuntu32 2.6.32-26-generic #48-Ubuntu SMP Wed Nov 24 09:00:03 UTC 2010 i686 GNU/Linux
    • model name : Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz
  • 仮想マシン設定 > ハードウェア > プロセッサ > 優先モード
    • 自動
      • vmware.log
        Dec 12 17:27:11.858: vmx| MONITOR MODE: allowed modes          : BT32 HV
        Dec 12 17:27:11.858: vmx| MONITOR MODE: user requested modes   : BT32 HV HWMMU
        Dec 12 17:27:11.858: vmx| MONITOR MODE: guestOS preferred modes: HWMMU BT32 HV
        Dec 12 17:27:11.858: vmx| MONITOR MODE: filtered list          : BT32 HV
        Dec 12 17:27:11.858: vmx| HV Settings: virtual exec = 'dynamic'; virtual mmu = 'software'
        
      • /proc/cpuinfo
        flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht nx lm constant_tsc arch_perfmon pebs bts tsc_reliable aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida
        
    • Intel VT-xまたはAMD-V
      • diff of Ubuntu32.vmx
        -monitor.virtual_mmu = "automatic"
        -monitor.virtual_exec = "automatic"
        +monitor.virtual_mmu = "software"
        +monitor.virtual_exec = "hardware"
        
      • vmware.log
        Dec 12 17:53:28.574: vmx| MONITOR MODE: allowed modes          : BT32 HV
        Dec 12 17:53:28.574: vmx| MONITOR MODE: user requested modes   : HV
        Dec 12 17:53:28.574: vmx| MONITOR MODE: guestOS preferred modes: HWMMU BT32 HV
        Dec 12 17:53:28.574: vmx| MONITOR MODE: filtered list          : HV
        Dec 12 17:53:28.574: vmx| HV Settings: virtual exec = 'hardware'; virtual mmu = 'software'
        
      • /proc/cpuinfo
        flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht nx lm constant_tsc arch_perfmon pebs bts tsc_reliable aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida
        
  • 元々が「自動」だからなのか、CPU flagsに変化はなかった。

virtualize Intel VT / AMD-V

  • Intel EPT / AMD RVI 対応のCPUが必須

VMware on KVM

  • M/B ASUS H97M-PLUS
  • Core i7-4790 3.60GHz
  • Arch Linux 4.2.5-1-ARCH x86_64
  • qemu 2.4.1-1
  • seabios 1.8.2-1
  • libvirt 1.2.21-1
  • VMware Workstation 12.0.0 build-2985596
  • vmware-patch 12.0.0-4 (AUR)

Host (L0)

  • /etc/libvirt/qemu/GUEST.xml
      <cpu mode='host-model'>
        <model fallback='allow'/>
      </cpu>
    
    • Host (L0), KVM Guest (L1 Guest)のcpuinfoについは#proccpuinfoと同じ

L1 Guest

  • /etc/modprobe.d/vmware.conf
    blacklist kvm
    blacklist kvm-amd   # For AMD CPUs
    blacklist kvm-intel # For Intel CPUs
    
  • .vmx
    virtualHW.version = "12"
    guestOS = "other-64"
    ide1:0.fileName = "/var/lib/libvirt/images/archlinux-2015.12.01-dual.iso"
    virtualHW.productCompatibility = "hosted"
    vhv.enable = "FALSE"
    disable_acceleration = "FALSE"
    monitor.virtual_mmu = "automatic"
    monitor.virtual_exec = "automatic"
    vmx.allowNested = TRUE
    
    • vmx.allowNested = TRUEがないとこうなる

MONITOR PANIC

  • エラーで起動不能
  • vmware.log
    2015-12-06T00:35:35.029+09:00| vmx| I125: VTHREAD pthread_setschedparam failed: Invalid argument
    2015-12-06T00:35:35.029+09:00| vcpu-0| I125: VTHREAD start thread 4 "vcpu-0" tid 1081
    2015-12-06T00:35:35.029+09:00| vcpu-0| W115: MONITOR PANIC: vcpu-0:VMM fault 13: src=MONITOR rip=0xfffffffffc48c706 regs=0xfffffffffc607e60
    2015-12-06T00:35:35.029+09:00| vcpu-0| I125: Core dump with build build-2985596
    2015-12-06T00:35:35.029+09:00| vcpu-0| W115: Writing monitor corefile "/home/mitty/vmware/arch/vmmcores.gz"
    2015-12-06T00:35:35.029+09:00| vcpu-0| W115: Dumping core for vcpu-0
    2015-12-06T00:35:35.029+09:00| vcpu-0| I125: Beginning monitor coredump
    2015-12-06T00:35:35.436+09:00| vcpu-0| I125: End monitor coredump
    2015-12-06T00:35:35.452+09:00| vcpu-0| W115: 
    2015-12-06T00:35:35.452+09:00| vcpu-0| W115+ The core dump limit is set to ZERO; no core dump should be expected
    2015-12-06T00:35:35.493+09:00| vcpu-0| W115: Core dumped.
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125: Core dump pipes to process /usr/lib/systemd/systemd-coredump %P %u %g %s %t %e, core file unreliable
    2015-12-06T00:35:35.493+09:00| vcpu-0| W115: A core file is available in "/home/mitty/vmware/arch/core.1082"
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125: Msg_Post: Error
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125: [msg.log.error.unrecoverable] VMware Workstation unrecoverable error: (vcpu-0)
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125+ vcpu-0:VMM fault 13: src=MONITOR rip=0xfffffffffc48c706 regs=0xfffffffffc607e60
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125: [msg.panic.haveLog] A log file is available in "/home/mitty/vmware/arch/vmware.log".  
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125: [msg.panic.requestSupport.withoutLog] You can request support.  
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125: [msg.panic.requestSupport.vmSupport.windowsOrLinux] 
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125+ To collect data to submit to VMware support, choose "Collect Support Data" from the Help menu.
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125+ You can also run the "vm-support" script in the Workstation folder directly.
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125: [msg.panic.response] We will respond on the basis of your support entitlement.
    2015-12-06T00:35:35.493+09:00| vcpu-0| I125: ----------------------------------------
    2015-12-06T00:36:01.708+09:00| vcpu-0| W115: Writing monitor corefile "/home/mitty/vmware/arch/vmmcores.gz"
    2015-12-06T00:36:01.708+09:00| vcpu-0| W115: Dumping core for vcpu-0
    2015-12-06T00:36:01.709+09:00| vcpu-0| I125: Beginning monitor coredump
    2015-12-06T00:36:02.145+09:00| vcpu-0| I125: End monitor coredump
    

KVM

  • KVMはH/W仮想化支援が必須

KVM on KVM

  • Intel EPT / AMD RVI 対応のCPUが必須

Debian 6.0 with kernel backports

  • 3.x系(3.1以降?)のkernelで無いと、Intel CPUには対応出来ないらしい
  • virt-managerが古いせいか、ゲストの設定ファイル(xml)を直接編集した後、ゲストが起動しなくなった
  • mitty@walken:~$ lsb_release -a
    No LSB modules are available.
    Distributor ID: Debian
    Description:    Debian GNU/Linux 6.0.4 (squeeze)
    Release:        6.0.4
    Codename:       squeeze
    
  • mitty@walken:~$ uname -a
    Linux walken 3.2.0-0.bpo.2-amd64 #1 SMP Sun Mar 25 10:33:35 UTC 2012 x86_64 GNU/Linux
    
  • mitty@walken:~$ vmware -v
    VMware Workstation 8.0.2 build-591240
    
    • VMwareとのゲスト同時稼働も問題ない模様
  • mitty@walken:~$ kvm --version
    QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c) 2003-2008 Fabrice Bellard
    
  • mitty@walken:~$ virt-manager --version
    0.8.4
    
    • CPU設定に細かい設定項目がない
    • see also virt-manager
  • mitty@walken:~$ cat /proc/cpuinfo
    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 30
    model name      : Intel(R) Core(TM) i5 CPU         750  @ 2.67GHz
    stepping        : 5
    microcode       : 0x3
    cpu MHz         : 1197.000
    cache size      : 8192 KB
    physical id     : 0
    siblings        : 4
    core id         : 0
    cpu cores       : 4
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 11
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid
    bogomips        : 5319.48
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 36 bits physical, 48 bits virtual
    power management:
    
    (snip)
    

nested=1

  • mitty@walken:~$ modinfo kvm_intel |grep nested
    parm:           nested:bool
    
  • mitty@walken:~$ cat /sys/module/kvm_intel/parameters/nested
    N
    
  • mitty@walken:~$ sudo vim /etc/modprobe.d/kvm-nested.conf
    options kvm_intel nested=1
    
  • mitty@walken:~$ sudo modprobe -r kvm_intel
  • mitty@walken:~$ sudo modprobe kvm_intel
  • mitty@walken:~$ cat /sys/module/kvm_intel/parameters/nested
    Y
    
  • mitty@walken:~$ sudo vim /etc/libvirt/qemu/test.xml
    • test.xml

      old new  
      1414    <apic/> 
      1515    <pae/> 
      1616  </features> 
       17  <cpu match='exact'> 
       18    <model>Nehalem</model> 
       19    <vendor>Intel</vendor> 
       20    <feature policy='require' name='rdtscp'/> 
       21    <feature policy='require' name='ss'/> 
       22    <feature policy='require' name='hypervisor'/> 
       23    <feature policy='require' name='vme'/> 
       24    <feature policy='require' name='ds'/> 
       25    <feature policy='require' name='vmx'/> 
       26    <feature policy='require' name='acpi'/> 
       27  </cpu> 
      1728  <clock offset='utc'/> 
      1829  <on_poweroff>destroy</on_poweroff> 
      1930  <on_reboot>restart</on_reboot> 
    • Fedora16のLiveCDでvirt-managerを起動し、下記サイトを参考にGUI上から設定を行った場合に、xmlに書き込まれる差分をそのまま適用してみた
    • うまく起動しない

Attachments (23)

Download all attachments as: .zip