wiki:TipAndDoc/storage/RAID/rescue/RAID1

RAID1

  • mdadm による Software RAID1 の復旧テスト

two of two devices are alive and clean

  • see wiki:TipAndDoc/HA/NFS#mdadm for how to make device
  • 以下は済んでいるが、Logical volumeは作っていない
    1. mitty@ubuntu-haa:~$ sudo pvcreate /dev/md0
    2. mitty@ubuntu-haa:~$ sudo vgcreate -s 32 vgnfs /dev/md0
  • mitty@ubuntu-haa:~$ sudo mdadm -D /dev/md0
    /dev/md0:
            Version : 00.90
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
        Persistence : Superblock is persistent
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
    
               UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
             Events : 0.34
    
        Number   Major   Minor   RaidDevice State
           0       8       17        0      active sync   /dev/sdb1
           1       8       33        1      active sync   /dev/sdc1
    
  • mitty@ubuntu-haa:~$ sudo mdadm --stop /dev/md0
    mdadm: stopped /dev/md0
    
  • mitty@ubuntu-haa:~$ sudo mdadm -E /dev/sd[bc]1
    /dev/sdb1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e3889399 - correct
             Events : 34
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8       17        0      active sync   /dev/sdb1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       8       33        1      active sync   /dev/sdc1
    /dev/sdc1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e38893ab - correct
             Events : 34
    
    
          Number   Major   Minor   RaidDevice State
    this     1       8       33        1      active sync   /dev/sdc1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       8       33        1      active sync   /dev/sdc1
    

rescue with Knoppix

  • root@Knoppix:~# cat /etc/knoppix-version
    5.3.1 2008-03-26
    
  • root@Knoppix:~# uname -a
    Linux Knoppix 2.6.24.4 #2 SMP PREEMPT Tue Mar 25 21:35:28 CET 2008 i686 GNU/Linux
    

devices

  • root@Knoppix:~# ls -l /dev/sd?
    brw-rw---- 1 root disk 8,   0 Mar 25  2011 /dev/sda
    brw-rw---- 1 root disk 8,  16 Mar 25  2011 /dev/sdb
    

modprobe

  • root@Knoppix:~# cat /proc/mdstat
    cat: /proc/mdstat: No such file or directory
    
  • root@Knoppix:~# modprobe md_mod
  • root@Knoppix:~# cat /proc/mdstat
    Personalities :
    unused devices: <none>
    

scan devices

  • root@Knoppix:~# mdadm -E /dev/sd* 2>/dev/null
    /dev/sda1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e3889399 - correct
             Events : 0.34
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8       17        0      active sync   /dev/sdb1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       8       33        1      active sync   /dev/sdc1
    /dev/sdb1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e38893ab - correct
             Events : 0.34
    
    
          Number   Major   Minor   RaidDevice State
    this     1       8       33        1      active sync   /dev/sdc1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       8       33        1      active sync   /dev/sdc1
    
  • root@Knoppix:~# mdadm -Es
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=32578407:6f33f50b:893cf340:745f5dce
    

method 1

  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce
    mdadm: /dev/md0 has been started with 2 drives.
    
    • 以下のようなエラーが出た場合は、modprobe md_modし忘れている場合がある
      mdadm: error opening /dev/md0: No such device or address
      

method 2

  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sd[ab]1
    mdadm: /dev/md0 has been started with 2 drives.
    

result

  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    md0 : active raid1 sda1[0] sdb1[1]
          2047936 blocks [2/2] [UU]
    
    unused devices: <none>
    
  • root@Knoppix:~# mdadm -Ds
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=32578407:6f33f50b:893cf340:745f5dce
    
  • root@Knoppix:~# mdadm -D /dev/md0
    /dev/md0:
            Version : 00.90.03
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
        Persistence : Superblock is persistent
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
    
               UUID : 32578407:6f33f50b:893cf340:745f5dce
             Events : 0.34
    
        Number   Major   Minor   RaidDevice State
           0       8        1        0      active sync   /dev/sda1
           1       8       17        1      active sync   /dev/sdb1
    
  • root@Knoppix:~# pvdisplay -C
      PV         VG    Fmt  Attr PSize PFree
      /dev/md0   vgnfs lvm2 a-   1.94G 1.94G
    
  • root@Knoppix:~# vgdisplay -C
      VG    #PV #LV #SN Attr   VSize VFree
      vgnfs   1   0   0 wz--n- 1.94G 1.94G
    

degraded RAID1

devices

  • root@Knoppix:~# ls -l /dev/sd?
    brw-rw---- 1 root disk 8,   0 Mar 25  2011 /dev/sda
    

modprobe

  • root@Knoppix:~# cat /proc/mdstat
    cat: /proc/mdstat: No such file or directory
    
  • root@Knoppix:~# modprobe md_mod
  • root@Knoppix:~# cat /proc/mdstat
    Personalities :
    unused devices: <none>
    
  • root@Knoppix:~# mdadm -E /dev/sda1
    /dev/sda1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e3889399 - correct
             Events : 0.34
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8       17        0      active sync   /dev/sdb1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       8       33        1      active sync   /dev/sdc1
    
  • root@Knoppix:~# mdadm -Es
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=32578407:6f33f50b:893cf340:745f5dce
    

method 1

  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
    

method 2

  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sda1
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
    

result

  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    md0 : active raid1 sda1[0]
          2047936 blocks [2/1] [U_]
    
    unused devices: <none>
    
  • root@Knoppix:~# mdadm -Ds
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=32578407:6f33f50b:893cf340:745f5dce
    
  • root@Knoppix:~# mdadm -D /dev/md0
    /dev/md0:
            Version : 00.90.03
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 1
    Preferred Minor : 0
        Persistence : Superblock is persistent
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean, degraded
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 0
      Spare Devices : 0
    
               UUID : 32578407:6f33f50b:893cf340:745f5dce
             Events : 0.34
    
        Number   Major   Minor   RaidDevice State
           0       8        1        0      active sync   /dev/sda1
           1       0        0        1      removed
    

two of two devices alive but not same

  • degraded RAID1に対して、(片方のHDDだけattachされている状態で)何らかの操作を行い、後から古いHDDをre-addした際の挙動を調べる

make Logical volume on degraded RAID1

status

  • mitty@ubuntu-haa:~$ ls -l /dev/sd*
    brw-rw---- 1 root disk 8, 16 2011-03-25 16:16 /dev/sdb
    brw-rw---- 1 root disk 8, 17 2011-03-25 16:16 /dev/sdb1
    
  • mitty@ubuntu-haa:~$ cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : inactive sdb1[0](S)
          2047936 blocks
    
    unused devices: <none>
    
  • mitty@ubuntu-haa:~$ sudo mdadm -E /dev/sdb1
    /dev/sdb1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e3889399 - correct
             Events : 34
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8       17        0      active sync   /dev/sdb1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       8       33        1      active sync
    
  • mitty@ubuntu-haa:~$ sudo mdadm -R /dev/md0
    mdadm: started /dev/md0
    
  • mitty@ubuntu-haa:~$ cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : active raid1 sdb1[0]
          2047936 blocks [2/1] [U_]
    
    unused devices: <none>
    

make /dev/vgnfs/drbd

  • mitty@ubuntu-haa:~$ sudo lvcreate --size 1.5G -n drbd vgnfs
      Logical volume "drbd" created
    
  • mitty@ubuntu-haa:~$ sudo lvdisplay
      --- Logical volume ---
      LV Name                /dev/vgnfs/drbd
      VG Name                vgnfs
      LV UUID                dNxdNj-hZCZ-GMrC-woMk-0hA2-f3oR-sBsrHI
      LV Write Access        read/write
      LV Status              available
      # open                 0
      LV Size                1.50 GiB
      Current LE             48
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
      - currently set to     256
      Block device           251:0
    
    

mdadm Events increased

  • mitty@ubuntu-haa:~$ sudo mdadm -E /dev/sdb1
    /dev/sdb1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 1
    Preferred Minor : 0
    
        Update Time : Fri Mar 25 15:28:34 2011
              State : clean
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 1
      Spare Devices : 0
           Checksum : e389914b - correct
             Events : 36
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8       17        0      active sync   /dev/sdb1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       0        0        1      faulty removed
    

rescue with two devices on Knoppix

devices

  • root@Knoppix:~# mdadm -E /dev/sd[ab]1
    /dev/sda1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 1
    Preferred Minor : 0
    
        Update Time : Fri Mar 25 15:28:34 2011
              State : clean
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 1
      Spare Devices : 0
           Checksum : e389914b - correct
             Events : 0.36
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8       17        0      active sync   /dev/sdb1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       0        0        1      faulty removed
    /dev/sdb1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Thu Mar 24 21:25:32 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e38893ab - correct
             Events : 0.34
    
    
          Number   Major   Minor   RaidDevice State
    this     1       8       33        1      active sync   /dev/sdc1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       8       33        1      active sync   /dev/sdc1
    

assemble md0 and md1

  • 上記の # mdadm -E /dev/sd[ab]1 の結果から、RAID1デバイスのうち /dev/sda1 が最新であると推定されるが、その確認として /dev/sd[ab]1 をそれぞれRAID1としてAssembleする

assemble md0

  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
    
  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    md0 : active raid1 sda1[0]
          2047936 blocks [2/1] [U_]
    
    unused devices: <none>
    
    • 自動的に最新のデバイスが選択されるのか、/dev/sda1から/dev/md0がassembleされる
    • どちらが最新であるかどのように判断しているかが不明 (Events? Update Time?)

assemble md1

  • root@Knoppix:~# mdadm -A /dev/md1 /dev/sdb1
    mdadm: /dev/md1 assembled from 1 drive - need all 2 to start it (use --run to insist).
    
  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    md1 : inactive sdb1[1](S)
          2047936 blocks
    
    md0 : active raid1 sda1[0]
          2047936 blocks [2/1] [U_]
    
    unused devices: <none>
    
    • この時点では md1 は動いていない

scanning LVM

  • /dev/sda1 -> /dev/md0 のみが有効なので、/dev/vgnfs/drbdが正常に表示される
  • root@Knoppix:~# pvscan
      PV /dev/md0   VG vgnfs   lvm2 [1.94 GB / 448.00 MB free]
      Total: 1 [1.94 GB] / in use: 1 [1.94 GB] / in no VG: 0 [0   ]
    
  • root@Knoppix:~# vgscan
      Reading all physical volumes.  This may take a while...
      Found volume group "vgnfs" using metadata type lvm2
    
  • root@Knoppix:~# lvscan
      inactive          '/dev/vgnfs/drbd' [1.50 GB] inherit
    
  • root@Knoppix:~# lvdisplay
      --- Logical volume ---
      LV Name                /dev/vgnfs/drbd
      VG Name                vgnfs
      LV UUID                dNxdNj-hZCZ-GMrC-woMk-0hA2-f3oR-sBsrHI
      LV Write Access        read/write
      LV Status              NOT available
      LV Size                1.50 GB
      Current LE             48
      Segments               1
      Allocation             inherit
      Read ahead sectors     0
    
    

start md1

  • root@Knoppix:~# mdadm -R /dev/md1
    mdadm: started /dev/md1
    
  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    md1 : active raid1 sdb1[1]
          2047936 blocks [2/1] [_U]
    
    md0 : active raid1 sda1[0]
          2047936 blocks [2/1] [U_]
    
    unused devices: <none>
    

pvchange duplicate PV

  • root@Knoppix:~# pvscan
      Found duplicate PV Z2JXRPfa5gSYS5xzMsLq8C1JbhQPKihr: using /dev/md1 not /dev/md0
      PV /dev/md1   VG vgnfs   lvm2 [1.94 GB / 1.94 GB free]
      Total: 1 [1.94 GB] / in use: 1 [1.94 GB] / in no VG: 0 [0   ]
    
  • root@Knoppix:~# pvchange -u /dev/md1
      Found duplicate PV Z2JXRPfa5gSYS5xzMsLq8C1JbhQPKihr: using /dev/md0 not /dev/md1
      Found duplicate PV Z2JXRPfa5gSYS5xzMsLq8C1JbhQPKihr: using /dev/md1 not /dev/md0
      Physical volume "/dev/md1" changed
      1 physical volume changed / 0 physical volumes not changed
    
  • root@Knoppix:~# mdadm -E /dev/sd[ab]1
    /dev/sda1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 1
    Preferred Minor : 0
    
        Update Time : Fri Mar 25 15:28:34 2011
              State : clean
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 1
      Spare Devices : 0
           Checksum : e389914b - correct
             Events : 0.36
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8       17        0      active sync   /dev/sdb1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       0        0        1      faulty removed
    /dev/sdb1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 1
    Preferred Minor : 1
    
        Update Time : Fri Mar 25 16:58:00 2011
              State : clean
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e389a642 - correct
             Events : 0.36
    
    
          Number   Major   Minor   RaidDevice State
    this     1       8       17        1      active sync   /dev/sdb1
    
       0     0       0        0        0      removed
       1     1       8       17        1      active sync   /dev/sdb1
    
    • /dev/sdb1 -> /dev/md1 の Events が 0.34 -> 0.36 と増加

new Physical volume

  • UUIDを変更した場合、双方のPVが表示されるようになるが、VGは片方のみ表示される
  • root@Knoppix:~# pvdisplay /dev/md[01]
      --- Physical volume ---
      PV Name               /dev/md0
      VG Name               vgnfs
      PV Size               1.94 GB / not usable 0
      Allocatable           yes
      PE Size (KByte)       32768
      Total PE              62
      Free PE               14
      Allocated PE          48
      PV UUID               Z2JXRP-fa5g-SYS5-xzMs-Lq8C-1Jbh-QPKihr
    
      --- NEW Physical volume ---
      PV Name               /dev/md1
      VG Name
      PV Size               1.94 GB
      Allocatable           NO
      PE Size (KByte)       0
      Total PE              0
      Free PE               0
      Allocated PE          0
      PV UUID               A9rTHY-2IOl-mits-PiCZ-WZ5I-gQah-5265YU
    
    
  • root@Knoppix:~# vgdisplay
      --- Volume group ---
      VG Name               vgnfs
      System ID
      Format                lvm2
      Metadata Areas        2
      Metadata Sequence No  2
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                1
      Open LV               0
      Max PV                0
      Cur PV                1
      Act PV                1
      VG Size               1.94 GB
      PE Size               32.00 MB
      Total PE              62
      Alloc PE / Size       48 / 1.50 GB
      Free  PE / Size       14 / 448.00 MB
      VG UUID               I6vVoh-6gCJ-9uvA-v2MV-Fyva-7J8v-Cvftfi
    
    

stop md0

  • /dev/md0 を止めることで、 /dev/md1 上のVGを表示出来る
  • root@Knoppix:~# mdadm --stop /dev/md0
    mdadm: stopped /dev/md0
    
  • root@Knoppix:~# pvdisplay /dev/md[01]
      No physical volume label read from /dev/md0
      Failed to read physical volume "/dev/md0"
      --- Physical volume ---
      PV Name               /dev/md1
      VG Name               vgnfs
      PV Size               1.94 GB / not usable 0
      Allocatable           yes
      PE Size (KByte)       32768
      Total PE              62
      Free PE               62
      Allocated PE          0
      PV UUID               A9rTHY-2IOl-mits-PiCZ-WZ5I-gQah-5265YU
    
    
  • root@Knoppix:~# vgdisplay
      --- Volume group ---
      VG Name               vgnfs
      System ID
      Format                lvm2
      Metadata Areas        1
      Metadata Sequence No  2
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                0
      Open LV               0
      Max PV                0
      Cur PV                1
      Act PV                1
      VG Size               1.94 GB
      PE Size               32.00 MB
      Total PE              62
      Alloc PE / Size       0 / 0
      Free  PE / Size       62 / 1.94 GB
      VG UUID               I6vVoh-6gCJ-9uvA-v2MV-Fyva-7J8v-Cvftfi
    
    
    • /dev/md1 はLVが作られていないので、PEが全てFreeのまま

re-assemble /dev/md0 with two device

  • #assemblemd0andmd1 の過程で、PVのUUID等が不一致となったデバイスからRAID1を再構築する

stop md0 and md1

  • root@Knoppix:~# mdadm --stop /dev/md0
    mdadm: stopped /dev/md0
    
  • root@Knoppix:~# mdadm --stop /dev/md1
    mdadm: stopped /dev/md1
    
  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    unused devices: <none>
    

mdadm status

  • root@Knoppix:~# mdadm -E /dev/sd[ab]1
    /dev/sda1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 1
    Preferred Minor : 0
    
        Update Time : Fri Mar 25 15:28:34 2011
              State : clean
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 1
      Spare Devices : 0
           Checksum : e389914b - correct
             Events : 0.36
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8       17        0      active sync   /dev/sdb1
    
       0     0       8       17        0      active sync   /dev/sdb1
       1     1       0        0        1      faulty removed
    /dev/sdb1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 1
    Preferred Minor : 1
    
        Update Time : Fri Mar 25 16:58:00 2011
              State : clean
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e389a642 - correct
             Events : 0.36
    
    
          Number   Major   Minor   RaidDevice State
    this     1       8       17        1      active sync   /dev/sdb1
    
       0     0       0        0        0      removed
       1     1       8       17        1      active sync   /dev/sdb1
    

assemble /dev/md0

  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sd[ab]1
    mdadm: device 1 in /dev/md0 has wrong state in superblock, but /dev/sdb1 seems ok
    mdadm: failed to add /dev/sda1 to /dev/md0: Invalid argument
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
    
  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce
    mdadm: device 1 in /dev/md0 has wrong state in superblock, but /dev/sdb1 seems ok
    mdadm: failed to add /dev/sda1 to /dev/md0: Invalid argument
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
    
  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    md0 : active raid1 sdb1[1]
          2047936 blocks [2/1] [_U]
    
    unused devices: <none>
    
  • Update Time がより新しい /dev/sdb1 から /dev/md0 が構成されてしまう
  • root@Knoppix:~# lvdisplay -C
    • no output

assemble /dev/md0 with /dev/sda1

  • 明示的に /dev/sda1 から /dev/md0 を再構成する
  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sda1
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
    
  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    md0 : active raid1 sda1[0]
          2047936 blocks [2/1] [U_]
    
    unused devices: <none>
    
  • root@Knoppix:~# lvdisplay -C
      LV   VG    Attr   LSize Origin Snap%  Move Log Copy%
      drbd vgnfs -wi--- 1.50G
    
    • LVの drbd が復活する

re-add /dev/sdb1 to /dev/md0

  • root@Knoppix:~# mdadm /dev/md0 -a /dev/sdb1
    mdadm: re-added /dev/sdb1
    
  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    md0 : active raid1 sdb1[1] sda1[0]
          2047936 blocks [2/2] [UU]
    
    unused devices: <none>
    
  • root@Knoppix:~# dmesg | tail -n 20
    RAID1 conf printout:
     --- wd:1 rd:2
     disk 0, wo:0, o:1, dev:sda1
     disk 1, wo:1, o:1, dev:sdb1
    md: recovery of RAID array md0
    md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
    md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
    md: using 128k window, over a total of 2047936 blocks.
    md: md0: recovery done.
    RAID1 conf printout:
     --- wd:2 rd:2
     disk 0, wo:0, o:1, dev:sda1
     disk 1, wo:0, o:1, dev:sdb1
    
  • root@Knoppix:~# mdadm -E /dev/sd[ab]1
    /dev/sda1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Fri Mar 25 17:13:19 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e389a9dc - correct
             Events : 0.42
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8        1        0      active sync   /dev/sda1
    
       0     0       8        1        0      active sync   /dev/sda1
       1     1       8       17        1      active sync   /dev/sdb1
    /dev/sdb1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Fri Mar 25 17:13:19 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e389a9ee - correct
             Events : 0.42
    
    
          Number   Major   Minor   RaidDevice State
    this     1       8       17        1      active sync   /dev/sdb1
    
       0     0       8        1        0      active sync   /dev/sda1
       1     1       8       17        1      active sync   /dev/sdb1
    
    • Events がさらに増加して 0.42 となる

something wrong on PV on RAID1

  • /dev/sda1 と /dev/sdb1 が同期されたように見えて、実際にはされていない?
  • root@Knoppix:~# mdadm --stop /dev/md0
    mdadm: stopped /dev/md0
    
  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce
    mdadm: /dev/md0 has been started with 2 drives.
    
  • root@Knoppix:~# pvdisplay
      get_pv_from_vg_by_id: vg_read failed to read VG
      --- NEW Physical volume ---
      PV Name               /dev/md0
      VG Name
      PV Size               1.95 GB
      Allocatable           NO
      PE Size (KByte)       0
      Total PE              0
      Free PE               0
      Allocated PE          0
      PV UUID               Z2JXRP-fa5g-SYS5-xzMs-Lq8C-1Jbh-QPKihr
    
    

correct PV with degraded RAID1

  • root@Knoppix:~# mdadm --stop /dev/md0
    mdadm: stopped /dev/md0
    
  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sda1 --run
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
    
  • root@Knoppix:~# pvdisplay /dev/md0
      --- Physical volume ---
      PV Name               /dev/md0
      VG Name               vgnfs
      PV Size               1.94 GB / not usable 0
      Allocatable           yes
      PE Size (KByte)       32768
      Total PE              62
      Free PE               14
      Allocated PE          48
      PV UUID               Z2JXRP-fa5g-SYS5-xzMs-Lq8C-1Jbh-QPKihr
    
    

PV goes wrong when re-add /dev/sdb1

  • root@Knoppix:~# mdadm /dev/md0 -a /dev/sdb1
    mdadm: re-added /dev/sdb1
    
  • root@Knoppix:~# pvdisplay /dev/md0
      get_pv_from_vg_by_id: vg_read failed to read VG
      --- NEW Physical volume ---
      PV Name               /dev/md0
      VG Name
      PV Size               1.94 GB
      Allocatable           NO
      PE Size (KByte)       0
      Total PE              0
      Free PE               0
      Allocated PE          0
      PV UUID               A9rTHY-2IOl-mits-PiCZ-WZ5I-gQah-5265YU
    
    
  • PV UUID が変化してしまっている
  • LVMのmetadataが不整合になっている模様
  • root@Knoppix:~# mdadm -E /dev/sd[ab]1 | grep Event
             Events : 0.48
             Events : 0.48
    

erase superblock of sdb1

  • root@Knoppix:~# mdadm --stop /dev/md0
    mdadm: stopped /dev/md0
    
  • root@Knoppix:~# mdadm --zero-superblock /dev/sdb1
  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
    
  • root@Knoppix:~# lvdisplay -C
      LV   VG    Attr   LSize Origin Snap%  Move Log Copy%
      drbd vgnfs -wi--- 1.50G
    

rebuilding /dev/md0

  • root@Knoppix:~# mdadm /dev/md0 -a /dev/sdb1
    mdadm: added /dev/sdb1
    
  • root@Knoppix:~# cat /proc/mdstat
    Personalities : [raid1]
    md0 : active raid1 sdb1[2] sda1[0]
          2047936 blocks [2/1] [U_]
          [================>....]  recovery = 84.7% (1735936/2047936) finish=0.0min speed=96440K/sec
    
    unused devices: <none>
    
  • root@Knoppix:~# mdadm -E /dev/sd[ab]1
    /dev/sda1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Fri Mar 25 17:36:17 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e389af56 - correct
             Events : 0.54
    
    
          Number   Major   Minor   RaidDevice State
    this     0       8        1        0      active sync   /dev/sda1
    
       0     0       8        1        0      active sync   /dev/sda1
       1     1       8       17        1      active sync   /dev/sdb1
    /dev/sdb1:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : 32578407:6f33f50b:893cf340:745f5dce
      Creation Time : Thu Mar 24 17:31:36 2011
         Raid Level : raid1
      Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
         Array Size : 2047936 (2000.27 MiB 2097.09 MB)
       Raid Devices : 2
      Total Devices : 2
    Preferred Minor : 0
    
        Update Time : Fri Mar 25 17:36:17 2011
              State : clean
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
           Checksum : e389af68 - correct
             Events : 0.54
    
    
          Number   Major   Minor   RaidDevice State
    this     1       8       17        1      active sync   /dev/sdb1
    
       0     0       8        1        0      active sync   /dev/sda1
       1     1       8       17        1      active sync   /dev/sdb1
    

result

  • root@Knoppix:~# pvdisplay
      --- Physical volume ---
      PV Name               /dev/md0
      VG Name               vgnfs
      PV Size               1.94 GB / not usable 0
      Allocatable           yes
      PE Size (KByte)       32768
      Total PE              62
      Free PE               14
      Allocated PE          48
      PV UUID               Z2JXRP-fa5g-SYS5-xzMs-Lq8C-1Jbh-QPKihr
    
    
Last modified 13 years ago Last modified on Mar 26, 2011 12:03:30 AM