wiki:TipAndDoc/storage/RAID/rescue/RAID1

Version 8 (modified by mitty, 14 years ago) (diff)

--

  1. RAID1
  2. two of two devices are alive and clean
  3. rescue with Knoppix
    1. devices
    2. modprobe
    3. scan devices
    4. method 1
    5. method 2
    6. result
  4. degraded RAID1
    1. devices
    2. modprobe
    3. method 1
    4. method 2
    5. result
  5. two of two devices alive but not same
    1. make Logical volume on degraded RAID1
      1. status
      2. make /dev/vgnfs/drbd
      3. mdadm Events increased
  6. rescue with two devices on Knoppix
    1. devices
    2. assemble md0 and md1
      1. assemble md0
      2. assemble md1
      3. scanning LVM
      4. start md1
      5. pvchange duplicate PV
      6. new Physical volume
      7. stop md0
    3. re-assemble /dev/md0 with two device
      1. stop md0 and md1
      2. mdadm status
      3. assemble /dev/md0
      4. assemble /dev/md0 with /dev/sda1
      5. re-add /dev/sdb1 to /dev/md0
      6. something wrong on PV on RAID1
      7. correct PV with degraded RAID1
      8. PV goes wrong when re-add /dev/sdb1
      9. erase superblock of sdb1
      10. rebuilding /dev/md0
    4. result

RAID1

  • mdadm による Software RAID1 の復旧テスト

two of two devices are alive and clean

  • see wiki:TipAndDoc/HA/NFS#mdadm for how to make device
  • 以下は済んでいるが、Logical volumeは作っていない
    1. mitty@ubuntu-haa:~$ sudo pvcreate /dev/md0
    2. mitty@ubuntu-haa:~$ sudo vgcreate -s 32 vgnfs /dev/md0
  • mitty@ubuntu-haa:~$ sudo mdadm -D /dev/md0 
    /dev/md0:
        Version : 00.90
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
         Events : 0.34

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
  • mitty@ubuntu-haa:~$ sudo mdadm --stop /dev/md0 
    mdadm: stopped /dev/md0
  • mitty@ubuntu-haa:~$ sudo mdadm -E /dev/sd[bc]1 
    /dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e3889399 - correct
         Events : 34


      Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       8       33        1      active sync   /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e38893ab - correct
         Events : 34


      Number   Major   Minor   RaidDevice State
this     1       8       33        1      active sync   /dev/sdc1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       8       33        1      active sync   /dev/sdc1

rescue with Knoppix

  • root@Knoppix:~# cat /etc/knoppix-version 
    5.3.1 2008-03-26
  • root@Knoppix:~# uname -a 
    Linux Knoppix 2.6.24.4 #2 SMP PREEMPT Tue Mar 25 21:35:28 CET 2008 i686 GNU/Linux

devices

  • root@Knoppix:~# ls -l /dev/sd? 
    brw-rw---- 1 root disk 8,   0 Mar 25  2011 /dev/sda
brw-rw---- 1 root disk 8,  16 Mar 25  2011 /dev/sdb

modprobe

  • root@Knoppix:~# cat /proc/mdstat 
    cat: /proc/mdstat: No such file or directory
  • root@Knoppix:~# modprobe md_mod
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities :
unused devices: <none>

scan devices

  • root@Knoppix:~# mdadm -E /dev/sd* 2>/dev/null 
    /dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e3889399 - correct
         Events : 0.34


      Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       8       33        1      active sync   /dev/sdc1
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e38893ab - correct
         Events : 0.34


      Number   Major   Minor   RaidDevice State
this     1       8       33        1      active sync   /dev/sdc1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       8       33        1      active sync   /dev/sdc1
  • root@Knoppix:~# mdadm -Es 
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=32578407:6f33f50b:893cf340:745f5dce

method 1

  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce 
    mdadm: /dev/md0 has been started with 2 drives.
    • 以下のようなエラーが出た場合は、modprobe md_modし忘れている場合がある 
      mdadm: error opening /dev/md0: No such device or address

method 2

  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sd[ab]1 
    mdadm: /dev/md0 has been started with 2 drives.

result

  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
md0 : active raid1 sda1[0] sdb1[1]
      2047936 blocks [2/2] [UU]

unused devices: <none>
  • root@Knoppix:~# mdadm -Ds 
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=32578407:6f33f50b:893cf340:745f5dce
  • root@Knoppix:~# mdadm -D /dev/md0 
    /dev/md0:
        Version : 00.90.03
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 32578407:6f33f50b:893cf340:745f5dce
         Events : 0.34

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
  • root@Knoppix:~# pvdisplay -C 
      PV         VG    Fmt  Attr PSize PFree
  /dev/md0   vgnfs lvm2 a-   1.94G 1.94G
  • root@Knoppix:~# vgdisplay -C 
      VG    #PV #LV #SN Attr   VSize VFree
  vgnfs   1   0   0 wz--n- 1.94G 1.94G

degraded RAID1

devices

  • root@Knoppix:~# ls -l /dev/sd? 
    brw-rw---- 1 root disk 8,   0 Mar 25  2011 /dev/sda

modprobe

  • root@Knoppix:~# cat /proc/mdstat 
    cat: /proc/mdstat: No such file or directory
  • root@Knoppix:~# modprobe md_mod
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities :
unused devices: <none>
  • root@Knoppix:~# mdadm -E /dev/sda1 
    /dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e3889399 - correct
         Events : 0.34


      Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       8       33        1      active sync   /dev/sdc1
  • root@Knoppix:~# mdadm -Es 
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=32578407:6f33f50b:893cf340:745f5dce

method 1

  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce 
    mdadm: /dev/md0 has been started with 1 drive (out of 2).

method 2

  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sda1 
    mdadm: /dev/md0 has been started with 1 drive (out of 2).

result

  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
md0 : active raid1 sda1[0]
      2047936 blocks [2/1] [U_]

unused devices: <none>
  • root@Knoppix:~# mdadm -Ds 
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=32578407:6f33f50b:893cf340:745f5dce
  • root@Knoppix:~# mdadm -D /dev/md0 
    /dev/md0:
        Version : 00.90.03
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : 32578407:6f33f50b:893cf340:745f5dce
         Events : 0.34

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       0        0        1      removed

two of two devices alive but not same

  • degraded RAID1に対して、(片方のHDDだけattachされている状態で)何らかの操作を行い、後から古いHDDをre-addした際の挙動を調べる

make Logical volume on degraded RAID1

status

  • mitty@ubuntu-haa:~$ ls -l /dev/sd* 
    brw-rw---- 1 root disk 8, 16 2011-03-25 16:16 /dev/sdb
brw-rw---- 1 root disk 8, 17 2011-03-25 16:16 /dev/sdb1
  • mitty@ubuntu-haa:~$ cat /proc/mdstat 
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : inactive sdb1[0](S)
      2047936 blocks

unused devices: <none>
  • mitty@ubuntu-haa:~$ sudo mdadm -E /dev/sdb1 
    /dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e3889399 - correct
         Events : 34


      Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       8       33        1      active sync
  • mitty@ubuntu-haa:~$ sudo mdadm -R /dev/md0 
    mdadm: started /dev/md0
  • mitty@ubuntu-haa:~$ cat /proc/mdstat 
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[0]
      2047936 blocks [2/1] [U_]

unused devices: <none>

make /dev/vgnfs/drbd

  • mitty@ubuntu-haa:~$ sudo lvcreate --size 1.5G -n drbd vgnfs 
      Logical volume "drbd" created
  • mitty@ubuntu-haa:~$ sudo lvdisplay 
      --- Logical volume ---
  LV Name                /dev/vgnfs/drbd
  VG Name                vgnfs
  LV UUID                dNxdNj-hZCZ-GMrC-woMk-0hA2-f3oR-sBsrHI
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                1.50 GiB
  Current LE             48
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:0

mdadm Events increased

  • mitty@ubuntu-haa:~$ sudo mdadm -E /dev/sdb1 
    /dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce (local to host ubuntu-haa)
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0

    Update Time : Fri Mar 25 15:28:34 2011
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0
       Checksum : e389914b - correct
         Events : 36


      Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       0        0        1      faulty removed

rescue with two devices on Knoppix

devices

  • root@Knoppix:~# mdadm -E /dev/sd[ab]1 
    /dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0

    Update Time : Fri Mar 25 15:28:34 2011
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0
       Checksum : e389914b - correct
         Events : 0.36


      Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       0        0        1      faulty removed
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Thu Mar 24 21:25:32 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e38893ab - correct
         Events : 0.34


      Number   Major   Minor   RaidDevice State
this     1       8       33        1      active sync   /dev/sdc1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       8       33        1      active sync   /dev/sdc1

assemble md0 and md1

  • 上記の # mdadm -E /dev/sd[ab]1 の結果から、RAID1デバイスのうち /dev/sda1 が最新であると推定されるが、その確認として /dev/sd[ab]1 をそれぞれRAID1としてAssembleする

assemble md0

  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce 
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
md0 : active raid1 sda1[0]
      2047936 blocks [2/1] [U_]

unused devices: <none>
    • 自動的に最新のデバイスが選択されるのか、/dev/sda1から/dev/md0がassembleされる
    • どちらが最新であるかどのように判断しているかが不明 (Events? Update Time?)

assemble md1

  • root@Knoppix:~# mdadm -A /dev/md1 /dev/sdb1 
    mdadm: /dev/md1 assembled from 1 drive - need all 2 to start it (use --run to insist).
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
md1 : inactive sdb1[1](S)
      2047936 blocks

md0 : active raid1 sda1[0]
      2047936 blocks [2/1] [U_]

unused devices: <none>
    • この時点では md1 は動いていない

scanning LVM

  • /dev/sda1 -> /dev/md0 のみが有効なので、/dev/vgnfs/drbdが正常に表示される
  • root@Knoppix:~# pvscan 
      PV /dev/md0   VG vgnfs   lvm2 [1.94 GB / 448.00 MB free]
  Total: 1 [1.94 GB] / in use: 1 [1.94 GB] / in no VG: 0 [0   ]
  • root@Knoppix:~# vgscan 
      Reading all physical volumes.  This may take a while...
  Found volume group "vgnfs" using metadata type lvm2
  • root@Knoppix:~# lvscan 
      inactive          '/dev/vgnfs/drbd' [1.50 GB] inherit
  • root@Knoppix:~# lvdisplay 
      --- Logical volume ---
  LV Name                /dev/vgnfs/drbd
  VG Name                vgnfs
  LV UUID                dNxdNj-hZCZ-GMrC-woMk-0hA2-f3oR-sBsrHI
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                1.50 GB
  Current LE             48
  Segments               1
  Allocation             inherit
  Read ahead sectors     0

start md1

  • root@Knoppix:~# mdadm -R /dev/md1 
    mdadm: started /dev/md1
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
md1 : active raid1 sdb1[1]
      2047936 blocks [2/1] [_U]

md0 : active raid1 sda1[0]
      2047936 blocks [2/1] [U_]

unused devices: <none>

pvchange duplicate PV

  • root@Knoppix:~# pvscan 
      Found duplicate PV Z2JXRPfa5gSYS5xzMsLq8C1JbhQPKihr: using /dev/md1 not /dev/md0
  PV /dev/md1   VG vgnfs   lvm2 [1.94 GB / 1.94 GB free]
  Total: 1 [1.94 GB] / in use: 1 [1.94 GB] / in no VG: 0 [0   ]
  • root@Knoppix:~# pvchange -u /dev/md1 
      Found duplicate PV Z2JXRPfa5gSYS5xzMsLq8C1JbhQPKihr: using /dev/md0 not /dev/md1
  Found duplicate PV Z2JXRPfa5gSYS5xzMsLq8C1JbhQPKihr: using /dev/md1 not /dev/md0
  Physical volume "/dev/md1" changed
  1 physical volume changed / 0 physical volumes not changed
  • root@Knoppix:~# mdadm -E /dev/sd[ab]1 
    /dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0

    Update Time : Fri Mar 25 15:28:34 2011
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0
       Checksum : e389914b - correct
         Events : 0.36


      Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       0        0        1      faulty removed
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 1

    Update Time : Fri Mar 25 16:58:00 2011
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e389a642 - correct
         Events : 0.36


      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       0        0        0      removed
   1     1       8       17        1      active sync   /dev/sdb1
    • /dev/sdb1 -> /dev/md1 の Events が 0.34 -> 0.36 と増加

new Physical volume

  • UUIDを変更した場合、双方のPVが表示されるようになるが、VGは片方のみ表示される
  • root@Knoppix:~# pvdisplay /dev/md[01] 
      --- Physical volume ---
  PV Name               /dev/md0
  VG Name               vgnfs
  PV Size               1.94 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              62
  Free PE               14
  Allocated PE          48
  PV UUID               Z2JXRP-fa5g-SYS5-xzMs-Lq8C-1Jbh-QPKihr

  --- NEW Physical volume ---
  PV Name               /dev/md1
  VG Name
  PV Size               1.94 GB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               A9rTHY-2IOl-mits-PiCZ-WZ5I-gQah-5265YU
  • root@Knoppix:~# vgdisplay 
      --- Volume group ---
  VG Name               vgnfs
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.94 GB
  PE Size               32.00 MB
  Total PE              62
  Alloc PE / Size       48 / 1.50 GB
  Free  PE / Size       14 / 448.00 MB
  VG UUID               I6vVoh-6gCJ-9uvA-v2MV-Fyva-7J8v-Cvftfi

stop md0

  • /dev/md0 を止めることで、 /dev/md1 上のVGを表示出来る
  • root@Knoppix:~# mdadm --stop /dev/md0 
    mdadm: stopped /dev/md0
  • root@Knoppix:~# pvdisplay /dev/md[01] 
      No physical volume label read from /dev/md0
  Failed to read physical volume "/dev/md0"
  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               vgnfs
  PV Size               1.94 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              62
  Free PE               62
  Allocated PE          0
  PV UUID               A9rTHY-2IOl-mits-PiCZ-WZ5I-gQah-5265YU
  • root@Knoppix:~# vgdisplay 
      --- Volume group ---
  VG Name               vgnfs
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.94 GB
  PE Size               32.00 MB
  Total PE              62
  Alloc PE / Size       0 / 0
  Free  PE / Size       62 / 1.94 GB
  VG UUID               I6vVoh-6gCJ-9uvA-v2MV-Fyva-7J8v-Cvftfi
    • /dev/md1 はLVが作られていないので、PEが全てFreeのまま

re-assemble /dev/md0 with two device

  • #assemblemd0andmd1 の過程で、PVのUUID等が不一致となったデバイスからRAID1を再構築する

stop md0 and md1

  • root@Knoppix:~# mdadm --stop /dev/md0 
    mdadm: stopped /dev/md0
  • root@Knoppix:~# mdadm --stop /dev/md1 
    mdadm: stopped /dev/md1
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
unused devices: <none>

mdadm status

  • root@Knoppix:~# mdadm -E /dev/sd[ab]1 
    /dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0

    Update Time : Fri Mar 25 15:28:34 2011
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0
       Checksum : e389914b - correct
         Events : 0.36


      Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       0        0        1      faulty removed
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 1

    Update Time : Fri Mar 25 16:58:00 2011
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e389a642 - correct
         Events : 0.36


      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       0        0        0      removed
   1     1       8       17        1      active sync   /dev/sdb1

assemble /dev/md0

  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sd[ab]1 
    mdadm: device 1 in /dev/md0 has wrong state in superblock, but /dev/sdb1 seems ok
mdadm: failed to add /dev/sda1 to /dev/md0: Invalid argument
mdadm: /dev/md0 has been started with 1 drive (out of 2).
  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce 
    mdadm: device 1 in /dev/md0 has wrong state in superblock, but /dev/sdb1 seems ok
mdadm: failed to add /dev/sda1 to /dev/md0: Invalid argument
mdadm: /dev/md0 has been started with 1 drive (out of 2).
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
md0 : active raid1 sdb1[1]
      2047936 blocks [2/1] [_U]

unused devices: <none>
  • Update Time がより新しい /dev/sdb1 から /dev/md0 が構成されてしまう
  • root@Knoppix:~# lvdisplay -C
    • no output

assemble /dev/md0 with /dev/sda1

  • 明示的に /dev/sda1 から /dev/md0 を再構成する
  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sda1 
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
md0 : active raid1 sda1[0]
      2047936 blocks [2/1] [U_]

unused devices: <none>
  • root@Knoppix:~# lvdisplay -C 
      LV   VG    Attr   LSize Origin Snap%  Move Log Copy%
  drbd vgnfs -wi--- 1.50G
    • LVの drbd が復活する

re-add /dev/sdb1 to /dev/md0

  • root@Knoppix:~# mdadm /dev/md0 -a /dev/sdb1 
    mdadm: re-added /dev/sdb1
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
      2047936 blocks [2/2] [UU]

unused devices: <none>
  • root@Knoppix:~# dmesg | tail -n 20 
    RAID1 conf printout:
 --- wd:1 rd:2
 disk 0, wo:0, o:1, dev:sda1
 disk 1, wo:1, o:1, dev:sdb1
md: recovery of RAID array md0
md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
md: using 128k window, over a total of 2047936 blocks.
md: md0: recovery done.
RAID1 conf printout:
 --- wd:2 rd:2
 disk 0, wo:0, o:1, dev:sda1
 disk 1, wo:0, o:1, dev:sdb1
  • root@Knoppix:~# mdadm -E /dev/sd[ab]1 
    /dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Fri Mar 25 17:13:19 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e389a9dc - correct
         Events : 0.42


      Number   Major   Minor   RaidDevice State
this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Fri Mar 25 17:13:19 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e389a9ee - correct
         Events : 0.42


      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
    • Events がさらに増加して 0.42 となる

something wrong on PV on RAID1

  • /dev/sda1 と /dev/sdb1 が同期されたように見えて、実際にはされていない？
  • root@Knoppix:~# mdadm --stop /dev/md0 
    mdadm: stopped /dev/md0
  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce 
    mdadm: /dev/md0 has been started with 2 drives.
  • root@Knoppix:~# pvdisplay 
      get_pv_from_vg_by_id: vg_read failed to read VG
  --- NEW Physical volume ---
  PV Name               /dev/md0
  VG Name
  PV Size               1.95 GB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               Z2JXRP-fa5g-SYS5-xzMs-Lq8C-1Jbh-QPKihr

correct PV with degraded RAID1

  • root@Knoppix:~# mdadm --stop /dev/md0 
    mdadm: stopped /dev/md0
  • root@Knoppix:~# mdadm -A /dev/md0 /dev/sda1 --run 
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
  • root@Knoppix:~# pvdisplay /dev/md0 
      --- Physical volume ---
  PV Name               /dev/md0
  VG Name               vgnfs
  PV Size               1.94 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              62
  Free PE               14
  Allocated PE          48
  PV UUID               Z2JXRP-fa5g-SYS5-xzMs-Lq8C-1Jbh-QPKihr

PV goes wrong when re-add /dev/sdb1

  • root@Knoppix:~# mdadm /dev/md0 -a /dev/sdb1 
    mdadm: re-added /dev/sdb1
  • root@Knoppix:~# pvdisplay /dev/md0 
      get_pv_from_vg_by_id: vg_read failed to read VG
  --- NEW Physical volume ---
  PV Name               /dev/md0
  VG Name
  PV Size               1.94 GB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               A9rTHY-2IOl-mits-PiCZ-WZ5I-gQah-5265YU
  • PV UUID が変化してしまっている
  • LVMのmetadataが不整合になっている模様
  • root@Knoppix:~# mdadm -E /dev/sd[ab]1 | grep Event 
             Events : 0.48
         Events : 0.48

erase superblock of sdb1

  • root@Knoppix:~# mdadm --stop /dev/md0 
    mdadm: stopped /dev/md0
  • root@Knoppix:~# mdadm --zero-superblock /dev/sdb1
  • root@Knoppix:~# mdadm -A /dev/md0 -u 32578407:6f33f50b:893cf340:745f5dce 
    mdadm: /dev/md0 has been started with 1 drive (out of 2).
  • root@Knoppix:~# lvdisplay -C 
      LV   VG    Attr   LSize Origin Snap%  Move Log Copy%
  drbd vgnfs -wi--- 1.50G

rebuilding /dev/md0

  • root@Knoppix:~# mdadm /dev/md0 -a /dev/sdb1 
    mdadm: added /dev/sdb1
  • root@Knoppix:~# cat /proc/mdstat 
    Personalities : [raid1]
md0 : active raid1 sdb1[2] sda1[0]
      2047936 blocks [2/1] [U_]
      [================>....]  recovery = 84.7% (1735936/2047936) finish=0.0min speed=96440K/sec

unused devices: <none>
  • root@Knoppix:~# mdadm -E /dev/sd[ab]1 
    /dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Fri Mar 25 17:36:17 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e389af56 - correct
         Events : 0.54


      Number   Major   Minor   RaidDevice State
this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 32578407:6f33f50b:893cf340:745f5dce
  Creation Time : Thu Mar 24 17:31:36 2011
     Raid Level : raid1
  Used Dev Size : 2047936 (2000.27 MiB 2097.09 MB)
     Array Size : 2047936 (2000.27 MiB 2097.09 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Fri Mar 25 17:36:17 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e389af68 - correct
         Events : 0.54


      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1

result

  • root@Knoppix:~# pvdisplay 
      --- Physical volume ---
  PV Name               /dev/md0
  VG Name               vgnfs
  PV Size               1.94 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              62
  Free PE               14
  Allocated PE          48
  PV UUID               Z2JXRP-fa5g-SYS5-xzMs-Lq8C-1Jbh-QPKihr