wiki:TipAndDoc/HA/NFS

Version 8 (modified by mitty, 13 years ago) (diff)

--

HA-NFS

  • Highly Available NFS with DRBD and Pacemaker
  • Ubuntu 10.04 Lucid 2.6.32-30-generic-pae
    • mdadm 2.6.7.1-1ubuntu15
    • lvm2 2.02.54-1ubuntu4.1
    • drbd8-utils 2:8.3.7-1ubuntu2.1
    • nfs-kernel-server 1:1.2.0-4ubuntu4.1
    • pacemaker 1.0.8+hg15494-2ubuntu2
  • Tested with VMware Workstation 7.1.3
  • see log for verbose log

DRBD on LVM with RAID1

  • Primary
    • /dev/sd[bc]1 -> /dev/md0 (RAID1)
    • /dev/md0 -> /dev/vgnfs/drbd (LVM)
  • Secondary
    • /dev/sdb1 -> /dev/vgnfs/drbd (LVM)
  • 全体図
                                 drbd/Primary (ubuntu-haa)
               --------------------------------------------------------------
               |                   -------------------------                |
               |               sda | system[ / ]  |  swap  |                |
               |                   -------------------------                |
               |                                                            |
      NIC(192.168.234.101)      sdb                     sdc                 |
       ||      |   ||      --------------          --------------           |
       ||      ----||------|            |----------|            |------------
       ||          ||      |   Linux    |          |   Linux    |
       ||          ||      |   raid     |          |   raid     |
       ||          ||      | autodetect<=== RAID1 ==>autodetect |
       ||          ||      |------------|    |     |------------|
       ||          ||      --------------    |     --------------
       ||          ||                        |
       ||          ||             ----------LVM----------
       ||          ||             |                 |   |
       ||          ======================drbd       |*1 |  *1: snapshot
       ||                         |                 |   |
       ||                         -----------------------
       ||
       ||
       ||
       ||
       ||
       ||         drbd/Secondar (ubuntu-hab)
       ||      -----------------------------------
       ||      |      -------------------------  |
       ||      |  sda | system[ / ]  |  swap  |  |
       ||      |      -------------------------  |
       ||      |                                 |
      NIC(192.168.234.102)      sdb              |
               |   ||      |------------|        |
               ----||------|            |---------
                   ||      |    LVM     |
                   ||      |            |----|
                   ||      |------------|    |
                   ||      --------------    |
                   ||                        |
                   ||             ----------LVM----------
                   ||             |                 |   |
                   ======================drbd       |*1 |  *1: snapshot
                                  |                 |   |
                                  -----------------------
    

packages

  • sudo aptitude install -R mdadm
    • -R が無いと postfix を同時インストールされる
  • sudo aptitude install lvm2
  • sudo aptitude install drbd8-utils -R
    • -R が無いと build-essential 等を同時インストールされる

setup block devices

ubuntu-haa

fdisk

  • mitty@ubuntu-haa:~$ ls -l /dev/sd?
    brw-rw---- 1 root disk 8,  0 2011-03-24 17:20 /dev/sda
    brw-rw---- 1 root disk 8, 16 2011-03-24 17:20 /dev/sdb
    brw-rw---- 1 root disk 8, 32 2011-03-24 17:20 /dev/sdc
    
  • mitty@ubuntu-haa:~$ sudo fdisk -u /dev/sdb
    Command (m for help): n
    Command action
       e   extended
       p   primary partition (1-4)
    p
    Partition number (1-4): 1
    First sector (63-4194303, default 63):
    Using default value 63
    Last sector, +sectors or +size{K,M,G} (63-4194303, default 4194303): +2000M
    
    Command (m for help): t
    Selected partition 1
    Hex code (type L to list codes): fd
    Changed system type of partition 1 to fd (Linux raid autodetect)
    
    Command (m for help): p
    
    Disk /dev/sdb: 2147 MB, 2147483648 bytes
    255 heads, 63 sectors/track, 261 cylinders, total 4194304 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x7faa0456
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdb1              63     4096063     2048000+  fd  Linux raid autodetect
    Partition 1 does not end on cylinder boundary.
    
    Command (m for help): w
    The partition table has been altered!
    
    Calling ioctl() to re-read partition table.
    Syncing disks.
    
    • 同じ容量のHDDでも、メーカによって微妙にサイズが異なることがあるので、交換する際のことを考えるとディスク全体をパーティションに割り当てない方が良い
  • mitty@ubuntu-haa:~$ sudo fdisk -u /dev/sdc
    Disk identifier: 0x509a8b0b
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdc1              63     4096063     2048000+  fd  Linux raid autodetect
    Partition 1 does not end on cylinder boundary.
    

mdadm

  • mitty@ubuntu-haa:~$ sudo mdadm --create /dev/md0 -l1 -n2 /dev/sd[bc]1
    mdadm: array /dev/md0 started.
    
  • mitty@ubuntu-haa:~$ cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : active raid1 sdc1[1] sdb1[0]
          2047936 blocks [2/2] [UU]
          [>....................]  resync =  0.2% (4160/2047936) finish=8.1min speed=4160K/sec
    
    unused devices: <none>
    
  • mitty@ubuntu-haa:~$ sudo mdadm -D /dev/md0
  • mitty@ubuntu-haa:~$ sudo mdadm -Es
    ARRAY /dev/md0 level=raid1 num-devices=2 UUID=32578407:6f33f50b:893cf340:745f5dce
    
  • mitty@ubuntu-haa:~$ sudo sh -c "mdadm -Es >> /etc/mdadm/mdadm.conf"
  • mitty@ubuntu-haa:~$ sudo dpkg-reconfigure mdadm
      ┌──────────────────────────┤ Configuring mdadm ├─
    
    (snip)
    
    • see log
    • see also wiki:TipAndDoc/RAID#dpkg-reconfiguremdadm?
    • command result
       * Stopping MD monitoring service mdadm --monitor                        [ OK ]
      Generating array device nodes... /var/lib/dpkg/info/mdadm.postinst: line 27: /dev/MAKEDEV: No such file or directory
      failed.
       Removing any system startup links for /etc/init.d/mdadm-raid ...
      update-initramfs: Generating /boot/initrd.img-2.6.32-30-generic-pae
      update-rc.d: warning: mdadm start runlevel arguments (2 3 4 5) do not match LSB Default-Start values (S)
      update-rc.d: warning: mdadm stop runlevel arguments (0 1 6) do not match LSB Default-Stop values (0 6)
       * Starting MD monitoring service mdadm --monitor                        [ OK ]
      

pvcreate/vgcreate/lvcreate

  • mitty@ubuntu-haa:~$ sudo pvcreate /dev/md0
      Physical volume "/dev/md0" successfully created
    
    • see log for detail
  • mitty@ubuntu-haa:~$ sudo vgcreate -s 32 vgnfs /dev/md0
      Volume group "vgnfs" successfully created
    
    • see log for detail
  • mitty@ubuntu-haa:~$ sudo lvcreate --size 1.5G -n drbd vgnfs
      Logical volume "drbd" created
    
    • see log for detail

ubuntu-hab

fdisk

  • mitty@ubuntu-hab:~$ sudo fdisk -u /dev/sdb
    Command (m for help): n
    Command action
       e   extended
       p   primary partition (1-4)
    p
    Partition number (1-4): 1
    First sector (63-4194303, default 63):
    Using default value 63
    Last sector, +sectors or +size{K,M,G} (63-4194303, default 4194303): +2000M
    
    Command (m for help): t
    Selected partition 1
    Hex code (type L to list codes): 8e
    Changed system type of partition 1 to 8e (Linux LVM)
    
    Command (m for help): p
    
    Disk /dev/sdb: 2147 MB, 2147483648 bytes
    255 heads, 63 sectors/track, 261 cylinders, total 4194304 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x4ab4a81c
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdb1              63     4096063     2048000+  8e  Linux LVM
    Partition 1 does not end on cylinder boundary.
    
    Command (m for help): w
    The partition table has been altered!
    
    Calling ioctl() to re-read partition table.
    Syncing disks.
    

pvcreate/vgcreate/lvcreate

  • mitty@ubuntu-hab:~$ sudo pvcreate /dev/sdb1
      Physical volume "/dev/sdb1" successfully created
    
  • mitty@ubuntu-hab:~$ sudo vgcreate -s 32 vgnfs /dev/sdb1
      Volume group "vgnfs" successfully created
    
  • mitty@ubuntu-hab:~$ sudo lvcreate --size 1.5G -n drbd vgnfs
      Logical volume "drbd" created
    
    • see log for detail