Diff for "Molmol"

Differences between revisions 6 and 8 (spanning 2 versions)

molmol

Molmol is the File Server. See also: MoneyMoneyMoney/NewStorage.

Hardware

The "Sick of your moaning" design is *almost* what we have. Except the PSU, case, SAS expander card, and (hopefully soon) an SSD bay. It is basically the same. People who care to correct this.

Root Filesystem

There are 2 SSDs, partitioned with a GUID Partition Table (GPT).

Partition 1 on both stores the boot loader (Grub). Partition 2 on sda will contain a FreeBSD partition. Partition 2 on sdb stores a RAID which holds a LVM group, containing /, /boot and swap. Partition 3 on both forms a mirror for the ZFS SLOG (journal). Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache).

root@molmol:~# gdisk -l /dev/sda
Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 500118192 sectors, 238.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 77FC147A-5A20-486B-88E2-9EA0FAEC4D15
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 500118158
Partitions will be aligned on 2-sector boundaries
Total free space is 1 sectors (512 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  BIOS boot partition
   2            2048        83886080   40.0 GiB    FD00  molmol-system
   3        83886082        88080385   2.0 GiB     A504  molmol-slog
   4        88080386       500118158   196.5 GiB   A504  molmol-l2arc

root@molmol:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Disk /dev/sdb: 500118192 sectors, 238.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 80F4D19D-44F2-4851-90B5-E7CBEC7B23C3
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 500118158
Partitions will be aligned on 2-sector boundaries
Total free space is 1 sectors (512 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  BIOS boot partition
   2            2048        83886080   40.0 GiB    FD00  molmol-system
   3        83886082        88080385   2.0 GiB     A504  molmol-slog
   4        88080386       500118158   196.5 GiB   A504  molmol-l2arc

root@molmol:~# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sdb2[1]
      41909120 blocks super 1.2 [2/1] [_U]
      
unused devices: <none>

root@molmol:~# lvs
  LV   VG     Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  boot molmol -wi-ao-- 512.00m                                           
  root molmol -wi-ao--  30.00g                                           
  swap molmol -wi-ao--   4.00g                                           
root@molmol:~# pvs
  PV         VG     Fmt  Attr PSize  PFree
  /dev/md1   molmol lvm2 a--  39.96g 5.46g

ZFS

Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/

root@molmol:~# zpool status
  pool: space
 state: ONLINE
  scan: resilvered 0 in 0h0m with 0 errors on Tue Jul 29 12:42:34 2014
config:

        NAME                                                      STATE     READ WRITE CKSUM
        space                                                     ONLINE       0     0     0
          mirror-0                                                ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKN64              ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8371196              ONLINE       0     0     0
          mirror-1                                                ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A83E2255              ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8372507              ONLINE       0     0     0
          mirror-2                                                ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPU73              ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPT93              ONLINE       0     0     0
          mirror-3                                                ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83JZD83              ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKM57              ONLINE       0     0     0
        logs
          mirror-4                                                ONLINE       0     0     0
            ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part3  ONLINE       0     0     0
            ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part3  ONLINE       0     0     0
        cache
          ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part4    ONLINE       0     0     0
          ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part4    ONLINE       0     0     0

errors: No known data errors

root@molmol:~# zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
space           40.4M  3.57T   168K  /space
space/away       136K  3.57T   136K  /space/away
space/scratch   39.2M  3.57T  39.2M  /space/scratch
space/services   136K  3.57T   136K  /space/services
space/vmstore    136K  3.57T   136K  /space/vmstore

GLHFDD.

-  ⇤ ← Revision 6 as of 2014-07-27 20:46:12 → 
  Size: 5011
  Editor: SamMoore
  Comment:
+   ← Revision 8 as of 2014-07-29 13:20:13 → ⇥
  Size: 5419
  Editor: DavidAdam
  Comment: new ZFS layout
-Deletions are marked like this.
+Additions are marked like this.
 Line 10:
-There are 2 SSDs with 3 partitions in RAID1.
+There are 2 SSDs, partitioned with a GUID Partition Table (GPT).
 Line 12:
-  * /dev/md0 is /dev/sda1 and /dev/sdb1 - /boot, 512MB, ext2
  * /dev/md1 is /dev/sda2 and /dev/sdb2 - swap, 4GB
  * /dev/md2 is /dev/sda5 and /dev/sdb5 - /, LVM ext4, 227GB

In theory this will work.
In practice it did work but only after a reinstall (keeping the first install's disk layouts).
{{attachment:toerrornottoerr.jpg|toerrornottoerr|width=100}}
{{attachment:grubrescue.jpg|grubrescue|width=100}}

Swap is in raid1 because [SLX] wanted swap but not in raid1, and I wanted `/dev/sda` and `/dev/sdb` to be identical, so I put swap on both in raid1.
We probably don't need swap. It's 2% of the disk. Deal with it.
+Partition 1 on both stores the boot loader (Grub).
Partition 2 on sda will contain a FreeBSD partition.
Partition 2 on sdb stores a RAID which holds a LVM group, containing /, /boot and swap.
Partition 3 on both forms a mirror for the ZFS SLOG (journal).
Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache).
-Line 25:
+Line 19:
-root@molmol:~# cat /proc/mdstat
Personalities : [raid1] 
md2 : active raid1 sdb5[3] sda5[2]
      245520192 blocks super 1.2 [2/2] [UU]
      
md1 : active raid1 sda2[2] sdb2[1]
      3904448 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 sdb1[3] sda1[2]
      498368 blocks super 1.2 [2/2] [UU]
+root@molmol:~# gdisk -l /dev/sda
Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 500118192 sectors, 238.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 77FC147A-5A20-486B-88E2-9EA0FAEC4D15
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 500118158
Partitions will be aligned on 2-sector boundaries
Total free space is 1 sectors (512 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  BIOS boot partition
   2            2048        83886080   40.0 GiB    FD00  molmol-system
   3        83886082        88080385   2.0 GiB     A504  molmol-slog
   4        88080386       500118158   196.5 GiB   A504  molmol-l2arc
}}}
{{{
root@molmol:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Disk /dev/sdb: 500118192 sectors, 238.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 80F4D19D-44F2-4851-90B5-E7CBEC7B23C3
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 500118158
Partitions will be aligned on 2-sector boundaries
Total free space is 1 sectors (512 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02  BIOS boot partition
   2            2048        83886080   40.0 GiB    FD00  molmol-system
   3        83886082        88080385   2.0 GiB     A504  molmol-slog
   4        88080386       500118158   196.5 GiB   A504  molmol-l2arc
-Line 38:
+Line 54:
-root@molmol:~# fdisk -l /dev/sda /dev/sdb
+root@molmol:~# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sdb2[1]
      41909120 blocks super 1.2 [2/1] [_U]
      
unused devices: <none>
}}}
-Line 40:
+Line 62:
-Disk /dev/sda: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0004e00d

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      999423      498688   fd  Linux raid autodetect
/dev/sda2          999424     8812543     3906560   fd  Linux raid autodetect
/dev/sda3         8814590   500117503   245651457    5  Extended
/dev/sda5         8814592   500117503   245651456   fd  Linux raid autodetect

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048      999423      498688   fd  Linux raid autodetect
/dev/sdb2          999424     8812543     3906560   fd  Linux raid autodetect
/dev/sdb3         8814590   500117503   245651457    5  Extended
/dev/sdb5         8814592   500117503   245651456   fd  Linux raid autodetect\
+{{{
root@molmol:~# lvs
  LV   VG     Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  boot molmol -wi-ao-- 512.00m                                           
  root molmol -wi-ao--  30.00g                                           
  swap molmol -wi-ao--   4.00g                                           
root@molmol:~# pvs
  PV         VG     Fmt  Attr PSize  PFree
  /dev/md1   molmol lvm2 a--  39.96g 5.46g
-Line 73:
+Line 76:
-Followed this guide: [[http://bernaerts.dyndns.org/linux/75-debian/279-debian-wheezy-zfs-raidz-pool|http://bernaerts.dyndns.org/linux/75-debian/279-debian-wheezy-zfs-raidz-pool]]

Didn't setup snapshots.

{{{
root@molmol:~# zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
over  7.25T  1.64M  7.25T     0%  1.00x  ONLINE  -
}}}
+Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/
-Line 85:
+Line 80:
-  pool: over
+  pool: space
-Line 87:
+Line 82:
-  scan: scrub repaired 0 in 0h0m with 0 errors on Sun Jul 27 18:02:05 2014
+  scan: resilvered 0 in 0h0m with 0 errors on Tue Jul 29 12:42:34 2014
-Line 90:
+Line 85:
- NAME        STATE     READ WRITE CKSUM
 over        ONLINE       0     0     0
   raidz1-0  ONLINE       0     0     0
     sdc     ONLINE       0     0     0
     sdd     ONLINE       0     0     0
     sde     ONLINE       0     0     0
     sdf     ONLINE       0     0     0
     sdg     ONLINE       0     0     0
     sdh     ONLINE       0     0     0
     sdi     ONLINE       0     0     0
     sdj     ONLINE       0     0     0
+        NAME                                                      STATE     READ WRITE CKSUM
        space                                                     ONLINE       0     0     0
          mirror-0                                                ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKN64              ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8371196              ONLINE       0     0     0
          mirror-1                                                ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A83E2255              ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8372507              ONLINE       0     0     0
          mirror-2                                                ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPU73              ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPT93              ONLINE       0     0     0
          mirror-3                                                ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83JZD83              ONLINE       0     0     0
            ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKM57              ONLINE       0     0     0
        logs
          mirror-4                                                ONLINE       0     0     0
            ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part3  ONLINE       0     0     0
            ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part3  ONLINE       0     0     0
        cache
          ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part4    ONLINE       0     0     0
          ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part4    ONLINE       0     0     0
-Line 105:
+Line 110:
-The `zfs` is `/there` and is currently exported to [[Motsugo]].
-Line 108:
+Line 111:
-root@molmol:~# df -h
Filesystem                                    Size  Used Avail Use% Mounted on
rootfs                                        227G  1.5G  214G   1% /
udev                                           10M     0   10M   0% /dev
tmpfs                                         1.6G  540K  1.6G   1% /run
/dev/mapper/molmol-molmol--rootfs             227G  1.5G  214G   1% /
tmpfs                                         5.0M     0  5.0M   0% /run/lock
tmpfs                                         3.9G     0  3.9G   0% /run/shm
/dev/md0                                      457M   19M  414M   5% /boot
services.ucc.gu.uwa.edu.au:/space/away/home   1.9T  1.7T   26G  99% /away
home.ucc.gu.uwa.edu.au:/home                  2.0T  952G  963G  50% /home
nortel.ucc.gu.uwa.edu.au:/vol/space/services  884G  674G  211G  77% /services
over/there                                    6.1T  256K  6.1T   1% /there
}}}

{{{
root@molmol:~# cat /etc/exports
/there	motsugo(rw,sync,no_root_squash,mountpoint,no_subtree_check,secure)
+root@molmol:~# zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
space           40.4M  3.57T   168K  /space
space/away       136K  3.57T   136K  /space/away
space/scratch   39.2M  3.57T  39.2M  /space/scratch
space/services   136K  3.57T   136K  /space/services
space/vmstore    136K  3.57T   136K  /space/vmstore

Wiki

Page

User

Hardware

Root Filesystem

ZFS