4851
Comment:
|
5652
zpool create
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
{{attachment:molmol.jpg|molmol|width=400}} |
|
Line 8: | Line 10: |
There are 2 SSDs with 3 partitions in RAID1. | There are 2 SSDs, partitioned with a GUID Partition Table (GPT). |
Line 10: | Line 12: |
* /dev/md0 is /dev/sda1 and /dev/sdb1 - /boot, 512MB, ext2 * /dev/md1 is /dev/sda2 and /dev/sdb2 - swap, 4GB * /dev/md2 is /dev/sda5 and /dev/sdb5 - /, LVM ext4, 227GB In theory this will work. In practice it did work but only after a reinstall (keeping the first install's disk layouts). Swap is in raid1 because [SLX] wanted swap but not in raid1, and I wanted `/dev/sda` and `/dev/sdb` to be identical, so I put swap on both in raid1. We probably don't need swap. It's 2% of the disk. Deal with it. |
* Partition 1 on both stores the boot loader (Grub). * Partition 2 on sda will contain a FreeBSD partition. * Partition 2 on sdb stores a RAID group which holds a LVM group, containing /, /boot and swap. * Partition 3 on both forms a mirror for the ZFS SLOG (journal). * Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache). |
Line 21: | Line 19: |
root@molmol:~# cat /proc/mdstat Personalities : [raid1] md2 : active raid1 sdb5[3] sda5[2] 245520192 blocks super 1.2 [2/2] [UU] md1 : active raid1 sda2[2] sdb2[1] 3904448 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdb1[3] sda1[2] 498368 blocks super 1.2 [2/2] [UU] |
root@molmol:~# gdisk -l /dev/sda Found valid GPT with protective MBR; using GPT. Disk /dev/sda: 500118192 sectors, 238.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 77FC147A-5A20-486B-88E2-9EA0FAEC4D15 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 500118158 Partitions will be aligned on 2-sector boundaries Total free space is 1 sectors (512 bytes) Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 BIOS boot partition 2 2048 83886080 40.0 GiB FD00 molmol-system 3 83886082 88080385 2.0 GiB A504 molmol-slog 4 88080386 500118158 196.5 GiB A504 molmol-l2arc }}} {{{ root@molmol:~# gdisk -l /dev/sdb Found valid GPT with protective MBR; using GPT. Disk /dev/sdb: 500118192 sectors, 238.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 80F4D19D-44F2-4851-90B5-E7CBEC7B23C3 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 500118158 Partitions will be aligned on 2-sector boundaries Total free space is 1 sectors (512 bytes) Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 BIOS boot partition 2 2048 83886080 40.0 GiB FD00 molmol-system 3 83886082 88080385 2.0 GiB A504 molmol-slog 4 88080386 500118158 196.5 GiB A504 molmol-l2arc |
Line 34: | Line 54: |
root@molmol:~# fdisk -l /dev/sda /dev/sdb | root@molmol:~# cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdb2[1] 41909120 blocks super 1.2 [2/1] [_U] unused devices: <none> }}} |
Line 36: | Line 62: |
Disk /dev/sda: 256.1 GB, 256060514304 bytes 255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0004e00d Device Boot Start End Blocks Id System /dev/sda1 * 2048 999423 498688 fd Linux raid autodetect /dev/sda2 999424 8812543 3906560 fd Linux raid autodetect /dev/sda3 8814590 500117503 245651457 5 Extended /dev/sda5 8814592 500117503 245651456 fd Linux raid autodetect WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted. Disk /dev/sdb: 256.1 GB, 256060514304 bytes 255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 * 2048 999423 498688 fd Linux raid autodetect /dev/sdb2 999424 8812543 3906560 fd Linux raid autodetect /dev/sdb3 8814590 500117503 245651457 5 Extended /dev/sdb5 8814592 500117503 245651456 fd Linux raid autodetect\ |
{{{ root@molmol:~# lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert boot molmol -wi-ao-- 512.00m root molmol -wi-ao-- 30.00g swap molmol -wi-ao-- 4.00g root@molmol:~# pvs PV VG Fmt Attr PSize PFree /dev/md1 molmol lvm2 a-- 39.96g 5.46g |
Line 69: | Line 76: |
Followed this guide: [[http://bernaerts.dyndns.org/linux/75-debian/279-debian-wheezy-zfs-raidz-pool|http://bernaerts.dyndns.org/linux/75-debian/279-debian-wheezy-zfs-raidz-pool]] Didn't setup snapshots. {{{ root@molmol:~# zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT over 7.25T 1.64M 7.25T 0% 1.00x ONLINE - }}} |
Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/ |
Line 81: | Line 80: |
pool: over | pool: space |
Line 83: | Line 82: |
scan: scrub repaired 0 in 0h0m with 0 errors on Sun Jul 27 18:02:05 2014 | scan: resilvered 0 in 0h0m with 0 errors on Tue Jul 29 12:42:34 2014 |
Line 86: | Line 85: |
NAME STATE READ WRITE CKSUM over ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 sdi ONLINE 0 0 0 sdj ONLINE 0 0 0 |
NAME STATE READ WRITE CKSUM space ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKN64 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8371196 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A83E2255 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8372507 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPU73 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPT93 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83JZD83 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKM57 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part3 ONLINE 0 0 0 ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part3 ONLINE 0 0 0 cache ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part4 ONLINE 0 0 0 ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part4 ONLINE 0 0 0 |
Line 101: | Line 110: |
The `zfs` is `/there` and is currently exported to [[Motsugo]]. |
|
Line 104: | Line 111: |
root@molmol:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 227G 1.5G 214G 1% / udev 10M 0 10M 0% /dev tmpfs 1.6G 540K 1.6G 1% /run /dev/mapper/molmol-molmol--rootfs 227G 1.5G 214G 1% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /run/shm /dev/md0 457M 19M 414M 5% /boot services.ucc.gu.uwa.edu.au:/space/away/home 1.9T 1.7T 26G 99% /away home.ucc.gu.uwa.edu.au:/home 2.0T 952G 963G 50% /home nortel.ucc.gu.uwa.edu.au:/vol/space/services 884G 674G 211G 77% /services over/there 6.1T 256K 6.1T 1% /there }}} {{{ root@molmol:~# cat /etc/exports /there motsugo(rw,sync,no_root_squash,mountpoint,no_subtree_check,secure) |
root@molmol:~# zfs list NAME USED AVAIL REFER MOUNTPOINT space 40.4M 3.57T 168K /space space/away 136K 3.57T 136K /space/away space/scratch 39.2M 3.57T 39.2M /space/scratch space/services 136K 3.57T 136K /space/services space/vmstore 136K 3.57T 136K /space/vmstore |
Line 125: | Line 121: |
== FreeBSD == {{{ zpool create space mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7 log mirror /dev/gpt/molmol-slog0 /dev/gpt/molmol-slog cache /dev/gpt/molmol-l2arc0 /dev/gpt/molmol-l2arc }}} |
Molmol is the File Server. See also: MoneyMoneyMoney/NewStorage.
Hardware
The "Sick of your moaning" design is *almost* what we have. Except the PSU, case, SAS expander card, and (hopefully soon) an SSD bay. It is basically the same. People who care to correct this.
Root Filesystem
There are 2 SSDs, partitioned with a GUID Partition Table (GPT).
- Partition 1 on both stores the boot loader (Grub).
- Partition 2 on sda will contain a FreeBSD partition.
- Partition 2 on sdb stores a RAID group which holds a LVM group, containing /, /boot and swap.
- Partition 3 on both forms a mirror for the ZFS SLOG (journal).
- Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache).
root@molmol:~# gdisk -l /dev/sda Found valid GPT with protective MBR; using GPT. Disk /dev/sda: 500118192 sectors, 238.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 77FC147A-5A20-486B-88E2-9EA0FAEC4D15 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 500118158 Partitions will be aligned on 2-sector boundaries Total free space is 1 sectors (512 bytes) Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 BIOS boot partition 2 2048 83886080 40.0 GiB FD00 molmol-system 3 83886082 88080385 2.0 GiB A504 molmol-slog 4 88080386 500118158 196.5 GiB A504 molmol-l2arc
root@molmol:~# gdisk -l /dev/sdb Found valid GPT with protective MBR; using GPT. Disk /dev/sdb: 500118192 sectors, 238.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 80F4D19D-44F2-4851-90B5-E7CBEC7B23C3 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 500118158 Partitions will be aligned on 2-sector boundaries Total free space is 1 sectors (512 bytes) Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 BIOS boot partition 2 2048 83886080 40.0 GiB FD00 molmol-system 3 83886082 88080385 2.0 GiB A504 molmol-slog 4 88080386 500118158 196.5 GiB A504 molmol-l2arc
root@molmol:~# cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdb2[1] 41909120 blocks super 1.2 [2/1] [_U] unused devices: <none>
root@molmol:~# lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert boot molmol -wi-ao-- 512.00m root molmol -wi-ao-- 30.00g swap molmol -wi-ao-- 4.00g root@molmol:~# pvs PV VG Fmt Attr PSize PFree /dev/md1 molmol lvm2 a-- 39.96g 5.46g
ZFS
Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/
root@molmol:~# zpool status pool: space state: ONLINE scan: resilvered 0 in 0h0m with 0 errors on Tue Jul 29 12:42:34 2014 config: NAME STATE READ WRITE CKSUM space ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKN64 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8371196 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A83E2255 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8372507 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPU73 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPT93 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83JZD83 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKM57 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part3 ONLINE 0 0 0 ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part3 ONLINE 0 0 0 cache ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part4 ONLINE 0 0 0 ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part4 ONLINE 0 0 0 errors: No known data errors
root@molmol:~# zfs list NAME USED AVAIL REFER MOUNTPOINT space 40.4M 3.57T 168K /space space/away 136K 3.57T 136K /space/away space/scratch 39.2M 3.57T 39.2M /space/scratch space/services 136K 3.57T 136K /space/services space/vmstore 136K 3.57T 136K /space/vmstore
GLHFDD.
FreeBSD
zpool create space mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7 log mirror /dev/gpt/molmol-slog0 /dev/gpt/molmol-slog cache /dev/gpt/molmol-l2arc0 /dev/gpt/molmol-l2arc