5652
Comment: zpool create
|
← Revision 16 as of 2016-08-03 17:08:37 ⇥
4931
|
Deletions are marked like this. | Additions are marked like this. |
Line 3: | Line 3: |
Molmol is the File Server. See also: [[MoneyMoneyMoney/NewStorage]]. | Molmol is the File Server. See also: [[MoneyMoneyMoney/NewStorage]], where it was motivated and specc'd out. |
Line 8: | Line 10: |
== Root Filesystem == | == System Drives == |
Line 12: | Line 14: |
* Partition 1 on both stores the boot loader (Grub). * Partition 2 on sda will contain a FreeBSD partition. * Partition 2 on sdb stores a RAID group which holds a LVM group, containing /, /boot and swap. |
* Partition 1 on both stores the boot loader (FreeBSD's `gptboot`). * Partition 2 on both contains a FreeBSD GEOM MIRROR (with a GPT type of `freebsd-ufs`) containing the root file system. |
Line 17: | Line 18: |
* Partition 3 on both are two swap partitions, both of which are enabled as separate swap devices (i.e. spanned, not mirrored). | |
Line 19: | Line 21: |
root@molmol:~# gdisk -l /dev/sda Found valid GPT with protective MBR; using GPT. Disk /dev/sda: 500118192 sectors, 238.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 77FC147A-5A20-486B-88E2-9EA0FAEC4D15 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 500118158 Partitions will be aligned on 2-sector boundaries Total free space is 1 sectors (512 bytes) |
root@molmol ~# gpart show # Start Size Part# Type => 34 500118125 ada0 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G) |
Line 29: | Line 31: |
Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 BIOS boot partition 2 2048 83886080 40.0 GiB FD00 molmol-system 3 83886082 88080385 2.0 GiB A504 molmol-slog 4 88080386 500118158 196.5 GiB A504 molmol-l2arc }}} {{{ root@molmol:~# gdisk -l /dev/sdb Found valid GPT with protective MBR; using GPT. Disk /dev/sdb: 500118192 sectors, 238.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 80F4D19D-44F2-4851-90B5-E7CBEC7B23C3 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 500118158 Partitions will be aligned on 2-sector boundaries Total free space is 1 sectors (512 bytes) Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 BIOS boot partition 2 2048 83886080 40.0 GiB FD00 molmol-system 3 83886082 88080385 2.0 GiB A504 molmol-slog 4 88080386 500118158 196.5 GiB A504 molmol-l2arc |
=> 34 500118125 ada1 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs [bootme] (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G) |
Line 53: | Line 40: |
{{{ root@molmol:~# cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdb2[1] 41909120 blocks super 1.2 [2/1] [_U] unused devices: <none> }}} {{{ root@molmol:~# lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert boot molmol -wi-ao-- 512.00m root molmol -wi-ao-- 30.00g swap molmol -wi-ao-- 4.00g root@molmol:~# pvs PV VG Fmt Attr PSize PFree /dev/md1 molmol lvm2 a-- 39.96g 5.46g }}} |
The GEOM MIRROR RAID is monitored by setting `daily_status_gmirror_enable="YES"` in `/etc/periodic.conf`. |
Line 82: | Line 50: |
scan: resilvered 0 in 0h0m with 0 errors on Tue Jul 29 12:42:34 2014 | scan: scrub repaired 0 in 2h9m with 0 errors on Tue Oct 21 21:30:01 2014 |
Line 85: | Line 53: |
NAME STATE READ WRITE CKSUM space ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKN64 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8371196 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A83E2255 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8372507 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPU73 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPT93 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83JZD83 ONLINE 0 0 0 ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKM57 ONLINE 0 0 0 |
NAME STATE READ WRITE CKSUM space ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 |
Line 100: | Line 68: |
mirror-4 ONLINE 0 0 0 ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part3 ONLINE 0 0 0 ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part3 ONLINE 0 0 0 |
mirror-4 ONLINE 0 0 0 gpt/molmol-slog ONLINE 0 0 0 gpt/molmol-slog0 ONLINE 0 0 0 |
Line 104: | Line 72: |
ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part4 ONLINE 0 0 0 ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part4 ONLINE 0 0 0 |
gpt/molmol-l2arc ONLINE 0 0 0 gpt/molmol-l2arc0 ONLINE 0 0 0 |
Line 107: | Line 75: |
errors: No known data errors | |
Line 113: | Line 80: |
space 40.4M 3.57T 168K /space space/away 136K 3.57T 136K /space/away space/scratch 39.2M 3.57T 39.2M /space/scratch space/services 136K 3.57T 136K /space/services space/vmstore 136K 3.57T 136K /space/vmstore |
space 1.73T 1.83T 200K /space space/away 909G 1.83T 909G /space/away space/mp3s 281G 1.83T 281G /space/mp3s space/scratch 192K 1.83T 192K /space/scratch space/services 585G 1.83T 585G /space/services space/vmstore 144K 1.83T 144K /space/vmstore |
Line 120: | Line 88: |
GLHFDD. | Compression is on: `zfs set compression=lzjb space` - you can see how well it's working with `zfs get compressratio` |
Line 122: | Line 90: |
== FreeBSD == | == Other notes == In order to prevent a hang on shutdown, the following ([[https://github.com/pcbsd/freebsd/commit/a895b53fb9285924e343686e1857bc391109b87d|sourced from PCBSD]])has been added to `/etc/sysctl.conf`: |
Line 124: | Line 95: |
zpool create space mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7 log mirror /dev/gpt/molmol-slog0 /dev/gpt/molmol-slog cache /dev/gpt/molmol-l2arc0 /dev/gpt/molmol-l2arc | # Don't hang on shutdown when using USB disks hw.usb.no_shutdown_wait=1 |
Line 126: | Line 98: |
== HOW DO I FREEBSD == Don't panic. The documentation is super good - start with [[https://www.freebsd.org/doc/handbook/|The FreeBSD Handbook]]. There is a clear separation between "base system" and "additional software" in FreeBSD - more along the lines of how Windows does it. Most "additional software" gets installed and configured in `/usr/local`, so if in doubt, look there (e.g. `/usr/local/etc`). Install new software with [[https://www.freebsd.org/cgi/man.cgi?query=pkg&sektion=8|pkg(8)]] (e.g. `pkg install zsh`). Configuration of base system services is done with `/etc/rc.conf`, and additional software services in `/usr/local/etc/rc.conf`. Use `zpool` and `zfs` to manipulate ZFS - see [[https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/|the ZFS on Linux tutorial for a good introduction to the concepts]]. ---- CategoryMachines |
Molmol is the File Server. See also: MoneyMoneyMoney/NewStorage, where it was motivated and specc'd out.
Hardware
The "Sick of your moaning" design is *almost* what we have. Except the PSU, case, SAS expander card, and (hopefully soon) an SSD bay. It is basically the same. People who care to correct this.
System Drives
There are 2 SSDs, partitioned with a GUID Partition Table (GPT).
Partition 1 on both stores the boot loader (FreeBSD's gptboot).
Partition 2 on both contains a FreeBSD GEOM MIRROR (with a GPT type of freebsd-ufs) containing the root file system.
- Partition 3 on both forms a mirror for the ZFS SLOG (journal).
- Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache).
- Partition 3 on both are two swap partitions, both of which are enabled as separate swap devices (i.e. spanned, not mirrored).
root@molmol ~# gpart show # Start Size Part# Type => 34 500118125 ada0 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G) => 34 500118125 ada1 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs [bootme] (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G)
The GEOM MIRROR RAID is monitored by setting daily_status_gmirror_enable="YES" in /etc/periodic.conf.
ZFS
Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/
root@molmol:~# zpool status pool: space state: ONLINE scan: scrub repaired 0 in 2h9m with 0 errors on Tue Oct 21 21:30:01 2014 config: NAME STATE READ WRITE CKSUM space ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 gpt/molmol-slog ONLINE 0 0 0 gpt/molmol-slog0 ONLINE 0 0 0 cache gpt/molmol-l2arc ONLINE 0 0 0 gpt/molmol-l2arc0 ONLINE 0 0 0
root@molmol:~# zfs list NAME USED AVAIL REFER MOUNTPOINT space 1.73T 1.83T 200K /space space/away 909G 1.83T 909G /space/away space/mp3s 281G 1.83T 281G /space/mp3s space/scratch 192K 1.83T 192K /space/scratch space/services 585G 1.83T 585G /space/services space/vmstore 144K 1.83T 144K /space/vmstore
Compression is on: zfs set compression=lzjb space - you can see how well it's working with zfs get compressratio
Other notes
In order to prevent a hang on shutdown, the following (sourced from PCBSD)has been added to /etc/sysctl.conf:
# Don't hang on shutdown when using USB disks hw.usb.no_shutdown_wait=1
HOW DO I FREEBSD
Don't panic. The documentation is super good - start with The FreeBSD Handbook.
There is a clear separation between "base system" and "additional software" in FreeBSD - more along the lines of how Windows does it. Most "additional software" gets installed and configured in /usr/local, so if in doubt, look there (e.g. /usr/local/etc).
Install new software with pkg(8) (e.g. pkg install zsh).
Configuration of base system services is done with /etc/rc.conf, and additional software services in /usr/local/etc/rc.conf.
Use zpool and zfs to manipulate ZFS - see the ZFS on Linux tutorial for a good introduction to the concepts.