5102
Comment: updated new disk layouts
|
← Revision 16 as of 2016-08-03 17:08:37 ⇥
4931
|
Deletions are marked like this. | Additions are marked like this. |
Line 3: | Line 3: |
Molmol is the File Server. See also: [[MoneyMoneyMoney/NewStorage]]. | Molmol is the File Server. See also: [[MoneyMoneyMoney/NewStorage]], where it was motivated and specc'd out. |
Line 8: | Line 10: |
== Root Filesystem == | == System Drives == |
Line 12: | Line 14: |
Partition 1 on both stores the boot loader (Grub). Partition 2 on sda will contain a FreeBSD partition. Partition 2 on sdb stores a RAID which holds a LVM group, containing /, /boot and swap. Partition 3 on both forms a mirror for the ZFS SLOG (journal). Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache). |
* Partition 1 on both stores the boot loader (FreeBSD's `gptboot`). * Partition 2 on both contains a FreeBSD GEOM MIRROR (with a GPT type of `freebsd-ufs`) containing the root file system. * Partition 3 on both forms a mirror for the ZFS SLOG (journal). * Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache). * Partition 3 on both are two swap partitions, both of which are enabled as separate swap devices (i.e. spanned, not mirrored). |
Line 19: | Line 21: |
root@molmol:~# gdisk -l /dev/sda Found valid GPT with protective MBR; using GPT. Disk /dev/sda: 500118192 sectors, 238.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 77FC147A-5A20-486B-88E2-9EA0FAEC4D15 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 500118158 Partitions will be aligned on 2-sector boundaries Total free space is 1 sectors (512 bytes) |
root@molmol ~# gpart show # Start Size Part# Type => 34 500118125 ada0 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G) |
Line 29: | Line 31: |
Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 BIOS boot partition 2 2048 83886080 40.0 GiB FD00 molmol-system 3 83886082 88080385 2.0 GiB A504 molmol-slog 4 88080386 500118158 196.5 GiB A504 molmol-l2arc |
=> 34 500118125 ada1 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs [bootme] (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G) |
Line 35: | Line 39: |
The GEOM MIRROR RAID is monitored by setting `daily_status_gmirror_enable="YES"` in `/etc/periodic.conf`. == ZFS == Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/ |
|
Line 36: | Line 47: |
root@molmol:~# gdisk -l /dev/sdb Found valid GPT with protective MBR; using GPT. Disk /dev/sdb: 500118192 sectors, 238.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 80F4D19D-44F2-4851-90B5-E7CBEC7B23C3 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 500118158 Partitions will be aligned on 2-sector boundaries Total free space is 1 sectors (512 bytes) |
root@molmol:~# zpool status pool: space state: ONLINE scan: scrub repaired 0 in 2h9m with 0 errors on Tue Oct 21 21:30:01 2014 config: |
Line 46: | Line 53: |
Number Start (sector) End (sector) Size Code Name 1 34 2047 1007.0 KiB EF02 BIOS boot partition 2 2048 83886080 40.0 GiB FD00 molmol-system 3 83886082 88080385 2.0 GiB A504 molmol-slog 4 88080386 500118158 196.5 GiB A504 molmol-l2arc |
NAME STATE READ WRITE CKSUM space ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 gpt/molmol-slog ONLINE 0 0 0 gpt/molmol-slog0 ONLINE 0 0 0 cache gpt/molmol-l2arc ONLINE 0 0 0 gpt/molmol-l2arc0 ONLINE 0 0 0 |
Line 54: | Line 78: |
root@molmol:~# cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdb2[1] 41909120 blocks super 1.2 [2/1] [_U] unused devices: <none> |
root@molmol:~# zfs list NAME USED AVAIL REFER MOUNTPOINT space 1.73T 1.83T 200K /space space/away 909G 1.83T 909G /space/away space/mp3s 281G 1.83T 281G /space/mp3s space/scratch 192K 1.83T 192K /space/scratch space/services 585G 1.83T 585G /space/services space/vmstore 144K 1.83T 144K /space/vmstore |
Line 62: | Line 88: |
Compression is on: `zfs set compression=lzjb space` - you can see how well it's working with `zfs get compressratio` == Other notes == In order to prevent a hang on shutdown, the following ([[https://github.com/pcbsd/freebsd/commit/a895b53fb9285924e343686e1857bc391109b87d|sourced from PCBSD]])has been added to `/etc/sysctl.conf`: |
|
Line 63: | Line 95: |
root@molmol:~# lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert boot molmol -wi-ao-- 512.00m root molmol -wi-ao-- 30.00g swap molmol -wi-ao-- 4.00g root@molmol:~# pvs PV VG Fmt Attr PSize PFree /dev/md1 molmol lvm2 a-- 39.96g 5.46g |
# Don't hang on shutdown when using USB disks hw.usb.no_shutdown_wait=1 |
Line 73: | Line 99: |
== HOW DO I FREEBSD == | |
Line 74: | Line 101: |
Don't panic. The documentation is super good - start with [[https://www.freebsd.org/doc/handbook/|The FreeBSD Handbook]]. | |
Line 75: | Line 103: |
== ZFS == | There is a clear separation between "base system" and "additional software" in FreeBSD - more along the lines of how Windows does it. Most "additional software" gets installed and configured in `/usr/local`, so if in doubt, look there (e.g. `/usr/local/etc`). |
Line 77: | Line 105: |
Followed this guide: [[http://bernaerts.dyndns.org/linux/75-debian/279-debian-wheezy-zfs-raidz-pool|http://bernaerts.dyndns.org/linux/75-debian/279-debian-wheezy-zfs-raidz-pool]] | Install new software with [[https://www.freebsd.org/cgi/man.cgi?query=pkg&sektion=8|pkg(8)]] (e.g. `pkg install zsh`). |
Line 79: | Line 107: |
Didn't setup snapshots. | Configuration of base system services is done with `/etc/rc.conf`, and additional software services in `/usr/local/etc/rc.conf`. |
Line 81: | Line 109: |
{{{ root@molmol:~# zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT over 7.25T 1.64M 7.25T 0% 1.00x ONLINE - }}} |
Use `zpool` and `zfs` to manipulate ZFS - see [[https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/|the ZFS on Linux tutorial for a good introduction to the concepts]]. |
Line 87: | Line 111: |
{{{ root@molmol:~# zpool status pool: over state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Sun Jul 27 18:02:05 2014 config: NAME STATE READ WRITE CKSUM over ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 sdi ONLINE 0 0 0 sdj ONLINE 0 0 0 errors: No known data errors }}} The `zfs` is `/there` and is currently exported to [[Motsugo]]. {{{ root@molmol:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 227G 1.5G 214G 1% / udev 10M 0 10M 0% /dev tmpfs 1.6G 540K 1.6G 1% /run /dev/mapper/molmol-molmol--rootfs 227G 1.5G 214G 1% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /run/shm /dev/md0 457M 19M 414M 5% /boot services.ucc.gu.uwa.edu.au:/space/away/home 1.9T 1.7T 26G 99% /away home.ucc.gu.uwa.edu.au:/home 2.0T 952G 963G 50% /home nortel.ucc.gu.uwa.edu.au:/vol/space/services 884G 674G 211G 77% /services over/there 6.1T 256K 6.1T 1% /there }}} {{{ root@molmol:~# cat /etc/exports /there motsugo(rw,sync,no_root_squash,mountpoint,no_subtree_check,secure) }}} GLHFDD. |
---- CategoryMachines |
Molmol is the File Server. See also: MoneyMoneyMoney/NewStorage, where it was motivated and specc'd out.
Hardware
The "Sick of your moaning" design is *almost* what we have. Except the PSU, case, SAS expander card, and (hopefully soon) an SSD bay. It is basically the same. People who care to correct this.
System Drives
There are 2 SSDs, partitioned with a GUID Partition Table (GPT).
Partition 1 on both stores the boot loader (FreeBSD's gptboot).
Partition 2 on both contains a FreeBSD GEOM MIRROR (with a GPT type of freebsd-ufs) containing the root file system.
- Partition 3 on both forms a mirror for the ZFS SLOG (journal).
- Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache).
- Partition 3 on both are two swap partitions, both of which are enabled as separate swap devices (i.e. spanned, not mirrored).
root@molmol ~# gpart show # Start Size Part# Type => 34 500118125 ada0 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G) => 34 500118125 ada1 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs [bootme] (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G)
The GEOM MIRROR RAID is monitored by setting daily_status_gmirror_enable="YES" in /etc/periodic.conf.
ZFS
Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/
root@molmol:~# zpool status pool: space state: ONLINE scan: scrub repaired 0 in 2h9m with 0 errors on Tue Oct 21 21:30:01 2014 config: NAME STATE READ WRITE CKSUM space ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 gpt/molmol-slog ONLINE 0 0 0 gpt/molmol-slog0 ONLINE 0 0 0 cache gpt/molmol-l2arc ONLINE 0 0 0 gpt/molmol-l2arc0 ONLINE 0 0 0
root@molmol:~# zfs list NAME USED AVAIL REFER MOUNTPOINT space 1.73T 1.83T 200K /space space/away 909G 1.83T 909G /space/away space/mp3s 281G 1.83T 281G /space/mp3s space/scratch 192K 1.83T 192K /space/scratch space/services 585G 1.83T 585G /space/services space/vmstore 144K 1.83T 144K /space/vmstore
Compression is on: zfs set compression=lzjb space - you can see how well it's working with zfs get compressratio
Other notes
In order to prevent a hang on shutdown, the following (sourced from PCBSD)has been added to /etc/sysctl.conf:
# Don't hang on shutdown when using USB disks hw.usb.no_shutdown_wait=1
HOW DO I FREEBSD
Don't panic. The documentation is super good - start with The FreeBSD Handbook.
There is a clear separation between "base system" and "additional software" in FreeBSD - more along the lines of how Windows does it. Most "additional software" gets installed and configured in /usr/local, so if in doubt, look there (e.g. /usr/local/etc).
Install new software with pkg(8) (e.g. pkg install zsh).
Configuration of base system services is done with /etc/rc.conf, and additional software services in /usr/local/etc/rc.conf.
Use zpool and zfs to manipulate ZFS - see the ZFS on Linux tutorial for a good introduction to the concepts.