5011
Comment:
|
← Revision 16 as of 2016-08-03 17:08:37 ⇥
4931
|
Deletions are marked like this. | Additions are marked like this. |
Line 3: | Line 3: |
Molmol is the File Server. See also: [[MoneyMoneyMoney/NewStorage]]. | Molmol is the File Server. See also: [[MoneyMoneyMoney/NewStorage]], where it was motivated and specc'd out. |
Line 8: | Line 10: |
== Root Filesystem == | == System Drives == |
Line 10: | Line 12: |
There are 2 SSDs with 3 partitions in RAID1. | There are 2 SSDs, partitioned with a GUID Partition Table (GPT). |
Line 12: | Line 14: |
* /dev/md0 is /dev/sda1 and /dev/sdb1 - /boot, 512MB, ext2 * /dev/md1 is /dev/sda2 and /dev/sdb2 - swap, 4GB * /dev/md2 is /dev/sda5 and /dev/sdb5 - /, LVM ext4, 227GB In theory this will work. In practice it did work but only after a reinstall (keeping the first install's disk layouts). {{attachment:toerrornottoerr.jpg|toerrornottoerr|width=100}} {{attachment:grubrescue.jpg|grubrescue|width=100}} Swap is in raid1 because [SLX] wanted swap but not in raid1, and I wanted `/dev/sda` and `/dev/sdb` to be identical, so I put swap on both in raid1. We probably don't need swap. It's 2% of the disk. Deal with it. |
* Partition 1 on both stores the boot loader (FreeBSD's `gptboot`). * Partition 2 on both contains a FreeBSD GEOM MIRROR (with a GPT type of `freebsd-ufs`) containing the root file system. * Partition 3 on both forms a mirror for the ZFS SLOG (journal). * Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache). * Partition 3 on both are two swap partitions, both of which are enabled as separate swap devices (i.e. spanned, not mirrored). |
Line 25: | Line 21: |
root@molmol:~# cat /proc/mdstat Personalities : [raid1] md2 : active raid1 sdb5[3] sda5[2] 245520192 blocks super 1.2 [2/2] [UU] md1 : active raid1 sda2[2] sdb2[1] 3904448 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdb1[3] sda1[2] 498368 blocks super 1.2 [2/2] [UU] |
root@molmol ~# gpart show # Start Size Part# Type => 34 500118125 ada0 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G) => 34 500118125 ada1 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs [bootme] (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G) }}} The GEOM MIRROR RAID is monitored by setting `daily_status_gmirror_enable="YES"` in `/etc/periodic.conf`. == ZFS == Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/ {{{ root@molmol:~# zpool status pool: space state: ONLINE scan: scrub repaired 0 in 2h9m with 0 errors on Tue Oct 21 21:30:01 2014 config: NAME STATE READ WRITE CKSUM space ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 gpt/molmol-slog ONLINE 0 0 0 gpt/molmol-slog0 ONLINE 0 0 0 cache gpt/molmol-l2arc ONLINE 0 0 0 gpt/molmol-l2arc0 ONLINE 0 0 0 |
Line 38: | Line 78: |
root@molmol:~# fdisk -l /dev/sda /dev/sdb Disk /dev/sda: 256.1 GB, 256060514304 bytes 255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0004e00d Device Boot Start End Blocks Id System /dev/sda1 * 2048 999423 498688 fd Linux raid autodetect /dev/sda2 999424 8812543 3906560 fd Linux raid autodetect /dev/sda3 8814590 500117503 245651457 5 Extended /dev/sda5 8814592 500117503 245651456 fd Linux raid autodetect WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted. Disk /dev/sdb: 256.1 GB, 256060514304 bytes 255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 * 2048 999423 498688 fd Linux raid autodetect /dev/sdb2 999424 8812543 3906560 fd Linux raid autodetect /dev/sdb3 8814590 500117503 245651457 5 Extended /dev/sdb5 8814592 500117503 245651456 fd Linux raid autodetect\ |
root@molmol:~# zfs list NAME USED AVAIL REFER MOUNTPOINT space 1.73T 1.83T 200K /space space/away 909G 1.83T 909G /space/away space/mp3s 281G 1.83T 281G /space/mp3s space/scratch 192K 1.83T 192K /space/scratch space/services 585G 1.83T 585G /space/services space/vmstore 144K 1.83T 144K /space/vmstore |
Line 70: | Line 88: |
Compression is on: `zfs set compression=lzjb space` - you can see how well it's working with `zfs get compressratio` | |
Line 71: | Line 90: |
== ZFS == | == Other notes == |
Line 73: | Line 92: |
Followed this guide: [[http://bernaerts.dyndns.org/linux/75-debian/279-debian-wheezy-zfs-raidz-pool|http://bernaerts.dyndns.org/linux/75-debian/279-debian-wheezy-zfs-raidz-pool]] Didn't setup snapshots. |
In order to prevent a hang on shutdown, the following ([[https://github.com/pcbsd/freebsd/commit/a895b53fb9285924e343686e1857bc391109b87d|sourced from PCBSD]])has been added to `/etc/sysctl.conf`: |
Line 78: | Line 95: |
root@molmol:~# zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT over 7.25T 1.64M 7.25T 0% 1.00x ONLINE - |
# Don't hang on shutdown when using USB disks hw.usb.no_shutdown_wait=1 |
Line 83: | Line 99: |
{{{ root@molmol:~# zpool status pool: over state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Sun Jul 27 18:02:05 2014 config: |
== HOW DO I FREEBSD == |
Line 90: | Line 101: |
NAME STATE READ WRITE CKSUM over ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 sdi ONLINE 0 0 0 sdj ONLINE 0 0 0 |
Don't panic. The documentation is super good - start with [[https://www.freebsd.org/doc/handbook/|The FreeBSD Handbook]]. |
Line 102: | Line 103: |
errors: No known data errors }}} |
There is a clear separation between "base system" and "additional software" in FreeBSD - more along the lines of how Windows does it. Most "additional software" gets installed and configured in `/usr/local`, so if in doubt, look there (e.g. `/usr/local/etc`). |
Line 105: | Line 105: |
The `zfs` is `/there` and is currently exported to [[Motsugo]]. | Install new software with [[https://www.freebsd.org/cgi/man.cgi?query=pkg&sektion=8|pkg(8)]] (e.g. `pkg install zsh`). |
Line 107: | Line 107: |
{{{ root@molmol:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 227G 1.5G 214G 1% / udev 10M 0 10M 0% /dev tmpfs 1.6G 540K 1.6G 1% /run /dev/mapper/molmol-molmol--rootfs 227G 1.5G 214G 1% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /run/shm /dev/md0 457M 19M 414M 5% /boot services.ucc.gu.uwa.edu.au:/space/away/home 1.9T 1.7T 26G 99% /away home.ucc.gu.uwa.edu.au:/home 2.0T 952G 963G 50% /home nortel.ucc.gu.uwa.edu.au:/vol/space/services 884G 674G 211G 77% /services over/there 6.1T 256K 6.1T 1% /there }}} |
Configuration of base system services is done with `/etc/rc.conf`, and additional software services in `/usr/local/etc/rc.conf`. |
Line 123: | Line 109: |
{{{ root@molmol:~# cat /etc/exports /there motsugo(rw,sync,no_root_squash,mountpoint,no_subtree_check,secure) }}} |
Use `zpool` and `zfs` to manipulate ZFS - see [[https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/|the ZFS on Linux tutorial for a good introduction to the concepts]]. |
Line 128: | Line 111: |
GLHFDD. | ---- CategoryMachines |
Molmol is the File Server. See also: MoneyMoneyMoney/NewStorage, where it was motivated and specc'd out.
Hardware
The "Sick of your moaning" design is *almost* what we have. Except the PSU, case, SAS expander card, and (hopefully soon) an SSD bay. It is basically the same. People who care to correct this.
System Drives
There are 2 SSDs, partitioned with a GUID Partition Table (GPT).
Partition 1 on both stores the boot loader (FreeBSD's gptboot).
Partition 2 on both contains a FreeBSD GEOM MIRROR (with a GPT type of freebsd-ufs) containing the root file system.
- Partition 3 on both forms a mirror for the ZFS SLOG (journal).
- Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache).
- Partition 3 on both are two swap partitions, both of which are enabled as separate swap devices (i.e. spanned, not mirrored).
root@molmol ~# gpart show # Start Size Part# Type => 34 500118125 ada0 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G) => 34 500118125 ada1 GPT (238G) 34 1000 1 freebsd-boot (500K) 1034 1014 - free - (507K) 2048 79691649 2 freebsd-ufs [bootme] (38G) 79693697 4192385 5 freebsd-swap (2.0G) 83886082 4194304 3 freebsd-zfs (2.0G) 88080386 412037773 4 freebsd-zfs (196G)
The GEOM MIRROR RAID is monitored by setting daily_status_gmirror_enable="YES" in /etc/periodic.conf.
ZFS
Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/
root@molmol:~# zpool status pool: space state: ONLINE scan: scrub repaired 0 in 2h9m with 0 errors on Tue Oct 21 21:30:01 2014 config: NAME STATE READ WRITE CKSUM space ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 logs mirror-4 ONLINE 0 0 0 gpt/molmol-slog ONLINE 0 0 0 gpt/molmol-slog0 ONLINE 0 0 0 cache gpt/molmol-l2arc ONLINE 0 0 0 gpt/molmol-l2arc0 ONLINE 0 0 0
root@molmol:~# zfs list NAME USED AVAIL REFER MOUNTPOINT space 1.73T 1.83T 200K /space space/away 909G 1.83T 909G /space/away space/mp3s 281G 1.83T 281G /space/mp3s space/scratch 192K 1.83T 192K /space/scratch space/services 585G 1.83T 585G /space/services space/vmstore 144K 1.83T 144K /space/vmstore
Compression is on: zfs set compression=lzjb space - you can see how well it's working with zfs get compressratio
Other notes
In order to prevent a hang on shutdown, the following (sourced from PCBSD)has been added to /etc/sysctl.conf:
# Don't hang on shutdown when using USB disks hw.usb.no_shutdown_wait=1
HOW DO I FREEBSD
Don't panic. The documentation is super good - start with The FreeBSD Handbook.
There is a clear separation between "base system" and "additional software" in FreeBSD - more along the lines of how Windows does it. Most "additional software" gets installed and configured in /usr/local, so if in doubt, look there (e.g. /usr/local/etc).
Install new software with pkg(8) (e.g. pkg install zsh).
Configuration of base system services is done with /etc/rc.conf, and additional software services in /usr/local/etc/rc.conf.
Use zpool and zfs to manipulate ZFS - see the ZFS on Linux tutorial for a good introduction to the concepts.