Differences between revisions 13 and 16 (spanning 3 versions)
Revision 13 as of 2014-09-11 13:33:13
Size: 5887
Editor: NickBannon
Comment: backlink to CategoryMachines
Revision 16 as of 2016-08-03 17:08:37
Size: 4931
Editor: chubsucker
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
Molmol is the File Server. See also: [[MoneyMoneyMoney/NewStorage]]. Molmol is the File Server. See also: [[MoneyMoneyMoney/NewStorage]], where it was motivated and specc'd out.

Line 8: Line 10:
== Root Filesystem == == System Drives ==
Line 12: Line 14:
 * Partition 1 on both stores the boot loader (Grub).
 * Partition 2 on sda will contain a FreeBSD partition.
 * Partition 2 on sdb stores
a RAID group which holds a LVM group, containing /, /boot and swap.
 * Partition 1 on both stores the boot loader (FreeBSD's `gptboot`).
 * Partition 2 on both contains a FreeBSD GEOM MIRROR (with a GPT type of `freebsd-ufs`) containing the root file system.
Line 17: Line 18:
 * Partition 3 on both are two swap partitions, both of which are enabled as separate swap devices (i.e. spanned, not mirrored).
Line 19: Line 21:
root@molmol:~# gdisk -l /dev/sda
Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 500118192 sectors, 238.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 77FC147A-5A20-486B-88E2-9EA0FAEC4D15
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 500118158
Partitions will be aligned on 2-sector boundaries
Total free space is 1 sectors (512 bytes)
root@molmol ~# gpart show
# Start Size Part# Type
=> 34 500118125 ada0 GPT (238G)
         34 1000 1 freebsd-boot (500K)
       1034 1014 - free - (507K)
       2048 79691649 2 freebsd-ufs (38G)
   79693697 4192385 5 freebsd-swap (2.0G)
   83886082 4194304 3 freebsd-zfs (2.0G)
   88080386 412037773 4 freebsd-zfs (196G)
Line 29: Line 31:
Number Start (sector) End (sector) Size Code Name
   1 34 2047 1007.0 KiB EF02 BIOS boot partition
   2 2048 83886080 40.0 GiB FD00 molmol-system
   3 83886082 88080385 2.0 GiB A504 molmol-slog
   4 88080386 500118158 196.5 GiB A504 molmol-l2arc
}}}
{{{
root@molmol:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Disk /dev/sdb: 500118192 sectors, 238.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 80F4D19D-44F2-4851-90B5-E7CBEC7B23C3
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 500118158
Partitions will be aligned on 2-sector boundaries
Total free space is 1 sectors (512 bytes)

Number Start (sector) End (sector) Size Code Name
   1 34 2047 1007.0 KiB EF02 BIOS boot partition
   2 2048 83886080 40.0 GiB FD00 molmol-system
   3 83886082 88080385 2.0 GiB A504 molmol-slog
   4 88080386 500118158 196.5 GiB A504 molmol-l2arc
=> 34 500118125 ada1 GPT (238G)
         34 1000 1 freebsd-boot (500K)
       1034 1014 - free - (507K)
       2048 79691649 2 freebsd-ufs [bootme] (38G)
   79693697 4192385 5 freebsd-swap (2.0G)
   83886082 4194304 3 freebsd-zfs (2.0G)
   88080386 412037773 4 freebsd-zfs (196G)
Line 53: Line 40:
{{{
root@molmol:~# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb2[1]
      41909120 blocks super 1.2 [2/1] [_U]
      
unused devices: <none>
}}}

{{{
root@molmol:~# lvs
  LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
  boot molmol -wi-ao-- 512.00m
  root molmol -wi-ao-- 30.00g
  swap molmol -wi-ao-- 4.00g
root@molmol:~# pvs
  PV VG Fmt Attr PSize PFree
  /dev/md1 molmol lvm2 a-- 39.96g 5.46g
}}}
The GEOM MIRROR RAID is monitored by setting `daily_status_gmirror_enable="YES"` in `/etc/periodic.conf`.
Line 82: Line 50:
  scan: resilvered 0 in 0h0m with 0 errors on Tue Jul 29 12:42:34 2014   scan: scrub repaired 0 in 2h9m with 0 errors on Tue Oct 21 21:30:01 2014
Line 85: Line 53:
        NAME STATE READ WRITE CKSUM
        space ONLINE 0 0 0
          mirror-0 ONLINE 0 0 0
            ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKN64 ONLINE 0 0 0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8371196 ONLINE 0 0 0
          mirror-1 ONLINE 0 0 0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A83E2255 ONLINE 0 0 0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXF1A8372507 ONLINE 0 0 0
          mirror-2 ONLINE 0 0 0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPU73 ONLINE 0 0 0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83KPT93 ONLINE 0 0 0
          mirror-3 ONLINE 0 0 0
            ata-WDC_WD10JFCX-68N6GN0_WD-WXM1E83JZD83 ONLINE 0 0 0
            ata-WDC_WD10JFCX-68N6GN0_WD-WX11E83HKM57 ONLINE 0 0 0
        NAME STATE READ WRITE CKSUM
        space ONLINE 0 0 0
          mirror-0 ONLINE 0 0 0
            da0 ONLINE 0 0 0
            da1 ONLINE 0 0 0
          mirror-1 ONLINE 0 0 0
            da2 ONLINE 0 0 0
            da3 ONLINE 0 0 0
          mirror-2 ONLINE 0 0 0
            da4 ONLINE 0 0 0
            da5 ONLINE 0 0 0
          mirror-3 ONLINE 0 0 0
            da6 ONLINE 0 0 0
            da7 ONLINE 0 0 0
Line 100: Line 68:
          mirror-4                                     ONLINE 0 0 0
            ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part3 ONLINE 0 0 0
            ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part3 ONLINE 0 0 0
          mirror-4 ONLINE 0 0 0
            gpt/molmol-slog  ONLINE 0 0 0
            gpt/molmol-slog0 ONLINE 0 0 0
Line 104: Line 72:
          ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864731A-part4 ONLINE 0 0 0
          ata-Samsung_SSD_840_PRO_Series_S1ATNSAD864729Z-part4 ONLINE 0 0 0
          gpt/molmol-l2arc ONLINE 0 0 0
          gpt/molmol-l2arc0 ONLINE 0 0 0
Line 107: Line 75:
errors: No known data errors
Line 113: Line 80:
space 40.4M 3.57T 168K /space
space/away 136K 3.57T 136K /space/away
space/scratch 39.2M 3.57T 39.2M /space/scratch
space/services 136K 3.57T 136K /space/services
space/vmstore 136K 3.57T 136K /space/vmstore
space 1.73T 1.83T 200K /space
space/away 909G 1.83T 909G /space/away
space/mp3s 281G 1.83T 281G /space/mp3s
space/
scratch  192K 1.83T 192K /space/scratch
space/services 585G 1.83T 585G /space/services
space/vmstore 144K 1.83T 144K /space/vmstore
Line 120: Line 88:
GLHFDD. Compression is on: `zfs set compression=lzjb space` - you can see how well it's working with `zfs get compressratio`
Line 122: Line 90:
== FreeBSD == == Other notes ==

In order to prevent a hang on shutdown, the following ([[https://github.com/pcbsd/freebsd/commit/a895b53fb9285924e343686e1857bc391109b87d|sourced from PCBSD]])has been added to `/etc/sysctl.conf`:
Line 124: Line 95:
# Create the zpool layout
zpool create space mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7 log mirror /dev/gpt/molmol-slog0 /dev/gpt/molmol-slog cache /dev/gpt/molmol-l2arc0 /dev/gpt/molmol-l2arc
# Turn on compression
zfs set compression=lzjb space
# Create the zfs layout
zfs create space/away
zfs create space/services
zfs create space/vmstore
zfs create space/scratch
# Don't hang on shutdown when using USB disks
hw.usb.no_shutdown_wait=1
Line 134: Line 98:

== HOW DO I FREEBSD ==

Don't panic. The documentation is super good - start with [[https://www.freebsd.org/doc/handbook/|The FreeBSD Handbook]].

There is a clear separation between "base system" and "additional software" in FreeBSD - more along the lines of how Windows does it. Most "additional software" gets installed and configured in `/usr/local`, so if in doubt, look there (e.g. `/usr/local/etc`).

Install new software with [[https://www.freebsd.org/cgi/man.cgi?query=pkg&sektion=8|pkg(8)]] (e.g. `pkg install zsh`).

Configuration of base system services is done with `/etc/rc.conf`, and additional software services in `/usr/local/etc/rc.conf`.

Use `zpool` and `zfs` to manipulate ZFS - see [[https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/|the ZFS on Linux tutorial for a good introduction to the concepts]].

molmol

Molmol is the File Server. See also: MoneyMoneyMoney/NewStorage, where it was motivated and specc'd out.

Hardware

The "Sick of your moaning" design is *almost* what we have. Except the PSU, case, SAS expander card, and (hopefully soon) an SSD bay. It is basically the same. People who care to correct this.

System Drives

There are 2 SSDs, partitioned with a GUID Partition Table (GPT).

  • Partition 1 on both stores the boot loader (FreeBSD's gptboot).

  • Partition 2 on both contains a FreeBSD GEOM MIRROR (with a GPT type of freebsd-ufs) containing the root file system.

  • Partition 3 on both forms a mirror for the ZFS SLOG (journal).
  • Partition 4 on both forms a spanned (i.e. not mirrored) ZFS L2ARC (disk cache).
  • Partition 3 on both are two swap partitions, both of which are enabled as separate swap devices (i.e. spanned, not mirrored).

root@molmol ~# gpart show
#    Start        Size  Part# Type
=>       34  500118125  ada0  GPT  (238G)
         34       1000     1  freebsd-boot  (500K)
       1034       1014        - free -  (507K)
       2048   79691649     2  freebsd-ufs  (38G)
   79693697    4192385     5  freebsd-swap  (2.0G)
   83886082    4194304     3  freebsd-zfs  (2.0G)
   88080386  412037773     4  freebsd-zfs  (196G)

=>       34  500118125  ada1  GPT  (238G)
         34       1000     1  freebsd-boot  (500K)
       1034       1014        - free -  (507K)
       2048   79691649     2  freebsd-ufs  [bootme]  (38G)
   79693697    4192385     5  freebsd-swap  (2.0G)
   83886082    4194304     3  freebsd-zfs  (2.0G)
   88080386  412037773     4  freebsd-zfs  (196G)

The GEOM MIRROR RAID is monitored by setting daily_status_gmirror_enable="YES" in /etc/periodic.conf.

ZFS

Assembled the RAID along the lines of https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/

root@molmol:~# zpool status
  pool: space
 state: ONLINE
  scan: scrub repaired 0 in 2h9m with 0 errors on Tue Oct 21 21:30:01 2014
config:

        NAME                  STATE     READ WRITE CKSUM
        space                 ONLINE       0     0     0
          mirror-0            ONLINE       0     0     0
            da0               ONLINE       0     0     0
            da1               ONLINE       0     0     0
          mirror-1            ONLINE       0     0     0
            da2               ONLINE       0     0     0
            da3               ONLINE       0     0     0
          mirror-2            ONLINE       0     0     0
            da4               ONLINE       0     0     0
            da5               ONLINE       0     0     0
          mirror-3            ONLINE       0     0     0
            da6               ONLINE       0     0     0
            da7               ONLINE       0     0     0
        logs
          mirror-4            ONLINE       0     0     0
            gpt/molmol-slog   ONLINE       0     0     0
            gpt/molmol-slog0  ONLINE       0     0     0
        cache
          gpt/molmol-l2arc    ONLINE       0     0     0
          gpt/molmol-l2arc0   ONLINE       0     0     0

root@molmol:~# zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
space           1.73T  1.83T   200K  /space
space/away       909G  1.83T   909G  /space/away
space/mp3s       281G  1.83T   281G  /space/mp3s
space/scratch    192K  1.83T   192K  /space/scratch
space/services   585G  1.83T   585G  /space/services
space/vmstore    144K  1.83T   144K  /space/vmstore

Compression is on: zfs set compression=lzjb space - you can see how well it's working with zfs get compressratio

Other notes

In order to prevent a hang on shutdown, the following (sourced from PCBSD)has been added to /etc/sysctl.conf:

# Don't hang on shutdown when using USB disks
hw.usb.no_shutdown_wait=1

HOW DO I FREEBSD

Don't panic. The documentation is super good - start with The FreeBSD Handbook.

There is a clear separation between "base system" and "additional software" in FreeBSD - more along the lines of how Windows does it. Most "additional software" gets installed and configured in /usr/local, so if in doubt, look there (e.g. /usr/local/etc).

Install new software with pkg(8) (e.g. pkg install zsh).

Configuration of base system services is done with /etc/rc.conf, and additional software services in /usr/local/etc/rc.conf.

Use zpool and zfs to manipulate ZFS - see the ZFS on Linux tutorial for a good introduction to the concepts.


CategoryMachines