Differences between revisions 148 and 157 (spanning 9 versions)
Revision 148 as of 2021-10-06 15:04:31
Size: 16817
Editor: NickBannon
Comment: recovery - portal/uccportal
Revision 157 as of 2022-03-07 22:00:58
Size: 17817
Editor: NickBannon
Comment: ucc-ansible-soe fully rebuild idle-maltair, idle-medico
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
||[NTU][MTL]||2022-03-07||ucc-ansible-soe fully rebuild idle-maltair, idle-medico, now Debian 11||
||[MPT]||2022-03-03||Git-ify and modernise more of the club website||
||[MTL]||2022-02-28||Upgrade to latest gitlab||
||[MTL]||2022-02-28||Additional packages in UCC Ansible SOE for idlehosts||
||Chris+[DBA][MPT][NTU]||2022-01-24||reracking machop, DELL R710 x 2, ...||
||[NTU]||2022-01-24||motsugo:/tmp full||
||[MTL][NTU]||2022-01-10||quovadis.ucc.asn.au (self-serve letsencrypt) moved to using cloud-mooneye and RFC-2136||
||[MTL][NTU][TPG][333]||2022-01-10||mudkip dead fan 7, degraded fan 1: replaced, now boots happily||
||[MTL][NTU]||2021-12-09||uccmonitor cloud-alchemy grafana upgrade||
||[333]||2021-11-14||Implemented new Tech Todo List in [[https://gitlab.ucc.asn.au/UCC/tech-todo-list|GitLab]] to manage our tech/wheel action items better(er)*||
||[MTL][NTU][BOB]||2021-11-01||machop setup; start proxmox atlantic cluster upgrade v6.3 to v6.4; ansible-driven idle-machop SOE||

UCC Systems Change Log

Please enter a short description of any work performed on UCC systems here.

Whom

When

What

[NTU][MTL]

2022-03-07

ucc-ansible-soe fully rebuild idle-maltair, idle-medico, now Debian 11

[MPT]

2022-03-03

Git-ify and modernise more of the club website

[MTL]

2022-02-28

Upgrade to latest gitlab

[MTL]

2022-02-28

Additional packages in UCC Ansible SOE for idlehosts

Chris+[DBA][MPT][NTU]

2022-01-24

reracking machop, DELL R710 x 2, ...

[NTU]

2022-01-24

motsugo:/tmp full

[MTL][NTU]

2022-01-10

quovadis.ucc.asn.au (self-serve letsencrypt) moved to using cloud-mooneye and RFC-2136

[MTL][NTU][TPG][333]

2022-01-10

mudkip dead fan 7, degraded fan 1: replaced, now boots happily

[MTL][NTU]

2021-12-09

uccmonitor cloud-alchemy grafana upgrade

[333]

2021-11-14

Implemented new Tech Todo List in GitLab to manage our tech/wheel action items better(er)*

[MTL][NTU][BOB]

2021-11-01

machop setup; start proxmox atlantic cluster upgrade v6.3 to v6.4; ansible-driven idle-machop SOE

[BRD][MTL][NTU][BOB][TPG] [TBB][333][pcadmin] [DTA][MDD][SJY][RME]

2021-10-05

power outage 03:18 - recovery!, machop setup

[MTL][NTU][BOB][TPG][333]

2021-10-04

proxmox docs, discord-irc restarts, uccmonitor, magikarp recovery

[MPT][NTU]

2021-09-27

uccportal renewals analysis; update and make agenda.default sourceable/executable

[TPG]

2021-09-27

SSL/TLS cert for eggman

[MTL][NTU]

2021-09-27

Test prometheus/grafana monitoring of SMART on mooneye, motsugo

[BOB]

2021-08-31

Fixed apt sources on samson (was waia) and updated

[BOB]

2021-07-28

Free up space on maaxen and (manually) upgrade to latest available windows updates. Disabled print spooler to prevent printnightmare vulnerabilities

[MPT]

2021-06-30

Create wheel-runner VM and configure Gitlab CD for ucc-fw

[MPT][NTU]

2021-05-17

ucc-announce subscription backlog from ucc-adduser-ad

[TRS]

2021-05-10

Fix BigBlueButton, including upgrading to 2.3.0 release

[MTL]

2021-05-07

Fix webdispense

[MTL]

2021-05-06

Fix full /var on uccportal, add logrotate

[BOB]

2021-04-30

Fixed radiusd config on samson to suit updated requirements for full hostname and IP

[BOB][TPG][NTU]

2021-04-22

Fixed our long-broken IPSec VPN: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2021-April/005460.html

[333]

2021-04-20

Fixed MariaDB on Mussel, and thus OCSInventory - which depends on it

[333]

2021-04-18

Completely rebuilt the idle-medico VM using only the Ansible playbook that [MTL] wrote

[TPG]

2021-03-08

Fixed gitlab certbot

[MTL]

2021-02-22

Implement age command in uccpass. Migrate uccpass repo to UCC gitlab

[MTL]

2021-02-20

Move octoDNS to dedicated venv for UCC Cloudflare DNS sync

[MTL]

2021-01-20

Poke certs for irc.ucc.asn.au

[MPT]

2021-01-18

Remove mollitz mock internal subnet from murasoi

[MPT][NTU]

2021-01-18

Tinker with mollitz config and update wiki page to match

[TRS][MPT]

2021-01-12

Return mollitz legacy backups, offsite

[TPG]

2021-01-11

molmol freebsd-update

[NTU]

2021-01-10

atlantic cluster magikarp extend ceph osd.3

[BOB]

2021-01-10

atlantic cluster apt updates, reboots

[TPG][NTU]

2020-12-28,30

mailfish:~spamass training

[333][NTU]

2020-12-21

Replace molmol mirror SSD: root:geom-mirror boot swap zfs-slog

[TPG]

2020-12-21

webcam archiving and retrieval optimisation

[NTU]

2020-12-11

Test Ceph WAL on vmstore-bigssd, add discord-irc to legacy backups

[NTU][MTL]

2020-12-10

mailfish: Bounce due to NFS problems

[MPT]

2020-12-02

Fix Abe configuration and rancid

[MPT]

2020-12-02

Troubleshoot UniSFA TV network connection

[MPT]

2020-11-30

Reshuffle UCC wifi channels

[NTU][MTL]

2020-11-29

mussel: fix MoinMoin wiki textcha again

[NTU][CJS][MPT]

2020-10-19

place mollitz (legacy backups) on UCC network

[MPT]

2020-10-19

Create user VM for [DIE] for SS13 server

[MTL][MPT][DAA]

2020-10-12

Make login.ucc.asn.au work

[MPT][333][NTU]

2020-10-09--2020-10-12

CEPH! (Fix latency issues, lots of tweaks to Ceph config...)

[MTL]

2020-10-06

Fix gitlab.ucc.asn.au LE certs

[MTL]

2020-10-06

Fix portal.ucc.asn.au LE certs on uccportal. Clear full /tmp and /var partitions

[MTL]

2020-08-31

Fix webmail.ucc.asn.au redirect. Curse mussel apache config

[TPG]

2020-08-29

Fixed the wheel group page

[MTL]

2020-08-21

Fix wildcard certs on mailfish, mussel and motsugo via ucc-ansible-soe and Cloudflare API key

[MTL]

2020-07-28

Fix mentions of mooneye in mailfish config vis headers and spam

[MTL]

2020-06-08

Fix HTTP redirect to HTTPS for the wiki

[MTL]

2020-06-02

Add LDAP configuration for UCC AD to uccmonitor

[MPT]

2020-05-27

Enable nested KVM virtualisation on mudkip, check enabled on others

[NTU]

2020-05-24

lvextend -r murasoi:/var, add /etc/logrotate.d/ucc-ssh-auth

[BOB]

2020-04-25

dist-upgrade motsugo to buster

[BOB]

2020-04-25

Renewed letsencrypt certs on murasoi to fix ipsec vpn

[BOB]

2020-04-25

Changed main NS record with DNS registrar from mooneye.ucc.gu.uwa.edu.au to mooneye.ucc.asn.au, updated zone records to suit, and changed _acme-challenge.secure.ucc.asn.au zone to use mooneye.ucc.asn.au

[BOB]

2020-04-24

Fix rancid issues introduced by upgrade

[BOB]

2020-04-21

dist-upgrade mussel to buster

[BOB]

2020-04-21

dist-upgrade gitlab to buster

[MTL]

2020-04-21

dist-upgrade meersau to buster; rejoin AD and fix Xrdp logins - see https://github.com/neutrinolabs/xrdp/issues/906

[MPT]

2020-04-18

L3 setup of 4G modem, enabled policy-based routing for murasoi

[MPT]

2020-04-17

Physical and L2 setup of 4G modem in machine room

[MPT]

2020-04-17

Install replacement fans and recommission mudkip

[NTU][MTL]

2020-04-15

Surgical update of mussel's DebianPkg:libapache2-mod-wsgi to 4.5.17-1_i386.deb to fix moinmoin 866180

[MTL]

2020-04-13

Deploy FoldingAtHome to UCC clubroom machines to keep them busy during the COVID19 pandemic

[MTL]

2020-04-11

Migrate UCC wiki from mooneye to mussel

[NTU][TRS]

2020-04-09

Add mollitz (offsite backups) to monitoring

[MTL]

2020-04-06

Set up Mailauesi with haproxy for IMAPS, POP3S and SMTP submission frontend load balancer

[TRS]

2020-03-21

Fixed the VPN by pointing it at the current let's encrypt cert

[TRS]

2020-03-21

Installed BigBlueButton

[MSH]

2020-03-21

Make motsugo logrotate btmp (failed auth attempts) weekly so disk doesn't fill up

[DAS][MPT]

2020-02-26

Troubleshooting Unigames webcams, replugged loose patch cable, moved VLAN1 to new Lard trunk

[MPT][CFE]

2020-02-25

Increase 5G channel width to 40MHz into free space

[MPT][NTU]

2020-02-24

Shutdown Magikarp and diagnose fan issue

[MSH]

2020-02-15

Set mooneye:/etc/opendkim.conf "MaximumHeaders 0" to handle >64KB headers from looping mail to/from(?) [email protected]

[MPT]

2020-02-10

Convert all PVEVMUser permissions to newer UCC_VM_User, update documentation

[BOB]

2020-02-10

Audited user VM's

[BOB]

2020-01-28

Change ocsinventory to ssl only and roll out to all machines

[MPT]

2020-01-27

Resize /dev/md0 and /var on murasoi, with room to grow

[MPT]

2020-01-27

Fix uccpass regression preventing new keys being added

[TEC][NTU][MTL]

2020-01-13

Install Prometheus/Grafana monitoring, with Ansible, on uccmonitor, resize /var, add custom metric

[TEC]

2019-12-24

Install Prometheus/Grafana monitoring, with Ansible, on uccmonitor

[MPT][NTU]

2019-12-{16,17}

Provision new RPi 4 as user server titan, install SOE and useful programs

[MPT]

2019-12-12

Migrate UCC mail to new IronPort cloud appliances with UWA

[MPT][NTU]

2019-12-09

Outbound mail relayhost hostname {antivirus -> smtp}.uwa.edu.au per UniIT request

[NTU][TBB]

2019-12-07

merlo /usr/local/bin/serial-connect fixme resets VTxxx terminal

[MPT]

2019-12-02

Improved network documentation somewhat

[MPT][NTU][MTL]

2019-12-02

Provisioned new "IoT devices" network, including L2, L3 and PSK wifi

[NTU][MPT]

2019-12-02

Created an AWS account for UCC

[MPT][NTU]

2019-11-25

Fix cobra networking (broken small switch), /away automounting

[TEC][333]

2019-11-25

add magikarp and mudkip to Atlantic proxmox cluster

[TEC][MDD]

2019-11-25

update minecraft2017 -> minecraft2019

[333]

2019-11-21

Diagnose and fix smallwing's failing UniFi firmware updates

[MPT][TPG]

2019-11-16

Unborking a user account, restrict usernames to 3 or more characters

[MPT]

2019-11-05

Redirected fresher@ to rest of committee-only@ (successfully this time)

[MSH]

2019-10-22

Set kernel.dmesg_restrict=0 on motsugo and mussel

[MPT][THA][TPG][MSH][NTU]

2019-10-18

Service troubleshooting and restarts, after 0430--0521 power outage: proxmox/samba/ssd/zonemake?

[FVP]

2019-09-20--2019-10-11

username change?

[MTL]

2019-08-22

dpkg --add-architecture i386; apt update; apt install libpcre2-dev:i386 libncurses5-dev:i386 on motsugo for [DAA]. Also, free up space on full /var

[MPT]

2019-08-14

Implemented soft-blacklist for possibly-broken account names in ucc-adduser-ad

[FVP]

2019-08-13

Replaced faulty Microtik router sharpchin with spare OpenWRT device coromandel (in UniSFA)

[MPT]

2019-08-09

Changed ucc-adduser-ad to make new users' index.html readable by the webserver

[MPT]

2019-07-31

Updated MOTD to reflect [DBA]'s resignation from committee

[MPT]

2019-07-29

Removed ex-group members from their old groups on UCC systems (following group review)

[DAA]

2019-07-29

Tried and failed to upgrade Guacamole to 1.0.0 (using Postgres + RADIUS and then Postgres + LDAP), rolled back to 0.9.14

[333][NTU][TPG]

2019-07-22

Wifi - fix smallwing/nanoHD VLAN connectivity to kerosene, was curviceps

[MPT]

2019-07-20

Enabled daily updates of Linux desktops

[MPT]

2019-07-20

Created 90-ucc-desktops.conf for PolKit and deployed to all desktops, enabled unattended upgrades on all desktops

[MPT]

2019-07-18

Joined Catfish's Linux install to the domain, fixed network logins

[TPG][NTU]

2019-07-15

Switch complications?

[TRS][NTU]

2019-07-14

UWA DNS outage, check bind+dd/discord-irc, disable/enable mooneye DNS forwarders

[FVP][MPT][DAS]

2019-07-12

Replaced machine room switch with new chassis and supervisor with 10GE capability

[DAA]

2019-07-11

Enlarged /backups array, partition, logical volume and filesystem on Mollitz

[DAA]

2019-07-09

Installed FreeBSD updates to 11.2-RELEASE-p11 on Molmol & upgraded pkg set

[MPT]

2019-06-28

Reinstalled clownfish with updated Mint 19.1 on the new NVMe SSD

[BOB]

2019-06-25

Changed door mailing list settings to discard spam emails

[BOB][MPT][FVP][AJT][TRS]

2019-06-21--2019-06-22

Recovered murasoi from dual system disk failure

[MSH]

2019-06-18

Filtered SACK on murasoi ucc-fw to avoid SACK Panic

[MPT]

2019-06-10

Fixed /etc/fstab on pinball to boot new NVMe SSD

[MPT]

2019-06-10

Swapped in new fans to pinball and clownfish (with [AJT])

[MPT]

2019-06-09

Fixed network issues on upgraded clownfish

[AJT]

2019-06-07

Upgraded pinball, clownfish and porcupine with new components (with [MPT])

[DAA]

2019-05-27

Upgraded Samba on Samson from 4.9.2 to 4.9.5

[MTL]

2019-05-20

Some housekeeping done on mooneye /etc/aliases. All mailman mailing lists moved to bottom of file in prep for later mailman upgrade

[DAA]

2019-05-18

Removed a stack of obsolete Python packages that were breaking things

[DAA]

2019-05-18

Gitlab maintenance - fixed UCC user accounts to be internal, not external, and removed heaps of spam accounts

[DAA]

2019-05-15

Converted Apache on Mussel to Certbot and wildcard SSL; removed the acme-members scripts from crontab and zonemake; converted Dovecot on Motsugo to Certbot and wildcard SSL; converted IPsec on Murasoi to use DNS challenges; dropped acmetool on Mussel, Mooneye, Motsugo, Murasoi and Gitlab; wrote SSLCertificates page

[DAA]

2019-05-15

Security patches installed for Molmol, Meersau and Maaxen

[DAA]

2019-05-13

Converted inspircd on Mussel to Certbot

[DAA]

2019-05-13

Converted postfix and apache on Mooneye to Certbot and wildcard SSL

[DAA]

2019-05-12

Set up infrastructure for wildcard SSL certificates

[BOB][DBA]

2019-05-03

Installed replacement system/slog disk SSD in molmol

[DAA]

2019-05-03

Removed a large number of obsolete packages (mostly libraries and old development tools) from Mussel

[DAA]

2019-05-02

Upgraded Molmol to FreeBSD 11.2-RELEASE
Upgraded Samba to 4.8 (supported version) on molmol

[FVP]

2019-05-01

Refactored uccpass to put user keys in separate files, with committee and wheel directories

[MPT]

2019-04-30

Removed elliotnunn from local sudo /etc/group on clownfish to stop Polkit from asking everyone for his password

[MPT]

2019-04-30

Added [MLG] to committee user group, and updated motd

[DAA]

2019-04-29

Upgraded to Debian 9.9 on Mussel, Motsugo, Murasoi, Mooneye, Merlo, Samson, Mollitz, Gitlab, idle-Maltair, idle-Medico, Heathred, Medico, Maltair and Loveday; ceph restarted and checked for status

[BOB]

2019-04-27

Updated salmon and installed unattended upgrades. Sorted through VM cluster and tagged a heap of VM's for deletion

[BOB]

2019-04-26

Specced and purchased replacement+upgrade disks for molmol

[MSH]

2019-04-23

Made DNS notifies/AXFR from secondaries work

[MSH]

2019-04-23

Fixed member web subdomains and ssl

[DAA]

2019-04-20

Removed a bunch of obsolete packages from motsugo (gcc-4.6 ruby1.8 ruby1.9.1 php5-cli postgresql-client-9.4 python2.6-minimal zlib-bin libjasper1 hardening-includes python-reportbug python-support sysvinit)
upgraded the backport of lepton
upgraded Docker to a supported version

[BEN]

2019-04-18

Setup unisfa-freshers mailing list

[NTU][TEC]

2019-04-15

ocsinventory-agent deployed to corvo, napoli

[BOB]

2019-04-13

Set up ocsinventory.ucc.asn.au and deployed to most machines

[DAA]

2019-04-12

Upgraded idle-{medico,maltair} to Debian 9; installed Docker and Gitlab Runner on idle-medico

[DAA]

2019-04-10

Installed rust and cargo from testing on Motsugo

[SJH][MSH]

2019-04-09

Fixed user php7

[DAA]

2019-04-09

Updated Roundcube to the version in Debian

[BOB]

2019-04-09

Disabled php5 and enabled php7 on mussel

[DAA]

2019-04-09

Attempted to fix eggman mounting /mp3s on boot by enabling systemd-networkd-wait-online.service

[DAA]

2019-04-09

Removed motsugo pip packages for Pygments, Six, and youtube-dl as they were old and newer Debian packages were installed

[MTL]

2019-04-03

mussel package upgrades

[MSH]

2019-04-02

Removed and readded motsugo sdc and sde. Got rid of current-pending-sectors by overwriting them

[NTU] [MPT]

2019-04-01

Merge merlo:/var/log/dispense into ~coke/cokelog , restart rsyslogd on merlo, motsugo

[MPT]

2019-03-27

Remounted /home and /away on merlo to fix BAD PIN message

[BOB]

2019-03-26

Re-synced time on Loveday - clock drift was causing ceph to be unhealthy

[BOB]

2019-03-26

Installed LSI SAS card in new HP DL380 (maltair), updated assorted firmwares, fixed network interfaces and re-added to VM cluster

[DTA]

2019-03-26

Set passwords for flame users id, dos, linux - it might stop the problem with aserver.err filling up after a portscan

[MSH]

2019-03-23

put semrushbot in secure.ucc, wiki.ucc robots.txt (not sure if wiki worked

[NTU] [MPT]

2019-03-18

reset napoli passwd, add sudoers

[MSH]

2019-03-18

Got arpwatch running again on murasoi, had to put br0.* in /etc/arpwatch.conf

[NTU] [TPG]

2019-03-16

merlo not remote logging and BAD PIN? reboot, remount home, restart rsyslogd on merlo, motsugo

[NTU] [FVP] [JGM]

2019-03-16

Server rack in corridor, temporary? switch to VT320 with a narrower keyboard until a narrow PS/2 keyboard is found for the VT510 https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2019-March/005091.html

[MSH]

2019-03-16

Created mooneye:/var/log/journal so logs will persist across reboots

[RME]

2019-03-15

Created this page. Let's see if it sticks.

[BOB]

2019-03-14

Removed root keys of wheel members who are not current members

[BOB]

2019-03-05

Attended Ross's auctions to inspect HP DL380 Gen 8 server for maltair replacement. Bid and won in auction. Delivered to UCC and installed

[TRS]

2019-02-05

Pointed SOGo to AD (was still LDAP.)

[TRS]

2018-12-01

Upgraded SOGo on mussel to 4.0.4.

[FVP] [CFE]

2018-11 to 2019-02

Created portal.ucc.asn.au based on [DAA]'s MemberDB for membership information