Add fuuko (server for most production services)

simon commented

2021-01-04 12:22:54 +01:00

Owner

This replaces my current server.

Hardware

HP MicroServer Gen8
- 8 GiB ECC memory (4 GiB HP, 4 GiB Kingston)
- Xeon E3-1220L v2 (2(4)×2.3 GHz, 3.5 GHz Boost, 17(!) W TDP, hardware AES 🥳)
Get hard drives
- I got two 8 TB drives (Seagate Exos E 7E8 ST8000NM000A, WD Ultrastar DC HC320 0B36404)

Base system

SSD for system
- Attached to the first SATA port on the front, since HP decided to make the intenal ODD port
  - SATA2 only (which limits the SSD to half its speed)
  - not bootable (so I would need to install GRUB on a sd card)
- Crucial BX500 240GB
- ext4 on luks, randomly encrypted swap
RAID1 for data (btrfs on luks; using raw partition without partition table)
- Luks unlock key on system partition
- Compression
- How to unlock two encrypted partitions with key file in nixos without mounting each one indivitually? /etc/crypttab is your friend

services.btrfs.autoScrub.enable = true;
Remote unlock (#9)
systemd-networkd for networking

Services

No longer managed

i. e. to remove before beginnint with migration

Influxdb (just uses up resources)
Minio media gateway (I do not actually use it)
Docker hub prometheus exporter
Shiori
mangareader
namensschilder (if someone wants it to work, they should fix it)
Minio
Docker registry and registry ldap auth
Tor reverse proxy (I doubt anyone uses it and it does not offer any advantage)
LDAP (too complex, too fragile, too old IMHO and for my situation)
- Still used by
  - Hedgedoc
  - File listings (scan, media)
  - Matrix/Synapse
  - Docker registry
psitransfer
microsocks
opentracker
firefox sync (NixOS module is broken and going to be discontinued since upstream doesn’t upgrade to python 3)

DNS/DHCP (dnsmasq)

Has NixOS module
Test DHCP

Ankisyncd

Has nixos module

Hedgedoc/CodiMD

Has nixos module
Drop LDAP auth mechanism
Migrate database postgres → postgres

Gitea

Has nixos module
Test
Problem: I have been using the dev version for two years without noticing, so it will be a downgrade.
Migrate to postrgresql (see https://github.com/go-gitea/gitea/issues/5651#issuecomment-604047535)
- docker-compose exec -u git gitea gitea dump --database postgres --type tar.gz -f /data/dump.tar
- copy to target
- tar xvf dump.tar.gz
- chown -R gitea:gitea *
- rm -rf /var/lib/gitea/repositories /var/lib/gitea/data/avatars /data/gitea/lfs
- mv repos /var/lib/gitea/repositories
- mv data/avatars /var/lib/gitea/avatars
- mv data/lfs /data/gitea/lfs
- sed -e 's/BOOL DEFAULT false/BOOL DEFAULT false/g' -e 's/BOOL DEFAULT true/BOOL DEFAULT true/g' -i gitea-db.sql
- sudo -u gitea psql -f gitea-db.sql
- apply sql from https://github.com/go-gitea/gitea/issues/4407#issuecomment-469031232
- allow db downgrade UPDATE version SET version=156 WHERE id=1; dangerous
- all above commands should not throw any errors
- start gitea
- profit

Drone CI

Until I evaluate usage of hydra and (maybe) adopt it, I want to keep it
Exec runner in container or as sandboxed systemd service (should be able to use nix daemon as unprivileged user)
Enterprise edition, since OSS edition does not support postgres and out-of-band secrets management
Create NixOS module for server (see https://github.com/Mic92/dotfiles/blob/master/nixos/eve/modules/drone/server.nix and https://github.com/NixOS/nixpkgs/pull/73971)
Create NixOS module for exec runner (see https://github.com/Mic92/dotfiles/blob/master/nixos/eve/modules/drone/exec-runner.nix)

Prometheus + Alertmanager

Have nixos module
Store data on HDD (high compression efficiency on btrfs, many writes wears ssd)
Send alerts to go-neb

Exporters

FritzBox exporter: Has nixos module (probably needs manual update)
Drone (part of drone, see drone)
Matrix (part of synapse)
Aria2 (custom)
- Package
- Create NixOS module

Grafana

Has nixos module
Declarative prometheus datasource

Mosquitto + Wordclock script

Create package for script (added in cce86ac2c9)
Create module for script (added in cce86ac2c9)
Mosquitto has nixos module
test
figure out why systemd service does not change color, manually invoking however does (→ it doesn’t automatically reconnect after timing out)

HTTP file listings

Build common module for it (done in e0ef586e5e)
media
- htpasswd on filesystem (is state)

FTP (for scanner)

vsftpd has nixos module
Homemade conversion script (tiff → png)
- Create module (~~using systemd path unit~~ systemd path units do not support this use case)

Matrix

Postgres database (quite a bit traffic when joining a room and due to federation)
How to handle LDAP → no ldap transition?
- Add bcrypt hashes in password_hash column in users table
Synapse
coturn (added on vueko because of NAT in b6297d0153)
mautrix-whatsapp (adapt from https://github.com/NixOS/nixpkgs/pull/59211)
Go-neb for alertmanager

Torrent

Custom module, since my configuration is quite complex
Package ariang (packaged in fork (https://git.sbruder.de/simon/AriaNg), since packaging otherwise is not trivial)
Systemd service
htpasswd on filesystem (is state) for file listing
Make aria2 exporter available to prometheus declaratively

Misc

Postgresql backups (3a5568a136)

Migration order

Base system (in git)
Prometheus (dns)
Grafana (dns)
Scan (dns, scanner settings, remove from media-proxy)
Media (dns, media-proxy)
Torrent (data, saved session)
dnsmasq
Mosquitto (port forwarding in router)
ankisyncd
codimd
gitea
drone
matrix

This replaces my current server. ## Hardware - HP MicroServer Gen8 - 8 GiB ECC memory (4 GiB HP, 4 GiB Kingston) - [X] [Xeon E3-1220L v2](https://ark.intel.com/content/www/us/en/ark/products/65735/intel-xeon-processor-e3-1220l-v2-3m-cache-2-30-ghz.html) (2(4)×2.3 GHz, 3.5 GHz Boost, 17(!) W TDP, **hardware AES** 🥳) - [X] Get hard drives - I got two 8 TB drives (Seagate Exos E 7E8 ST8000NM000A, WD Ultrastar DC HC320 0B36404) ## Base system * SSD for system + Attached to the first SATA port on the front, since HP decided to make the intenal ODD port - SATA2 only (which limits the SSD to half its speed) - not bootable (so I would need to install GRUB on a sd card) + Crucial BX500 240GB + ext4 on luks, randomly encrypted swap * RAID1 for data (btrfs on luks; using raw partition without partition table) + Luks unlock key on system partition + Compression - [X] How to unlock two encrypted partitions with key file in nixos without mounting each one indivitually? `/etc/crypttab` is your friend - [X] `services.btrfs.autoScrub.enable = true;` - [X] Remote unlock (#9) - [X] systemd-networkd for networking ## Services ### No longer managed i. e. to remove before beginnint with migration - [X] Influxdb (just uses up resources) - [X] Minio media gateway (I do not actually use it) - [X] Docker hub prometheus exporter - [X] Shiori - [X] mangareader - [X] namensschilder (if someone wants it to work, they should fix it) - [X] Minio - [X] Docker registry and registry ldap auth - [X] Tor reverse proxy (I doubt anyone uses it and it does not offer any advantage) - [X] LDAP (too complex, too fragile, too old IMHO and for my situation) - Still used by - [X] Hedgedoc - [X] File listings (scan, media) - [X] Matrix/Synapse - [X] Docker registry - [X] psitransfer - [X] microsocks - [X] opentracker - [X] firefox sync (NixOS module is broken and going to be discontinued since upstream doesn’t upgrade to python 3) ### DNS/DHCP (dnsmasq) - [X] Has NixOS module - [X] Test DHCP ### Ankisyncd - [X] Has nixos module ### Hedgedoc/CodiMD - [X] Has nixos module - [X] Drop LDAP auth mechanism - [X] Migrate database postgres → postgres ### Gitea - [X] Has nixos module - [X] Test - Problem: I have been using the dev version for two years without noticing, so it will be a downgrade. - [X] Migrate to postrgresql (see https://github.com/go-gitea/gitea/issues/5651#issuecomment-604047535) - `docker-compose exec -u git gitea gitea dump --database postgres --type tar.gz -f /data/dump.tar` - copy to target - `tar xvf dump.tar.gz` - `chown -R gitea:gitea *` - `rm -rf /var/lib/gitea/repositories /var/lib/gitea/data/avatars /data/gitea/lfs` - `mv repos /var/lib/gitea/repositories` - `mv data/avatars /var/lib/gitea/avatars` - `mv data/lfs /data/gitea/lfs` - `sed -e 's/BOOL DEFAULT false/BOOL DEFAULT false/g' -e 's/BOOL DEFAULT true/BOOL DEFAULT true/g' -i gitea-db.sql` - `sudo -u gitea psql -f gitea-db.sql` - apply sql from https://github.com/go-gitea/gitea/issues/4407#issuecomment-469031232 - allow db downgrade `UPDATE version SET version=156 WHERE id=1;` **dangerous** - all above commands should not throw any errors - start gitea - profit ### Drone CI - Until I evaluate usage of hydra and (maybe) adopt it, I want to keep it - Exec runner in container or as sandboxed systemd service (should be able to use nix daemon as unprivileged user) - *Enterprise* edition, since OSS edition does not support postgres and out-of-band secrets management - [X] Create NixOS module for server (see https://github.com/Mic92/dotfiles/blob/master/nixos/eve/modules/drone/server.nix and https://github.com/NixOS/nixpkgs/pull/73971) - [X] Create NixOS module for exec runner (see https://github.com/Mic92/dotfiles/blob/master/nixos/eve/modules/drone/exec-runner.nix) ### Prometheus + Alertmanager - [X] Have nixos module - [X] Store data on HDD (high compression efficiency on btrfs, many writes wears ssd) - [X] Send alerts to go-neb #### Exporters - [X] FritzBox exporter: Has nixos module (probably needs manual update) - [X] Drone (part of drone, see drone) - [X] Matrix (part of synapse) - [X] Aria2 (custom) - [X] Package - [X] Create NixOS module ### Grafana - [X] Has nixos module - [X] Declarative prometheus datasource ### Mosquitto + Wordclock script - [X] Create package for script (added in cce86ac2c92a74bc7aa37001945cf19b3824a859) - [X] Create module for script (added in cce86ac2c92a74bc7aa37001945cf19b3824a859) - [X] Mosquitto has nixos module - [X] test - [X] figure out why systemd service does not change color, manually invoking however does (→ it doesn’t automatically reconnect after timing out) ### HTTP file listings - [X] Build common module for it (done in e0ef586e5eaf2f562d6c862a5595297cab629bca) - [X] media - [X] htpasswd on filesystem (is state) ### FTP (for scanner) - [X] vsftpd has nixos module - [X] Homemade conversion script (tiff → png) - [X] Create module (~~using systemd path unit~~ systemd path units do not support this use case) ### Matrix - Postgres database (quite a bit traffic when joining a room and due to federation) - [X] How to handle LDAP → no ldap transition? - [X] Add bcrypt hashes in `password_hash` column in `users` table - [X] Synapse - [X] coturn (added on vueko because of NAT in b6297d0153a6791d33c6ad06a705e9f8ac6ade3b) - [X] mautrix-whatsapp (adapt from https://github.com/NixOS/nixpkgs/pull/59211) - [X] Go-neb for alertmanager ### Torrent - [X] Custom module, since my configuration is quite complex - [X] Package ariang (packaged in fork (https://git.sbruder.de/simon/AriaNg), since packaging otherwise is not trivial) - [X] Systemd service - [X] htpasswd on filesystem (is state) for file listing - [X] Make aria2 exporter available to prometheus declaratively ### Misc - [X] Postgresql backups (3a5568a136acd2dc0942ae2e274a58b0115b8bcb) ## Migration order - [X] Base system (in git) - [X] Prometheus (dns) - [X] Grafana (dns) - [X] Scan (dns, scanner settings, remove from media-proxy) - [X] Media (dns, media-proxy) - [X] Torrent (data, saved session) - [X] dnsmasq - [X] Mosquitto (port forwarding in router) - [X] ankisyncd - [X] codimd - [X] gitea - [X] drone - [X] matrix