simon
/
nixos-config
1
1
Fork 0
Code Issues 8 Pull Requests Projects Releases Activity

Add fuuko (server for most production services) #18

New Issue
Closed
opened 2021-01-04 12:22:54 +01:00 by simon · 1 comment
simon commented 2021-01-04 12:22:54 +01:00
Owner

This replaces my current server.

Hardware

  • HP MicroServer Gen8
    • 8 GiB ECC memory (4 GiB HP, 4 GiB Kingston)
    • Xeon E3-1220L v2 (2(4)×2.3 GHz, 3.5 GHz Boost, 17(!) W TDP, hardware AES 🥳)
  • Get hard drives
    • I got two 8 TB drives (Seagate Exos E 7E8 ST8000NM000A, WD Ultrastar DC HC320 0B36404)

Base system

  • SSD for system
    • Attached to the first SATA port on the front, since HP decided to make the intenal ODD port
      • SATA2 only (which limits the SSD to half its speed)
      • not bootable (so I would need to install GRUB on a sd card)
    • Crucial BX500 240GB
    • ext4 on luks, randomly encrypted swap
  • RAID1 for data (btrfs on luks; using raw partition without partition table)
    • Luks unlock key on system partition
    • Compression
    • How to unlock two encrypted partitions with key file in nixos without mounting each one indivitually? /etc/crypttab is your friend
  • services.btrfs.autoScrub.enable = true;
  • Remote unlock (#9)
  • systemd-networkd for networking

Services

No longer managed

i. e. to remove before beginnint with migration

  • Influxdb (just uses up resources)
  • Minio media gateway (I do not actually use it)
  • Docker hub prometheus exporter
  • Shiori
  • mangareader
  • namensschilder (if someone wants it to work, they should fix it)
  • Minio
  • Docker registry and registry ldap auth
  • Tor reverse proxy (I doubt anyone uses it and it does not offer any advantage)
  • LDAP (too complex, too fragile, too old IMHO and for my situation)
    • Still used by
      • Hedgedoc
      • File listings (scan, media)
      • Matrix/Synapse
      • Docker registry
  • psitransfer
  • microsocks
  • opentracker
  • firefox sync (NixOS module is broken and going to be discontinued since upstream doesn’t upgrade to python 3)

DNS/DHCP (dnsmasq)

  • Has NixOS module
  • Test DHCP

Ankisyncd

  • Has nixos module

Hedgedoc/CodiMD

  • Has nixos module
  • Drop LDAP auth mechanism
  • Migrate database postgres → postgres

Gitea

  • Has nixos module
  • Test
  • Problem: I have been using the dev version for two years without noticing, so it will be a downgrade.
  • Migrate to postrgresql (see https://github.com/go-gitea/gitea/issues/5651#issuecomment-604047535)
    • docker-compose exec -u git gitea gitea dump --database postgres --type tar.gz -f /data/dump.tar
    • copy to target
    • tar xvf dump.tar.gz
    • chown -R gitea:gitea *
    • rm -rf /var/lib/gitea/repositories /var/lib/gitea/data/avatars /data/gitea/lfs
    • mv repos /var/lib/gitea/repositories
    • mv data/avatars /var/lib/gitea/avatars
    • mv data/lfs /data/gitea/lfs
    • sed -e 's/BOOL DEFAULT false/BOOL DEFAULT false/g' -e 's/BOOL DEFAULT true/BOOL DEFAULT true/g' -i gitea-db.sql
    • sudo -u gitea psql -f gitea-db.sql
    • apply sql from https://github.com/go-gitea/gitea/issues/4407#issuecomment-469031232
    • allow db downgrade UPDATE version SET version=156 WHERE id=1; dangerous
    • all above commands should not throw any errors
    • start gitea
    • profit

Drone CI

Prometheus + Alertmanager

  • Have nixos module
  • Store data on HDD (high compression efficiency on btrfs, many writes wears ssd)
  • Send alerts to go-neb

Exporters

  • FritzBox exporter: Has nixos module (probably needs manual update)
  • Drone (part of drone, see drone)
  • Matrix (part of synapse)
  • Aria2 (custom)
    • Package
    • Create NixOS module

Grafana

  • Has nixos module
  • Declarative prometheus datasource

Mosquitto + Wordclock script

  • Create package for script (added in cce86ac2c9)
  • Create module for script (added in cce86ac2c9)
  • Mosquitto has nixos module
  • test
  • figure out why systemd service does not change color, manually invoking however does (→ it doesn’t automatically reconnect after timing out)

HTTP file listings

  • Build common module for it (done in e0ef586e5e)
  • media
    • htpasswd on filesystem (is state)

FTP (for scanner)

  • vsftpd has nixos module
  • Homemade conversion script (tiff → png)
    • Create module (using systemd path unit systemd path units do not support this use case)

Matrix

  • Postgres database (quite a bit traffic when joining a room and due to federation)
  • How to handle LDAP → no ldap transition?
    • Add bcrypt hashes in password_hash column in users table
  • Synapse
  • coturn (added on vueko because of NAT in b6297d0153)
  • mautrix-whatsapp (adapt from https://github.com/NixOS/nixpkgs/pull/59211)
  • Go-neb for alertmanager

Torrent

  • Custom module, since my configuration is quite complex
  • Package ariang (packaged in fork (https://git.sbruder.de/simon/AriaNg), since packaging otherwise is not trivial)
  • Systemd service
  • htpasswd on filesystem (is state) for file listing
  • Make aria2 exporter available to prometheus declaratively

Misc

Migration order

  • Base system (in git)
  • Prometheus (dns)
  • Grafana (dns)
  • Scan (dns, scanner settings, remove from media-proxy)
  • Media (dns, media-proxy)
  • Torrent (data, saved session)
  • dnsmasq
  • Mosquitto (port forwarding in router)
  • ankisyncd
  • codimd
  • gitea
  • drone
  • matrix
This replaces my current server. ## Hardware - HP MicroServer Gen8 - 8 GiB ECC memory (4 GiB HP, 4 GiB Kingston) - [X] [Xeon E3-1220L v2](https://ark.intel.com/content/www/us/en/ark/products/65735/intel-xeon-processor-e3-1220l-v2-3m-cache-2-30-ghz.html) (2(4)×2.3 GHz, 3.5 GHz Boost, 17(!) W TDP, **hardware AES** 🥳) - [X] Get hard drives - I got two 8 TB drives (Seagate Exos E 7E8 ST8000NM000A, WD Ultrastar DC HC320 0B36404) ## Base system * SSD for system + Attached to the first SATA port on the front, since HP decided to make the intenal ODD port - SATA2 only (which limits the SSD to half its speed) - not bootable (so I would need to install GRUB on a sd card) + Crucial BX500 240GB + ext4 on luks, randomly encrypted swap * RAID1 for data (btrfs on luks; using raw partition without partition table) + Luks unlock key on system partition + Compression - [X] How to unlock two encrypted partitions with key file in nixos without mounting each one indivitually? `/etc/crypttab` is your friend - [X] `services.btrfs.autoScrub.enable = true;` - [X] Remote unlock (#9) - [X] systemd-networkd for networking ## Services ### No longer managed i. e. to remove before beginnint with migration - [X] Influxdb (just uses up resources) - [X] Minio media gateway (I do not actually use it) - [X] Docker hub prometheus exporter - [X] Shiori - [X] mangareader - [X] namensschilder (if someone wants it to work, they should fix it) - [X] Minio - [X] Docker registry and registry ldap auth - [X] Tor reverse proxy (I doubt anyone uses it and it does not offer any advantage) - [X] LDAP (too complex, too fragile, too old IMHO and for my situation) - Still used by - [X] Hedgedoc - [X] File listings (scan, media) - [X] Matrix/Synapse - [X] Docker registry - [X] psitransfer - [X] microsocks - [X] opentracker - [X] firefox sync (NixOS module is broken and going to be discontinued since upstream doesn’t upgrade to python 3) ### DNS/DHCP (dnsmasq) - [X] Has NixOS module - [X] Test DHCP ### Ankisyncd - [X] Has nixos module ### Hedgedoc/CodiMD - [X] Has nixos module - [X] Drop LDAP auth mechanism - [X] Migrate database postgres → postgres ### Gitea - [X] Has nixos module - [X] Test - Problem: I have been using the dev version for two years without noticing, so it will be a downgrade. - [X] Migrate to postrgresql (see https://github.com/go-gitea/gitea/issues/5651#issuecomment-604047535) - `docker-compose exec -u git gitea gitea dump --database postgres --type tar.gz -f /data/dump.tar` - copy to target - `tar xvf dump.tar.gz` - `chown -R gitea:gitea *` - `rm -rf /var/lib/gitea/repositories /var/lib/gitea/data/avatars /data/gitea/lfs` - `mv repos /var/lib/gitea/repositories` - `mv data/avatars /var/lib/gitea/avatars` - `mv data/lfs /data/gitea/lfs` - `sed -e 's/BOOL DEFAULT false/BOOL DEFAULT false/g' -e 's/BOOL DEFAULT true/BOOL DEFAULT true/g' -i gitea-db.sql` - `sudo -u gitea psql -f gitea-db.sql` - apply sql from https://github.com/go-gitea/gitea/issues/4407#issuecomment-469031232 - allow db downgrade `UPDATE version SET version=156 WHERE id=1;` **dangerous** - all above commands should not throw any errors - start gitea - profit ### Drone CI - Until I evaluate usage of hydra and (maybe) adopt it, I want to keep it - Exec runner in container or as sandboxed systemd service (should be able to use nix daemon as unprivileged user) - *Enterprise* edition, since OSS edition does not support postgres and out-of-band secrets management - [X] Create NixOS module for server (see https://github.com/Mic92/dotfiles/blob/master/nixos/eve/modules/drone/server.nix and https://github.com/NixOS/nixpkgs/pull/73971) - [X] Create NixOS module for exec runner (see https://github.com/Mic92/dotfiles/blob/master/nixos/eve/modules/drone/exec-runner.nix) ### Prometheus + Alertmanager - [X] Have nixos module - [X] Store data on HDD (high compression efficiency on btrfs, many writes wears ssd) - [X] Send alerts to go-neb #### Exporters - [X] FritzBox exporter: Has nixos module (probably needs manual update) - [X] Drone (part of drone, see drone) - [X] Matrix (part of synapse) - [X] Aria2 (custom) - [X] Package - [X] Create NixOS module ### Grafana - [X] Has nixos module - [X] Declarative prometheus datasource ### Mosquitto + Wordclock script - [X] Create package for script (added in cce86ac2c92a74bc7aa37001945cf19b3824a859) - [X] Create module for script (added in cce86ac2c92a74bc7aa37001945cf19b3824a859) - [X] Mosquitto has nixos module - [X] test - [X] figure out why systemd service does not change color, manually invoking however does (→ it doesn’t automatically reconnect after timing out) ### HTTP file listings - [X] Build common module for it (done in e0ef586e5eaf2f562d6c862a5595297cab629bca) - [X] media - [X] htpasswd on filesystem (is state) ### FTP (for scanner) - [X] vsftpd has nixos module - [X] Homemade conversion script (tiff → png) - [X] Create module (~~using systemd path unit~~ systemd path units do not support this use case) ### Matrix - Postgres database (quite a bit traffic when joining a room and due to federation) - [X] How to handle LDAP → no ldap transition? - [X] Add bcrypt hashes in `password_hash` column in `users` table - [X] Synapse - [X] coturn (added on vueko because of NAT in b6297d0153a6791d33c6ad06a705e9f8ac6ade3b) - [X] mautrix-whatsapp (adapt from https://github.com/NixOS/nixpkgs/pull/59211) - [X] Go-neb for alertmanager ### Torrent - [X] Custom module, since my configuration is quite complex - [X] Package ariang (packaged in fork (https://git.sbruder.de/simon/AriaNg), since packaging otherwise is not trivial) - [X] Systemd service - [X] htpasswd on filesystem (is state) for file listing - [X] Make aria2 exporter available to prometheus declaratively ### Misc - [X] Postgresql backups (3a5568a136acd2dc0942ae2e274a58b0115b8bcb) ## Migration order - [X] Base system (in git) - [X] Prometheus (dns) - [X] Grafana (dns) - [X] Scan (dns, scanner settings, remove from media-proxy) - [X] Media (dns, media-proxy) - [X] Torrent (data, saved session) - [X] dnsmasq - [X] Mosquitto (port forwarding in router) - [X] ankisyncd - [X] codimd - [X] gitea - [X] drone - [X] matrix
simon added a new dependency 2021-01-04 12:23:11 +01:00
#9 SSH in initrd/remote LUKS unlock
simon added the
type
new machine
 label 2021-01-31 22:21:20 +01:00
simon added a new dependency 2021-02-11 14:13:03 +01:00
#15 Deprecate omnipresent docker
simon removed a dependency 2021-03-13 10:52:57 +01:00
#15 Deprecate omnipresent docker
simon added a new dependency 2021-03-27 16:30:55 +01:00
#49 fuuko: Add media backup
simon commented 2021-04-03 18:51:45 +02:00
Poster
Owner

all done 🥳

all done 🥳
simon closed this issue 2021-04-03 18:51:45 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
okarin2
hyper
catering
reuse
yuzuru2
renge2
23.11
nazuna
neomutt
sayuri-mesa-clover
upower
binfmt
ayu
restic-rest-server
Labels
Clear labels   
affects/hardware
   
affects/legal
   
affects/reproducibility
   
affects/security
   
affects/style
   
affects/usability
   
blocked by/release 21.05
   
blocked by/release 21.11
   
blocked by/release 22.05
   
blocked by/testing needed
   
blocked by/testing needed/fuuko
   
blocked by/testing needed/sayuri
   
blocked by/upstream
  
resolution
deferred
   
resolution
permanent workaround
   
resolution
upstream wontfix
   
resolution
wontfix
  
type
bug
   
type
chore
   
type
feature
   
type
new machine
   
type
question
   
type
regression
   
type
tracking
No Label
affects/hardware
affects/legal
affects/reproducibility
affects/security
affects/style
affects/usability
blocked by/release 21.05
blocked by/release 21.11
blocked by/release 22.05
blocked by/testing needed
blocked by/testing needed/fuuko
blocked by/testing needed/sayuri
blocked by/upstream
resolution
deferred
resolution
permanent workaround
resolution
upstream wontfix
resolution
wontfix
type
bug
type
chore
type
feature
type
new machine
type
question
type
regression
type
tracking
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Blocks
#49 fuuko: Add media backup
simon/nixos-config
Depends on
#9 SSH in initrd/remote LUKS unlock
simon/nixos-config
Reference: simon/nixos-config#18
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?