Tangled Upgrade

The initial cutover uses tested v1.20.0 packages. flake.lock still selects commit 2096d06d938e680297adeadd71b8320c9f3491d4; the input URL tracks upstream again so subsequent weekly updates can advance Tangled. Upstream's own nixpkgs is retained to reuse the built release binaries.

Run The Script

Pull the latest configuration on asusmini, then run:

sudo bash /etc/commonscomputing-nix/scripts/upgrade-knot.sh

The script rehearses first, prints the migration report, and asks you to type MIGRATE before stopping Go knot. Keep another administrative SSH session open, back up the master key off-host, and ask users to pause writes during cutover. --yes skips the approval prompt. Running the script authorizes migration and deployment, not merely a smoke test.

Root's Git identity must be able to fetch and push to origin. The script checks ls-remote and push --dry-run before downtime. Origin is the separate Gitea service on port 2222, not this knot.

If encrypted-secret edits prevent a fast-forward pull, preserve only those edits first; leave unrelated DNS changes alone:

sudo git -C /etc/commonscomputing-nix stash push \
  -m 'pre-cutover encrypted secret backup' -- secrets/knot-env.age secrets/secrets.nix
sudo git -C /etc/commonscomputing-nix pull --ff-only

The encrypted secret is already committed; do not pop an identical stash over it. Its SHA-256 is b26f36d098a337a98217aa4e73d2f39eb4e8864337dfcface937093f5557f0ac.

Happy Path

Lock out concurrent updates and validate the host, staging files, and Git access.
Import the prebuilt system/tools if missing, after checking the archive checksum.
Decrypt the persistent key into root-only runtime storage and rehearse.
Obtain downtime approval; stop the timer and Go knot.
Back up the database, repositories, SSH identity, and old NixOS generation.
Copy-migrate repositories/access grants while preserving all original source data.
Validate the imported identity/configuration and install the prebuilt system.
Set ownership, check agenix supplied the same key, and start Rust knot.
Check local/public APIs, service state, SSH fingerprint, and Caddy WebSocket upgrade.
Mark verification complete, start the weekly timer, and request an indexer crawl.

Source data stays in /home/git; Rust state lives in /var/lib/knot. The previous rehearsal found 23 repositories, 16 members, 12 collaborator grants, zero skips, and no ACL/table disagreement. Check the fresh report for changes.

Backups are root-only under /var/backups/knot-cutover.*. They contain private key material, including repo-signing-keys.json; back them up encrypted off-host. The master key is not printed and is scoped to migration/config validation. The runtime plaintext file is removed on ordinary success/failure and signals; SIGKILL may leave it root-readable in /run until cleanup or reboot.

Networking

Administrative SSH stays on 22. Rust knot Git SSH uses 2223, opened by the NixOS module. Use 100.122.40.27:2223 over Tailscale, or arrange external TCP 2223 forwarding to 192.168.1.249:2223. Existing port-22 Git URLs need an explicit port or client SSH configuration. Public 2222 belongs to Gitea and must not be repurposed. The script does not change router forwarding or sshd.

Caddy proxies all paths and WebSocket upgrades to 192.168.1.249:5555; Knot trusts forwarded addresses only from 192.168.1.11/32. The script checks /xrpc/com.atproto.sync.subscribeRepos through Caddy. It cannot test an authenticated user push without their credentials: after success, test a real clone/push/delete of a temporary branch over port 2223.

Automatic Updates

Both timer and service require /var/lib/knot/.upgrade-verified. The script creates it only after successful smoke tests, then starts the timer. A separate migration marker prevents Rust knot from initializing an empty identity.

Weekly updates fast-forward the configuration, update inputs, build, activate, and health-check before committing/pushing flake.lock. Preexisting staged changes, lockfile edits, or Nix configuration edits cause refusal. Unrelated unstaged DNS edits remain untouched. Git runs noninteractively; update and migration jobs share a process lock.

A failed build leaves the live generation unchanged and restores the script's own lockfile changes. After activation is attempted, failure stops the application services and timer and retains state/lockfile; old binaries are not automatically run against possibly forward-migrated data. Publication failure retains the healthy deployed generation and local changes; reconcile them with origin before the next update.

Future versions may compile on asusmini unless supplied by a binary cache. Only the initial v1.20.0 cutover is guaranteed to reuse staged binaries.

Failure And Retry

Before activation, failed migration targets are quarantined in the backup directory and unchanged Go knot is restarted. After activation is attempted, the script stops Rust knot/updates but preserves migrated state: blind rollback could lose accepted writes or use an incompatible format. Investigate before reopening writes. Runtime timer masks do not survive reboot; do not reboot an old generation unattended during recovery.

A completed rerun verifies the live Rust service and resumes the gated timer; it never migrates again or reinstalls an old staged generation. An incomplete target without a completion marker is refused. Power loss or SIGKILL midway requires inspection/recovery, not a blind rerun.

Staging And Tests

/home/anish/tangled-v1.20.0-preflight contains system-path, tools-path, upgrade.export.gz, and its checksum. The archive contains paths missing at staging time; avoid garbage collection before import. Import alone activates nothing. The root-authorized import accepts unsigned local products only for that command, without changing global Nix trust settings.

Local/server runtime and synthetic-migration smoke tests passed earlier. Mocked orchestration tests cover success, preflight refusal, migration failure, post-activation failure, secret cleanup, idempotence, and publication ordering. They do not simulate a full boot or replace the on-host checks.

Upstream's v1.20.0 spindle still reports hardcoded wire version v1.16.1 in its describe endpoint; identify its actual package by the service store path.

6.4 KiB Raw Blame History