From e3d4ae696dafa316f5a06afa0acbd14f11c90e25 Mon Sep 17 00:00:00 2001
From: Anish Lakhwara <anish+git@lakhwara.com>
Date: Sun, 1 Nov 2026 15:40:12 -0700
Subject: [PATCH] asusmini: script Tangled cutover and resume checked updates

---
 README.md                      |  12 +-
 TANGLED-MIGRATION.md           | 365 +++++++++-----------------------
 flake.lock                     |   3 -
 flake.nix                      |   7 +-
 hosts/asusmini/auto-update.nix | 118 ++++++++---
 scripts/upgrade-knot.sh        | 226 ++++++++++++++++++++
 tests/test_auto_update.py      | 203 ++++++++++++++++++
 tests/test_upgrade_knot.py     | 367 +++++++++++++++++++++++++++++++++
 8 files changed, 987 insertions(+), 314 deletions(-)
 create mode 100644 scripts/upgrade-knot.sh
 create mode 100644 tests/test_auto_update.py
 create mode 100644 tests/test_upgrade_knot.py

diff --git a/README.md b/README.md
index ec7e883..42551ed 100644
--- a/README.md
+++ b/README.md
@@ -84,16 +84,16 @@ sudo nixos-rebuild switch --flake .#asusmini
 
 #### Auto-Update
 
-The timer is currently disabled for the Tangled migration. Tangled is pinned
-to v1.20.0 and uses its own locked dependencies so the tested binaries can
-be copied to the server without recompilation. Re-enable updates deliberately
-only after cutover and verification.
+The timer is gated until `scripts/upgrade-knot.sh` completes the Tangled
+migration and smoke tests. It then resumes weekly updates, including Tangled.
+The initial lock selects tested v1.20.0 packages with their own dependencies
+so the cutover reuses copied binaries without recompilation.
 
 A systemd timer runs weekly to:
 1. Pull latest git changes
 2. Run `nix flake update`
-3. Commit and push flake.lock changes
-4. Run `nixos-rebuild switch` with automatic rollback on failure
+3. Build, activate, and health-check the new configuration
+4. Commit and push flake.lock only after successful validation
 
 **Setup required:**
 ```bash
diff --git a/TANGLED-MIGRATION.md b/TANGLED-MIGRATION.md
index 437137e..fca50f0 100644
--- a/TANGLED-MIGRATION.md
+++ b/TANGLED-MIGRATION.md
@@ -1,306 +1,129 @@
-# Tangled v1.20.0 Preflight
+# Tangled Upgrade
 
-Status: local and server smoke tests passed. The operator completed the
-real-data dry-run successfully: 23 repositories, 16 members, 12 collaborator
-grants, zero skips, and no ACL/table disagreement. The master key has an
-off-host backup. Nothing has been deployed or migrated.
+The initial cutover uses tested v1.20.0 packages. `flake.lock` still selects
+commit `2096d06d938e680297adeadd71b8320c9f3491d4`; the input URL tracks upstream
+again so subsequent weekly updates can advance Tangled. Upstream's own nixpkgs
+is retained to reuse the built release binaries.
 
-The operator created `secrets/knot-env.age` on asusmini. The encrypted file
-and its recipient rule have been synchronized into this checkout; the key
-has not been decrypted here. The new configuration wires it into agenix at
-`/run/agenix/knot-env`, root-owned with mode `0400`.
+## Run The Script
 
-## Release And Build Outputs
-
-Verified upstream tag: `v1.20.0`, commit
-`2096d06d938e680297adeadd71b8320c9f3491d4`.
-
-Built on the workstation using the release's own locked inputs, not this
-repository's `nixpkgs` override:
-
-| Package | Exact Store Path |
-| --- | --- |
-| Rust knot | `/nix/store/7wv0kkflzpihyzm9k7mahxdci97r8k0h-knot-server-1.20.0` |
-| Migrator | `/nix/store/h3d5c7wad0y6nmlkllrjq95g15j6zgpw-knot-migrate-1.20.0` |
-| Spindle | `/nix/store/id8ma5qaj5q6msfjw2llppsz7w46jy9i-spindle-1.20.0` |
-
-These outputs have local GC roots under `/tmp/opencode/`. Do not delete them
-until the outputs have durable roots on the server.
-
-## Completed Checks
-
-- All three packages built successfully.
-- Rust knot configuration validation and HTTP health passed at v1.20.0.
-- Knot describe/contact, DID document, and SSH host-key matching passed.
-- Unauthenticated repository creation was rejected with 401/403.
-- Knot signing identity and SSH identity persisted across restart.
-- Spindle started with disposable state; health, DID, and contact passed.
-- Synthetic Go-database migration passed both dry-run and copy migration.
-- Synthetic migration preserved the source database and bare repository.
-- A temporary NixOS configuration evaluated with Rust knot and spindle at
-  1.20.0, Go knot disabled, and Rust `wantedBy = []`.
-- Uploaded the closure archive and test scripts to the server and verified
-  that the archive checksum matches the local copy.
-
-The upstream v1.20.0 spindle source hardcodes `wireVersion = "v1.16.1"` in
-`spindle/xrpc/org_tangled_spindle_describe_spindle.go`. Its describe endpoint
-is not a reliable indicator of the package release; verify the store path.
-
-These are startup and migration smoke tests, not a full CI workflow test or
-an authenticated end-to-end clone/push test against migrated production data.
-
-## Server Artifacts
-
-SSH: `anish@pds.commonscomputer.com`.
-
-Directory: `/home/anish/tangled-v1.20.0-preflight`.
-
-Archive: `tangled-v1.20.0-closure.export.gz` (137 MiB compressed), containing
-the package closures plus Python/OpenSSL for the test harness.
-
-SHA-256:
-`aa50f7edf3f2e4721f27abc86e4cdd6476d66eeed3dd013ddc5252dcc228a4d3`.
-
-The plain `nix copy --to ssh-ng://anish@pds.commonscomputer.com` attempt was
-rejected because the build outputs are unsigned and only root is trusted.
-No signature policy or trusted-user configuration was changed.
-
-## Network Plan
-
-Administrative OpenSSH remains on port **22**. Rust knot uses **2223**;
-the upstream module opens that port in the NixOS firewall. Public port 2222
-already belongs to the separate Gitea service and must not be repurposed.
-
-The Ansible Caddy template already proxies all requests, including WebSocket
-upgrades, from `knot.commonscomputer.com` to `192.168.1.249:5555`. No path-specific
-`/events` rule needs replacing. Knot trusts forwarded addresses only from
-the Caddy LXC, `192.168.1.11/32`.
-
-WAN forwarding is not managed by either repository. Before cutover, either
-arrange external TCP 2223 -> `192.168.1.249:2223`, or use the existing Tailscale
-address `100.122.40.27:2223`. SSH Git clients must specify the new port. The
-old port-22 Git SSH URLs will not work after removing the Go module; do not
-redirect public port 22 if that is your only administrative access path.
-
-## Stage The Built System (No Downtime)
-
-The system build and migration-tools bundle are exported separately from
-the earlier smoke archive. The staging directory contains `system-path`,
-`tools-path`, `upgrade.export.gz`, and `upgrade.export.gz.sha256`.
-
-As root on asusmini, before stopping anything:
+Pull the latest configuration on asusmini, then run:
 
 ```sh
-sudo -i
+sudo bash /etc/commonscomputing-nix/scripts/upgrade-knot.sh
 ```
 
-In that root Bash session:
+The script rehearses first, prints the migration report, and asks you to type
+`MIGRATE` before stopping Go knot. Keep another administrative SSH session open,
+back up the master key off-host, and ask users to pause writes during cutover.
+`--yes` skips the approval prompt. Running the script authorizes migration and
+deployment, not merely a smoke test.
+
+Root's Git identity must be able to fetch and push to `origin`. The script
+checks `ls-remote` and `push --dry-run` before downtime. Origin is the separate
+Gitea service on port 2222, not this knot.
+
+If encrypted-secret edits prevent a fast-forward pull, preserve only those
+edits first; leave unrelated DNS changes alone:
 
 ```sh
-set -euo pipefail
-test "$EUID" -eq 0
-cd /home/anish/tangled-v1.20.0-preflight
-sha256sum --check upgrade.export.gz.sha256
-gzip -dc upgrade.export.gz | nix-store --import --option require-sigs false
-system=$(cat system-path)
-tools=$(cat tools-path)
-nix-store --realise "$system" --add-root /var/lib/knot-upgrade-system
-nix-store --realise "$tools" --add-root /var/lib/knot-upgrade-tools
-test -x "$system/bin/switch-to-configuration"
-test -x "$tools/bin/knot-migrate"
-```
-
-The archive contains only paths missing from the server at staging time.
-Do not garbage-collect the server store between staging and activation.
-The signature exception applies to this one root-authorized import, not to
-persistent Nix configuration. Importing and rooting paths do not activate them.
-
-The encrypted secret and recipient rule are already committed upstream.
-The server currently has the same edits locally, including a staged new file.
-Preserve those edits in a path-limited stash before pulling; do not create
-another secret commit and diverge the server branch again:
-
-```sh
-git -C /etc/commonscomputing-nix stash push \
+sudo git -C /etc/commonscomputing-nix stash push \
   -m 'pre-cutover encrypted secret backup' -- secrets/knot-env.age secrets/secrets.nix
-git -C /etc/commonscomputing-nix pull --ff-only
-sha256sum /etc/commonscomputing-nix/secrets/knot-env.age
+sudo git -C /etc/commonscomputing-nix pull --ff-only
 ```
 
-The encrypted file's SHA-256 should remain
+The encrypted secret is already committed; do not pop an identical stash over
+it. Its SHA-256 is
 `b26f36d098a337a98217aa4e73d2f39eb4e8864337dfcface937093f5557f0ac`.
-Keep the stash until cutover is verified; do not pop it over the incoming
-identical secret. The path-limited stash leaves unrelated DNS edits alone.
-Activation below uses the exact built path, not a new flake evaluation/update.
 
-## Decrypt And Rehearse With The Persistent Key
+## Happy Path
 
-Continue in the same root Bash session. Do not enable shell tracing (`set -x`).
+1. Lock out concurrent updates and validate the host, staging files, and Git access.
+2. Import the prebuilt system/tools if missing, after checking the archive checksum.
+3. Decrypt the persistent key into root-only runtime storage and rehearse.
+4. Obtain downtime approval; stop the timer and Go knot.
+5. Back up the database, repositories, SSH identity, and old NixOS generation.
+6. Copy-migrate repositories/access grants while preserving all original source data.
+7. Validate the imported identity/configuration and install the prebuilt system.
+8. Set ownership, check agenix supplied the same key, and start Rust knot.
+9. Check local/public APIs, service state, SSH fingerprint, and Caddy WebSocket upgrade.
+10. Mark verification complete, start the weekly timer, and request an indexer crawl.
 
-```sh
-export PATH="$tools/bin:$PATH"
-umask 077
-runtime=$(mktemp -d /run/knot-migration.XXXXXX)
-trap 'rm -f "$runtime/knot.env"; rmdir "$runtime"; unset KNOT_MASTER_KEY' EXIT
-agenix -d /etc/commonscomputing-nix/secrets/knot-env.age \
-  -i /etc/ssh/ssh_host_ed25519_key > "$runtime/knot.env"
-chmod 0400 "$runtime/knot.env"
-set -a
-. "$runtime/knot.env"
-set +a
-test -n "${KNOT_MASTER_KEY:-}"
+Source data stays in `/home/git`; Rust state lives in `/var/lib/knot`. The
+previous rehearsal found 23 repositories, 16 members, 12 collaborator grants,
+zero skips, and no ACL/table disagreement. Check the fresh report for changes.
 
-migration_args=(
-  --source-db /home/git/knotserver.db
-  --source-repos /home/git
-  --hostname knot.commonscomputer.com
-  --host-key /etc/ssh/ssh_host_ed25519_key
-  --plc-url https://plc.directory
-  --contact-email pds@email.commonscomputer.com
-  --master-key-env KNOT_MASTER_KEY
-  --object-format sha1
-  --target /var/lib/knot
-)
-knot-migrate "${migration_args[@]}" --dry-run
-```
+Backups are root-only under `/var/backups/knot-cutover.*`. They contain private
+key material, including `repo-signing-keys.json`; back them up encrypted off-host.
+The master key is not printed and is scoped to migration/config validation.
+The runtime plaintext file is removed on ordinary success/failure and signals;
+SIGKILL may leave it root-readable in `/run` until cleanup or reboot.
 
-Check the expected 23 repositories, 16 members, 12 grants, zero skips, and
-no ownership drift. SQLite may create WAL/SHM files for read-only access.
-This uses the real backed-up key, not the disposable preflight script's key.
-If it fails, stop here; production is still running.
+## Networking
 
-## Cut Over (Downtime Starts Here)
+Administrative SSH stays on **22**. Rust knot Git SSH uses **2223**, opened by
+the NixOS module. Use `100.122.40.27:2223` over Tailscale, or arrange external
+TCP 2223 forwarding to `192.168.1.249:2223`. Existing port-22 Git URLs need an
+explicit port or client SSH configuration. Public 2222 belongs to Gitea and
+must not be repurposed. The script does not change router forwarding or sshd.
 
-Run only after explicitly approving downtime and confirming the SSH plan.
-Keep a second administrative SSH connection open. Do not reboot midway.
+Caddy proxies all paths and WebSocket upgrades to `192.168.1.249:5555`; Knot
+trusts forwarded addresses only from `192.168.1.11/32`. The script checks
+`/xrpc/com.atproto.sync.subscribeRepos` through Caddy. It cannot test an
+authenticated user push without their credentials: after success, test a real
+clone/push/delete of a temporary branch over port 2223.
 
-```sh
-test ! -e /var/lib/knot/.migration-complete
-test ! -e /var/lib/knot/sealed-keys
-old_system=$(readlink -f /run/current-system)
-backup=/var/backups/knot-cutover-$(date +%Y%m%dT%H%M%S)
-install -d -m 0700 "$backup"
-printf '%s\n' "$old_system" > "$backup/old-system"
-nix-store --realise "$old_system" --add-root "$backup/system"
+## Automatic Updates
 
-systemctl stop knot.service
-sqlite3 'file:/home/git/knotserver.db?mode=ro' ".backup '$backup/knotserver.db'"
-tar --acls --xattrs --numeric-owner -C / -czf "$backup/go-knot.tar.gz" \
-  home/git etc/ssh/ssh_host_ed25519_key etc/ssh/ssh_host_ed25519_key.pub
-knot-migrate "${migration_args[@]}"
-```
+Both timer and service require `/var/lib/knot/.upgrade-verified`. The script
+creates it only after successful smoke tests, then starts the timer. A separate
+migration marker prevents Rust knot from initializing an empty identity.
 
-Inspect the migration report before continuing. The default is **copy** mode;
-never add `--consume-source`. Source repositories/database remain in place.
-On any error, do not create the marker or start Rust knot. The old service
-can be restarted while investigating (`systemctl start knot.service`). A
-failed migration may leave partial target state; inspect it and quarantine
-the entire target before a retry. Never blindly reuse it or delete the source.
+Weekly updates fast-forward the configuration, update inputs, build, activate,
+and health-check **before** committing/pushing `flake.lock`. Preexisting staged
+changes, lockfile edits, or Nix configuration edits cause refusal. Unrelated
+unstaged DNS edits remain untouched. Git runs noninteractively; update and
+migration jobs share a process lock.
 
-```sh
-test -s /var/lib/knot/sealed-keys
-test -f /var/lib/knot/sealed-keys
-test -s /var/lib/knot/ssh_host_key
-test -f /var/lib/knot/ssh_host_key
-ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
-ssh-keygen -y -f /var/lib/knot/ssh_host_key | ssh-keygen -lf -
-```
+A failed build leaves the live generation unchanged and restores the script's
+own lockfile changes. After activation is attempted, failure stops the application
+services and timer and retains state/lockfile; old binaries are not automatically
+run against possibly forward-migrated data. Publication failure retains the
+healthy deployed generation and local changes; reconcile them with origin
+before the next update.
 
-The two fingerprints must match. Then validate the actual deployment config:
+Future versions may compile on asusmini unless supplied by a binary cache.
+Only the initial v1.20.0 cutover is guaranteed to reuse staged binaries.
 
-```sh
-/nix/store/7wv0kkflzpihyzm9k7mahxdci97r8k0h-knot-server-1.20.0/bin/knot-server \
-  --config "$system/etc/knot/config.toml" validate
-if [ -f /var/lib/knot/repo-signing-keys.json ]; then
-  mv /var/lib/knot/repo-signing-keys.json "$backup/"
-fi
-```
+## Failure And Retry
 
-`repo-signing-keys.json` contains plaintext repository private keys. Keep the
-backup directory root-only and back it up encrypted off-host. Do not commit it.
+Before activation, failed migration targets are quarantined in the backup
+directory and unchanged Go knot is restarted. After activation is attempted,
+the script stops Rust knot/updates but preserves migrated state: blind rollback
+could lose accepted writes or use an incompatible format. Investigate before
+reopening writes. Runtime timer masks do not survive reboot; do not reboot an
+old generation unattended during recovery.
 
-Install the prebuilt system. The absent marker keeps Rust stopped while
-activation creates the `knot` service user and decrypts the agenix secret:
+A completed rerun verifies the live Rust service and resumes the gated timer;
+it never migrates again or reinstalls an old staged generation. An incomplete
+target without a completion marker is refused. Power loss or SIGKILL midway
+requires inspection/recovery, not a blind rerun.
 
-```sh
-nix-env --profile /nix/var/nix/profiles/system --set "$system"
-"$system/bin/switch-to-configuration" switch
-chown -R knot:knot /var/lib/knot
-chmod 0600 /var/lib/knot/ssh_host_key /var/lib/knot/sealed-keys
-test -s /run/agenix/knot-env
-cmp --silent "$runtime/knot.env" /run/agenix/knot-env
-touch /var/lib/knot/.migration-complete
-if ! systemctl start knot-rs.service || \
-   ! curl --fail --silent --show-error --retry 10 --retry-connrefused \
-     --retry-delay 1 http://127.0.0.1:5555/xrpc/_health; then
-  rm -f /var/lib/knot/.migration-complete
-  systemctl stop knot-rs.service
-  exit 1
-fi
-rm -f "$runtime/knot.env"
-rmdir "$runtime"
-unset KNOT_MASTER_KEY
-trap - EXIT
-```
+## Staging And Tests
 
-Stop on any activation error rather than continuing to create the marker.
-The marker allows the service to start normally on subsequent boots.
-Auto-update remains disabled until deliberately re-enabled after verification.
+`/home/anish/tangled-v1.20.0-preflight` contains `system-path`, `tools-path`,
+`upgrade.export.gz`, and its checksum. The archive contains paths missing at
+staging time; avoid garbage collection before import. Import alone activates
+nothing. The root-authorized import accepts unsigned local products only for
+that command, without changing global Nix trust settings.
 
-## Final Checks
+Local/server runtime and synthetic-migration smoke tests passed earlier. Mocked
+orchestration tests cover success, preflight refusal, migration failure,
+post-activation failure, secret cleanup, idempotence, and publication ordering.
+They do not simulate a full boot or replace the on-host checks.
 
-```sh
-systemctl is-active knot-rs spindle sshd
-systemctl is-active knot.service || true
-journalctl -u knot-rs -n 80 --no-pager
-curl --fail http://127.0.0.1:5555/xrpc/_health
-curl --fail https://knot.commonscomputer.com/xrpc/_health
-curl --fail https://knot.commonscomputer.com/xrpc/org.tangled.knot.describeKnot
-curl --fail https://knot.commonscomputer.com/.well-known/did.json
-ssh-keyscan -p 2223 -t ed25519 127.0.0.1 2>/dev/null | ssh-keygen -lf -
-```
-
-Expect v1.20.0, the original knot hostname/DID and contact, and the original
-SSH fingerprint. From a client, clone a known migrated repository, push a
-temporary branch using port 2223, then delete that branch. Check membership
-and collaborator access in the web UI. Test WebSocket upgrade for
-`/xrpc/com.atproto.sync.subscribeRepos` through Caddy; a successful HTTP health
-request alone does not prove the event stream works.
-
-Only after verification, request an indexer crawl:
-
-```sh
-curl --fail -X POST https://index.tangled.network/xrpc/com.atproto.sync.requestCrawl \
-  -H 'Content-Type: application/json' \
-  -d '{"hostname":"knot.commonscomputer.com"}'
-```
-
-## Roll Back If Needed
-
-Before activation, just restart `knot.service`; the source was copied, not
-moved. After activation, use the preserved generation:
-
-```sh
-systemctl stop knot-rs.service
-rm -f /var/lib/knot/.migration-complete
-old_system=$(cat "$backup/old-system")
-nix-env --profile /nix/var/nix/profiles/system --set "$old_system"
-"$old_system/bin/switch-to-configuration" switch
-systemctl stop auto-update.timer
-systemctl mask --runtime auto-update.timer
-systemctl start knot.service
-```
-
-The old generation may re-enable the timer, hence the explicit stop and
-runtime mask. The mask does not survive reboot: do not reboot the rolled-back
-generation until a declarative configuration with the timer disabled is ready.
-Retain all backups and migrated state. Writes made after cutover are not copied
-back to the Go database by rollback; coordinate with users before reopening
-writes. Never overwrite the source database unless recovery actually needs it.
-
-At the last check, production knot and spindle were active, the auto-update
-timer was inactive, and `/run/current-system` was unchanged:
-`/nix/store/sv9r0h1xs6fsfjvbqv2vwslj42wm08v4-nixos-system-asusmininix-26.11.20261027.31ecbac`.
+Upstream's v1.20.0 spindle still reports hardcoded wire version `v1.16.1` in
+its describe endpoint; identify its actual package by the service store path.
 
 ## Upstream Guides
 
diff --git a/flake.lock b/flake.lock
index 5e7cf42..bbddaab 100644
--- a/flake.lock
+++ b/flake.lock
@@ -521,9 +521,6 @@
         "url": "https://tangled.org/@tangled.org/core"
       },
       "original": {
-        "ref": "refs/tags/v1.20.0",
-        "rev": "2096d06d938e680297adeadd71b8320c9f3491d4",
-        "shallow": true,
         "type": "git",
         "url": "https://tangled.org/@tangled.org/core"
       }
diff --git a/flake.nix b/flake.nix
index 104489f..2629ca0 100644
--- a/flake.nix
+++ b/flake.nix
@@ -4,8 +4,9 @@
   inputs = {
     nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
     tangled = {
-      # Keep the release's own package set so deployment reuses tested binaries.
-      url = "git+https://tangled.org/@tangled.org/core?ref=refs/tags/v1.20.0&rev=2096d06d938e680297adeadd71b8320c9f3491d4&shallow=1";
+      # flake.lock initially selects tested v1.20.0; weekly updates track upstream.
+      # Keep upstream's own nixpkgs so copied release binaries remain reusable.
+      url = "git+https://tangled.org/@tangled.org/core";
     };
     agenix = {
       url = "github:ryantm/agenix";
@@ -57,6 +58,8 @@
             pkgs.sqlite
             pkgs.openssl
             pkgs.git
+            pkgs.curl
+            pkgs.jq
           ];
         };
 
diff --git a/hosts/asusmini/auto-update.nix b/hosts/asusmini/auto-update.nix
index 440ede3..9817a36 100644
--- a/hosts/asusmini/auto-update.nix
+++ b/hosts/asusmini/auto-update.nix
@@ -1,57 +1,116 @@
-{ config, pkgs, ... }:
+{ pkgs, ... }:
 
 {
   systemd.services.auto-update = {
     description = "Auto-update NixOS configuration";
     path = with pkgs; [
+      coreutils
+      curl
       git
+      jq
       nix
       openssh
+      util-linux
     ];
 
     serviceConfig = {
       Type = "oneshot";
       User = "root";
       WorkingDirectory = "/etc/commonscomputing-nix";
+      TimeoutStartSec = "6h";
+    };
+
+    restartIfChanged = false;
+    unitConfig.ConditionPathExists = "/var/lib/knot/.upgrade-verified";
+    unitConfig.X-StopOnRemoval = false;
+    environment = {
+      GIT_TERMINAL_PROMPT = "0";
+      GIT_SSH_COMMAND = "${pkgs.openssh}/bin/ssh -o BatchMode=yes -o ConnectTimeout=15";
     };
 
     script = ''
-      set -e
+      set -euo pipefail
+      exec 9>/run/lock/commonscomputing-update.lock
+      flock --nonblock 9 || exit 0
 
-      echo "Pulling latest changes..."
-      git pull
+      lock_snapshot="$(${pkgs.coreutils}/bin/mktemp)"
+      snapshot_ready=0
+      activated=0
+      validated=0
+      system_profile=/nix/var/nix/profiles/system
 
-      echo "Updating flake inputs..."
-      nix flake update
+      cleanup() {
+        status=$?
+        trap - EXIT
+        set +e
+        if [ "$status" -ne 0 ] && [ "$snapshot_ready" -eq 1 ] && [ "$activated" -eq 0 ]; then
+          ${pkgs.coreutils}/bin/cp "$lock_snapshot" flake.lock
+        fi
+        if [ "$status" -ne 0 ] && [ "$activated" -eq 1 ] && [ "$validated" -eq 0 ]; then
+          # New binaries may already have migrated persistent state. Never
+          # blindly roll back their data format or discard accepted writes.
+          rm -f /var/lib/knot/.upgrade-verified
+          ${pkgs.systemd}/bin/systemctl stop auto-update.timer
+          ${pkgs.systemd}/bin/systemctl stop knot-rs.service spindle.service
+          echo "Activation/health failed: application services and updates stopped; inspect state and retained flake.lock" >&2
+        elif [ "$status" -ne 0 ] && [ "$validated" -eq 1 ]; then
+          echo "System is healthy, but Git publication failed; retain/reconcile the local lockfile or commit before retrying" >&2
+        fi
+        ${pkgs.coreutils}/bin/rm -f "$lock_snapshot"
+        exit "$status"
+      }
+      trap cleanup EXIT
+      trap 'exit 130' INT
+      trap 'exit 143' TERM
 
-      # Tangled stays at the release pinned in flake.nix. Upgrade it deliberately
-      # after migration/testing rather than tracking master automatically.
-
-      # Check if there are changes to commit
-      if ! git diff --quiet flake.lock; then
-        echo "Committing flake.lock updates..."
-        git add flake.lock
-        git commit -m "auto-update: flake inputs $(date -Iseconds)"
-        
-        echo "Pushing changes..."
-        git push
-      else
-        echo "No flake.lock changes to commit"
-      fi
-
-      echo "Rebuilding system..."
-      if ! ${config.system.build.nixos-rebuild}/bin/nixos-rebuild switch --flake .#asusmini; then
-        echo "Build/switch failed, staying on current generation"
+      if ! git diff --cached --quiet; then
+        echo "Refusing to run with staged changes"
         exit 1
       fi
+      if ! git diff --quiet -- flake.lock; then
+        echo "Refusing to run with preexisting flake.lock changes"
+        exit 1
+      fi
+      if ! git diff --quiet -- '*.nix'; then
+        echo "Refusing to deploy uncommitted Nix configuration changes"
+        exit 1
+      fi
+
+      git pull --ff-only
+      git push --dry-run
+      ${pkgs.coreutils}/bin/cp flake.lock "$lock_snapshot"
+      snapshot_ready=1
+      nix flake update
+
+      echo "Building NixOS toplevel..."
+      toplevel="$(${pkgs.nix}/bin/nix build .#nixosConfigurations.asusmini.config.system.build.toplevel --no-link --print-out-paths)"
+
+      echo "Activating new system..."
+      activated=1
+      ${pkgs.nix}/bin/nix-env -p "$system_profile" --set "$toplevel"
+      "$toplevel/bin/switch-to-configuration" switch
+
+      ${pkgs.systemd}/bin/systemctl is-active --quiet knot-rs
+      ${pkgs.systemd}/bin/systemctl is-active --quiet spindle
+      ${pkgs.systemd}/bin/systemctl is-active --quiet sshd
+      ${pkgs.curl}/bin/curl --fail --silent --show-error --max-time 10 --retry 5 --retry-connrefused --retry-delay 1 http://127.0.0.1:5555/xrpc/_health | jq -e '.version | length > 0'
+      ${pkgs.curl}/bin/curl --fail --silent --show-error --max-time 10 --retry 5 --retry-connrefused --retry-delay 1 http://127.0.0.1:6555/_health | jq -e '.status == "ok"'
+      ${pkgs.curl}/bin/curl --fail --silent --show-error --max-time 15 https://knot.commonscomputer.com/xrpc/org.tangled.knot.describeKnot | jq -e '.did == "did:web:knot.commonscomputer.com" and .contact.email == "pds@email.commonscomputer.com"'
+      validated=1
+      snapshot_ready=0
+
+      if ! git diff --quiet -- flake.lock; then
+        git add flake.lock
+        git commit -m "auto-update: flake inputs $(${pkgs.coreutils}/bin/date -Iseconds)"
+      fi
+      git push
 
       echo "Auto-update completed successfully"
     '';
   };
 
   systemd.timers.auto-update = {
-    # Re-enable deliberately after the supervised Tangled migration is verified.
-    enable = false;
+    enable = true;
     description = "Auto-update timer";
     wantedBy = [ "timers.target" ];
     timerConfig = {
@@ -59,11 +118,6 @@
       Persistent = true; # Run on boot if missed
       RandomizedDelaySec = "1h"; # Add some randomness
     };
+    unitConfig.ConditionPathExists = "/var/lib/knot/.upgrade-verified";
   };
-
-  # TODO: Set up SSH key for git push access
-  # Options:
-  # 1. Deploy key with write access to the repo
-  # 2. Generate SSH key on server and add to GitHub
-  # Command to generate: ssh-keygen -t ed25519 -f /root/.ssh/commons-nix-deploy -N ""
 }
diff --git a/scripts/upgrade-knot.sh b/scripts/upgrade-knot.sh
new file mode 100644
index 0000000..a8b7c57
--- /dev/null
+++ b/scripts/upgrade-knot.sh
@@ -0,0 +1,226 @@
+#!/usr/bin/env bash
+set -euo pipefail
+unset KNOT_MASTER_KEY
+
+# These paths are deliberately fixed to this host and its rehearsed migration.
+stage=/home/anish/tangled-v1.20.0-preflight
+repo=/etc/commonscomputing-nix
+state=/var/lib/knot
+source_dir=/home/git
+backup_dir=/var/backups
+profile=/nix/var/nix/profiles/system
+current_system=/run/current-system
+ssh_key=/etc/ssh/ssh_host_ed25519_key
+lock_file=/run/lock/commonscomputing-update.lock
+runtime_template=/run/knot-upgrade.XXXXXX
+system_root=/var/lib/knot-upgrade-system
+tools_root=/var/lib/knot-upgrade-tools
+
+if [ "$(id -u)" != 0 ] || [ "$(hostname)" != asusmininix ]; then
+  printf 'Run this script as root on asusmininix.\n' >&2
+  exit 1
+fi
+if [ "$#" -gt 1 ] || { [ "$#" -eq 1 ] && [ "$1" != --yes ]; }; then
+  printf 'Usage: bash scripts/upgrade-knot.sh [--yes]\n' >&2
+  exit 1
+fi
+exec 9>"$lock_file"
+flock --nonblock 9 || { printf 'Another update is running.\n' >&2; exit 1; }
+if systemctl is-active --quiet auto-update.service; then
+  printf 'Wait for auto-update.service to finish before migrating.\n' >&2
+  exit 1
+fi
+
+system=$(<"$stage/system-path")
+tools=$(<"$stage/tools-path")
+case "$system" in /nix/store/*-nixos-system-asusmininix-*) ;; *) exit 1 ;; esac
+case "$tools" in /nix/store/*-knot-migration-tools) ;; *) exit 1 ;; esac
+if ! nix-store --check-validity "$system" "$tools"; then
+  read -r expected _ < "$stage/upgrade.export.gz.sha256"
+  actual=$(sha256sum "$stage/upgrade.export.gz")
+  test "${actual%% *}" = "$expected"
+  gzip -dc "$stage/upgrade.export.gz" | nix-store --import --option require-sigs false
+fi
+nix-store --realise "$system" --add-root "$system_root" >/dev/null
+nix-store --realise "$tools" --add-root "$tools_root" >/dev/null
+export PATH="$tools/bin:$PATH"
+test -x "$system/bin/switch-to-configuration"
+test -x "$system/sw/bin/knot-server"
+# An older staged build intentionally disabled the timer; do not install it.
+grep -Fq 'ConditionPathExists=/var/lib/knot/.upgrade-verified' \
+  "$system/etc/systemd/system/auto-update.timer"
+grep -Fq 'ConditionPathExists=/var/lib/knot/.upgrade-verified' \
+  "$system/etc/systemd/system/auto-update.service"
+
+smoke() {
+  for unit in knot-rs.service spindle.service sshd.service; do
+    systemctl is-active --quiet "$unit"
+  done
+  curl --fail --silent --show-error --max-time 10 --retry 5 --retry-connrefused \
+    --retry-delay 1 http://127.0.0.1:5555/xrpc/_health | jq -e '.version | length > 0'
+  curl --fail --silent --show-error --max-time 10 --retry 5 --retry-connrefused \
+    --retry-delay 1 http://127.0.0.1:6555/_health | jq -e '.status == "ok"'
+  curl --fail --silent --show-error --max-time 15 \
+    https://knot.commonscomputer.com/xrpc/org.tangled.knot.describeKnot \
+    | jq -e '.did == "did:web:knot.commonscomputer.com" and .contact.email == "pds@email.commonscomputer.com"'
+  read -r _ expected_key _ < <(ssh-keygen -lf "$ssh_key.pub")
+  scanned=$(ssh-keyscan -T 5 -p 2223 -t ed25519 127.0.0.1 2>/dev/null)
+  read -r _ actual_key _ < <(printf '%s\n' "$scanned" | ssh-keygen -lf -)
+  test "$expected_key" = "$actual_key"
+  # The upgraded endpoint is a long-lived WebSocket: timeout after the handshake.
+  status=$(curl --silent --max-time 3 --output /dev/null --write-out '%{http_code}' \
+    -H 'Connection: Upgrade' -H 'Upgrade: websocket' -H 'Sec-WebSocket-Version: 13' \
+    -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
+    https://knot.commonscomputer.com/xrpc/com.atproto.sync.subscribeRepos || true)
+  test "$status" = 101
+}
+
+export GIT_TERMINAL_PROMPT=0
+export GIT_SSH_COMMAND='ssh -o BatchMode=yes -o ConnectTimeout=15'
+git -C "$repo" ls-remote --exit-code origin HEAD >/dev/null
+git -C "$repo" push --dry-run origin HEAD:main
+git -C "$repo" var GIT_AUTHOR_IDENT >/dev/null
+if ! git -C "$repo" diff --cached --quiet || \
+   ! git -C "$repo" diff --quiet -- '*.nix' flake.lock; then
+  printf 'Commit/stash staged changes and Nix/lockfile edits before cutover.\n' >&2
+  exit 1
+fi
+if [ -e "$state/.migration-complete" ]; then
+  printf 'Migration marker already exists; checking the live Rust service only.\n'
+  live_system=$(readlink -f "$current_system")
+  grep -Fq 'ConditionPathExists=/var/lib/knot/.upgrade-verified' \
+    "$live_system/etc/systemd/system/auto-update.timer"
+  grep -Fq 'ConditionPathExists=/var/lib/knot/.upgrade-verified' \
+    "$live_system/etc/systemd/system/auto-update.service"
+  smoke
+  touch "$state/.upgrade-verified"
+  systemctl unmask --runtime auto-update.timer
+  systemctl start auto-update.timer
+  systemctl is-active --quiet auto-update.timer
+  printf 'Already migrated; smoke tests passed and automatic updates are active.\n'
+  exit 0
+fi
+if [ -e "$state" ]; then
+  printf '%s exists without a completed migration. Inspect/quarantine it before retrying.\n' "$state" >&2
+  exit 1
+fi
+systemctl is-active --quiet knot.service
+test -r "$source_dir/knotserver.db"
+test -s "$ssh_key"
+old_system=$(readlink -f "$current_system")
+runtime=""
+backup=""
+downtime=0
+activation_attempted=0
+migration_started=0
+verified=0
+
+cleanup() {
+  result=$?
+  trap - EXIT INT TERM
+  set +e
+  if [ "$result" -ne 0 ] && [ "$downtime" -eq 1 ] && [ "$verified" -eq 0 ]; then
+    systemctl stop auto-update.timer
+    systemctl mask --runtime auto-update.timer
+    systemctl stop knot-rs.service
+    rm -f "$state/.migration-complete" "$state/.upgrade-verified"
+    if [ "$activation_attempted" -eq 1 ]; then
+      printf 'Activation was attempted: state may have changed. Updates/Rust knot are stopped.\n' >&2
+      printf 'Do NOT blindly roll back; inspect %s and %s before reopening writes.\n' "$state" "$backup" >&2
+    else
+      printf 'Migration failed before activation; restarting the unchanged Go knot.\n' >&2
+      if [ "$migration_started" -eq 1 ] && [ -e "$state" ]; then
+        mv "$state" "$backup/failed-target"
+      fi
+      if ! systemctl start knot.service; then
+        printf 'URGENT: Go knot could not be restarted; inspect %s.\n' "$backup" >&2
+      fi
+    fi
+    printf 'Backups retained in %s. Timer runtime mask does not survive reboot.\n' "$backup" >&2
+  fi
+  if [ -n "$runtime" ]; then
+    rm -f "$runtime/knot.env"
+    rmdir "$runtime"
+  fi
+  unset master_key
+  exit "$result"
+}
+trap cleanup EXIT
+trap 'exit 130' INT
+trap 'exit 143' TERM
+
+umask 077
+runtime=$(mktemp -d "$runtime_template")
+agenix -d "$repo/secrets/knot-env.age" -i "$ssh_key" > "$runtime/knot.env"
+chmod 0400 "$runtime/knot.env"
+master_key=""
+export -n master_key
+while IFS= read -r line || [ -n "$line" ]; do
+  case "$line" in
+    KNOT_MASTER_KEY=*) master_key=${line#KNOT_MASTER_KEY=} ;;
+    ''|\#*) ;;
+    *) printf 'Unexpected entry in knot-env.age: expected only KNOT_MASTER_KEY.\n' >&2; exit 1 ;;
+  esac
+done < "$runtime/knot.env"
+if ! [[ "$master_key" =~ ^[A-Za-z0-9+/]{43}=$ ]]; then
+  printf 'KNOT_MASTER_KEY must be an unquoted base64 encoding of 32 random bytes.\n' >&2
+  exit 1
+fi
+migration_args=(
+  --source-db "$source_dir/knotserver.db" --source-repos "$source_dir"
+  --hostname knot.commonscomputer.com --host-key "$ssh_key"
+  --plc-url https://plc.directory --contact-email pds@email.commonscomputer.com
+  --master-key-env KNOT_MASTER_KEY --object-format sha1 --target "$state"
+)
+KNOT_MASTER_KEY="$master_key" knot-migrate "${migration_args[@]}" --dry-run
+printf '\nThis will stop Go knot, back it up, COPY its data, and install the staged Rust system.\n'
+printf 'Git SSH moves to port 2223; administrative SSH stays on 22. Use Tailscale or arrange WAN forwarding.\n'
+printf 'Confirm the master key is backed up off-host and keep a second administrative session open.\n'
+if [ "${1:-}" != --yes ]; then
+  read -r -p 'Type MIGRATE to approve downtime: ' approval
+  test "$approval" = MIGRATE
+fi
+
+backup=$(mktemp -d "$backup_dir/knot-cutover.XXXXXX")
+printf '%s\n' "$old_system" > "$backup/old-system"
+nix-store --realise "$old_system" --add-root "$backup/system" >/dev/null
+systemctl stop auto-update.timer
+downtime=1
+systemctl stop knot.service
+sqlite3 "file:$source_dir/knotserver.db?mode=ro" ".backup '$backup/knotserver.db'"
+tar --acls --xattrs --numeric-owner -czf "$backup/go-knot.tar.gz" \
+  "$source_dir" "$ssh_key" "$ssh_key.pub"
+migration_started=1
+KNOT_MASTER_KEY="$master_key" knot-migrate "${migration_args[@]}"
+test -f "$state/ssh_host_key" && test -s "$state/ssh_host_key"
+test -f "$state/sealed-keys" && test -s "$state/sealed-keys"
+read -r _ expected_key _ < <(ssh-keygen -lf "$ssh_key.pub")
+read -r _ imported_key _ < <(ssh-keygen -lf "$state/ssh_host_key")
+test "$expected_key" = "$imported_key"
+KNOT_MASTER_KEY="$master_key" "$system/sw/bin/knot-server" --config "$system/etc/knot/config.toml" validate
+if [ -f "$state/repo-signing-keys.json" ]; then
+  mv "$state/repo-signing-keys.json" "$backup/"
+fi
+
+activation_attempted=1
+nix-env --profile "$profile" --set "$system"
+"$system/bin/switch-to-configuration" switch
+chown -R knot:knot "$state"
+chmod 0600 "$state/ssh_host_key" "$state/sealed-keys"
+cmp --silent "$runtime/knot.env" /run/agenix/knot-env
+touch "$state/.migration-complete"
+systemctl start knot-rs.service
+smoke
+verified=1
+touch "$state/.upgrade-verified"
+systemctl unmask --runtime auto-update.timer
+systemctl start auto-update.timer
+systemctl is-active --quiet auto-update.timer
+if ! curl --fail --silent --show-error --max-time 15 -X POST \
+  https://index.tangled.network/xrpc/com.atproto.sync.requestCrawl \
+  -H 'Content-Type: application/json' -d '{"hostname":"knot.commonscomputer.com"}'; then
+  printf 'Warning: request the indexer crawl again later; the local migration passed.\n' >&2
+fi
+printf '\nSUCCESS: Rust knot and spindle are healthy; weekly automatic updates are active.\n'
+printf 'Backups: %s (includes plaintext repository signing keys; back up encrypted off-host).\n' "$backup"
+printf 'Next: test an authenticated clone/push over Git SSH port 2223.\n'
diff --git a/tests/test_auto_update.py b/tests/test_auto_update.py
new file mode 100644
index 0000000..9a8fe91
--- /dev/null
+++ b/tests/test_auto_update.py
@@ -0,0 +1,203 @@
+import re
+import os
+import shutil
+import stat
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+NIX_FILE = ROOT / "hosts/asusmini/auto-update.nix"
+BASH = shutil.which("bash") or "/run/current-system/sw/bin/bash"
+
+
+FAKE = r'''#!/usr/bin/env python3
+import os
+import shutil
+import sys
+from pathlib import Path
+
+name = Path(sys.argv[0]).name
+args = sys.argv[1:]
+log = Path(os.environ["FAKE_LOG"])
+with log.open("a") as f:
+    f.write(name + " " + " ".join(args) + "\n")
+
+def fail(flag):
+    return os.environ.get(flag) == "1"
+
+if name == "flock":
+    pass
+elif name == "mktemp":
+    p = Path(os.environ["TMP_ROOT"]) / "lock-snapshot"
+    p.touch()
+    print(p)
+elif name == "cp":
+    shutil.copyfile(args[-2], args[-1])
+elif name == "rm":
+    for arg in args:
+        if not arg.startswith("-"):
+            Path(arg).unlink(missing_ok=True)
+elif name == "date":
+    print("fixture-date")
+elif name == "git":
+    operation = " ".join(args)
+    if "diff --cached --quiet" in operation and fail("STAGED_CHANGES"):
+        raise SystemExit(1)
+    if "diff --quiet -- *.nix" in operation and fail("NIX_CHANGES"):
+        raise SystemExit(1)
+    if "diff --quiet -- flake.lock" in operation:
+        raise SystemExit(0 if not Path("flake.lock").read_text().startswith("updated") else 1)
+    if args[:1] == ["pull"]:
+        if fail("PULL_FAIL"):
+            raise SystemExit(1)
+        Path("flake.lock").write_text("pulled lock\n")
+    elif args[:1] == ["push"] and "--dry-run" not in args and fail("PUBLISH_FAIL"):
+        raise SystemExit(1)
+    elif args[:1] == ["commit"]:
+        if fail("COMMIT_FAIL"):
+            raise SystemExit(1)
+        Path("flake.lock").write_text("committed lock\n")
+elif name == "nix":
+    if args[:2] == ["flake", "update"]:
+        Path("flake.lock").write_text("updated lock\n")
+    elif args[:1] == ["build"]:
+        if fail("BUILD_FAIL"):
+            raise SystemExit(1)
+        print(os.environ["TOPLEVEL"])
+elif name == "nix-env":
+    if fail("ENV_SET_FAIL"):
+        raise SystemExit(1)
+elif name == "systemctl":
+    if args[:2] == ["is-active", "--quiet"] and fail("HEALTH_FAIL"):
+        raise SystemExit(1)
+    if args[:1] == ["stop"] and fail("STOP_FAIL"):
+        raise SystemExit(1)
+elif name == "curl":
+    if fail("HEALTH_FAIL"):
+        raise SystemExit(1)
+elif name == "jq":
+    sys.stdin.read()
+    if fail("HEALTH_FAIL"):
+        raise SystemExit(1)
+else:
+    Path(os.environ["UNKNOWN_LOG"]).open("a").write(name + "\n")
+    raise SystemExit(127)
+'''
+
+
+class AutoUpdateTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = Path(tempfile.mkdtemp(prefix="auto-update-test-"))
+        self.bin = self.tmp / "bin"
+        self.bin.mkdir()
+        self.log = self.tmp / "commands.log"
+        self.unknown = self.tmp / "unknown.log"
+        fake = self.bin / "fake.py"
+        fake.write_text(FAKE)
+        fake.chmod(fake.stat().st_mode | stat.S_IXUSR)
+        for name in "flock mktemp cp rm date git nix nix-env systemctl curl jq".split():
+            (self.bin / name).symlink_to(fake)
+
+        self.repo = self.tmp / "repo"
+        self.repo.mkdir()
+        (self.repo / "flake.lock").write_text("initial lock\n")
+        self.toplevel = self.tmp / "toplevel"
+        (self.toplevel / "bin").mkdir(parents=True)
+        switch = self.toplevel / "bin" / "switch-to-configuration"
+        switch.write_text("#!/bin/sh\nif [ \"${ACTIVATE_FAIL:-0}\" = 1 ]; then exit 1; fi\nexit 0\n")
+        switch.chmod(0o755)
+        self.state = self.tmp / "state"
+        (self.state).mkdir()
+        (self.state / ".upgrade-verified").touch()
+
+        script = NIX_FILE.read_text()
+        body = script.split("script = ''", 1)[1].split("'';", 1)[0]
+        body = re.sub(r"\$\{pkgs\.[^}]+\}/bin/", "", body)
+        replacements = {
+            "/run/lock/commonscomputing-update.lock": self.tmp / "update.lock",
+            "/var/lib/knot/.upgrade-verified": self.state / ".upgrade-verified",
+            "/nix/var/nix/profiles/system": self.tmp / "profile",
+        }
+        for old, new in replacements.items():
+            body = body.replace(old, str(new))
+        self.script = self.tmp / "auto-update.sh"
+        self.script.write_text("#!/usr/bin/env bash\n" + body)
+        self.script.chmod(0o755)
+        self.env = os.environ.copy()
+        self.env.update({"PATH": str(self.bin) + ":" + self.env["PATH"], "FAKE_LOG": str(self.log), "UNKNOWN_LOG": str(self.unknown), "TMP_ROOT": str(self.tmp), "TOPLEVEL": str(self.toplevel)})
+
+    def tearDown(self):
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def run_update(self, **flags):
+        env = self.env.copy()
+        env.update({k: str(v) for k, v in flags.items()})
+        return subprocess.run([BASH, str(self.script)], cwd=self.repo, env=env, text=True, capture_output=True)
+
+    def commands(self):
+        return self.log.read_text().splitlines() if self.log.exists() else []
+
+    def assert_no_unknown(self):
+        self.assertFalse(self.unknown.exists(), self.unknown.read_text() if self.unknown.exists() else "")
+
+    def test_build_failure_restores_post_pull_lock_without_activation_or_commit(self):
+        result = self.run_update(BUILD_FAIL=1)
+        self.assertNotEqual(result.returncode, 0)
+        self.assertEqual((self.repo / "flake.lock").read_text(), "pulled lock\n")
+        commands = self.commands()
+        self.assertFalse(any(x.startswith("nix-env") for x in commands))
+        self.assertFalse(any(x.startswith("git commit") for x in commands))
+        self.assert_no_unknown()
+
+    def test_activation_failure_pauses_updates_without_rollback_or_publish(self):
+        result = self.run_update(ACTIVATE_FAIL=1)
+        self.assertNotEqual(result.returncode, 0)
+        commands = self.commands()
+        self.assertIn("systemctl stop auto-update.timer", commands)
+        self.assertEqual(sum(x.startswith("nix-env") for x in commands), 1)
+        self.assertFalse(any(x.startswith("git commit") or x == "git push" for x in commands))
+        self.assertTrue((self.state / ".upgrade-verified").exists() is False)
+        self.assert_no_unknown()
+
+    def test_health_failure_pauses_updates_without_rollback_or_publish(self):
+        result = self.run_update(HEALTH_FAIL=1)
+        self.assertNotEqual(result.returncode, 0)
+        commands = self.commands()
+        self.assertIn("systemctl stop auto-update.timer", commands)
+        self.assertEqual(sum(x.startswith("nix-env") for x in commands), 1)
+        self.assertFalse(any(x.startswith("git commit") or x == "git push" for x in commands))
+        self.assert_no_unknown()
+
+    def test_happy_path_commits_and_pushes_after_health(self):
+        result = self.run_update()
+        self.assertEqual(result.returncode, 0, result.stderr)
+        commands = self.commands()
+        health = max(i for i, x in enumerate(commands) if x.startswith("curl "))
+        commit = next(i for i, x in enumerate(commands) if x.startswith("git commit"))
+        push = next(i for i, x in enumerate(commands) if x == "git push")
+        self.assertLess(health, commit)
+        self.assertLess(commit, push)
+        self.assert_no_unknown()
+
+    def test_publication_failure_retains_deployed_lock_without_rollback(self):
+        result = self.run_update(PUBLISH_FAIL=1)
+        self.assertNotEqual(result.returncode, 0)
+        self.assertEqual((self.repo / "flake.lock").read_text(), "committed lock\n")
+        commands = self.commands()
+        self.assertFalse(any(x.startswith("nix-env") and i > commands.index("git push") for i, x in enumerate(commands)))
+        self.assert_no_unknown()
+
+    def test_staged_and_nix_edits_refuse_before_pull(self):
+        for flag in ("STAGED_CHANGES", "NIX_CHANGES"):
+            with self.subTest(flag=flag):
+                result = self.run_update(**{flag: 1})
+                self.assertNotEqual(result.returncode, 0)
+                self.assertFalse(any(x.startswith("git pull") for x in self.commands()))
+                self.assert_no_unknown()
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/tests/test_upgrade_knot.py b/tests/test_upgrade_knot.py
new file mode 100644
index 0000000..bfea9f4
--- /dev/null
+++ b/tests/test_upgrade_knot.py
@@ -0,0 +1,367 @@
+import hashlib
+import os
+import shutil
+import stat
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+SCRIPT = ROOT / "scripts" / "upgrade-knot.sh"
+BASH = shutil.which("bash") or "/run/current-system/sw/bin/bash"
+
+
+DISPATCHER = r'''#!/usr/bin/env python3
+import hashlib
+import os
+import shutil
+import sys
+from pathlib import Path
+
+name = Path(sys.argv[0]).name
+args = sys.argv[1:]
+log = Path(os.environ["FAKE_LOG"])
+with log.open("a") as f:
+    f.write(name + " " + " ".join(args) + "\n")
+if os.environ.get("KNOT_MASTER_KEY") and name not in {"knot-migrate", "knot-server"}:
+    Path(os.environ["KEY_LEAK_LOG"]).write_text(name + "\n")
+
+def fail(flag):
+    return os.environ.get(flag) == "1"
+
+if name == "id":
+    print("0")
+elif name == "hostname":
+    print("asusmininix")
+elif name == "flock":
+    pass
+elif name == "systemctl":
+    if args[:2] == ["is-active", "--quiet"]:
+        unit = args[2]
+        if unit == "auto-update.service":
+            raise SystemExit(0 if fail("AUTO_UPDATE_ACTIVE") else 1)
+        if unit == "knot-rs.service" and fail("RUST_INACTIVE"):
+            raise SystemExit(1)
+        raise SystemExit(0)
+    if args[:1] == ["start"] and "auto-update.timer" in args and fail("TIMER_START_FAIL"):
+        raise SystemExit(1)
+elif name == "nix-store":
+    if args[:1] == ["--check-validity"]:
+        raise SystemExit(1 if fail("INVALID_STORE") else 0)
+    if args[:1] == ["--import"]:
+        sys.stdin.buffer.read()
+elif name == "sha256sum":
+    data = Path(args[0]).read_bytes()
+    print(hashlib.sha256(data).hexdigest() + "  " + args[0])
+elif name == "gzip":
+    try:
+        os.write(1, b"fixture export")
+    except OSError:
+        pass
+elif name == "grep":
+    path = Path(args[-1])
+    if not path.is_file() or args[1] not in path.read_text():
+        raise SystemExit(1)
+    if fail("LIVE_GATE_FAIL") and path.name in {"auto-update.timer", "auto-update.service"}:
+        raise SystemExit(1)
+elif name == "git":
+    if fail("GIT_PREFLIGHT_FAIL"):
+        raise SystemExit(1)
+elif name == "curl":
+    if "--write-out" in args:
+        print("101")
+    elif "requestCrawl" in " ".join(args) and fail("CRAWL_FAIL"):
+        raise SystemExit(1)
+    elif "5555" in " ".join(args) and fail("HEALTH_FAIL"):
+        print("{}")
+    else:
+        print("{}")
+elif name == "jq":
+    sys.stdin.read()
+    if fail("HEALTH_FAIL") and "version" in " ".join(args):
+        raise SystemExit(1)
+elif name == "ssh-keygen":
+    try:
+        os.write(1, b"256 TESTKEY fixture\n")
+    except OSError:
+        pass
+    os._exit(0)
+elif name == "ssh-keyscan":
+    try:
+        os.write(1, b"127.0.0.1 ssh-ed25519 TESTKEY\n")
+    except OSError:
+        pass
+    os._exit(0)
+elif name == "nix-env":
+    pass
+elif name == "agenix":
+    value = (b"KNOT_MASTER_KEY=" + (b"B" * 43 + b"=\n" if not fail("BAD_KEY") else b"bad\n"))
+    sys.stdout.buffer.write(value)
+    target = os.environ.get("AGENIX_TARGET")
+    if target:
+        Path(target).write_bytes(value)
+elif name == "knot-migrate":
+    if os.environ.get("KNOT_MASTER_KEY") != "B" * 43 + "=":
+        raise SystemExit(1)
+    if "--dry-run" in args:
+        raise SystemExit(1 if fail("DRYRUN_FAIL") else 0)
+    target = Path(args[args.index("--target") + 1])
+    target.mkdir(parents=True, exist_ok=True)
+    (target / "ssh_host_key").write_text("fixture-key\n")
+    (target / "sealed-keys").write_text("fixture-sealed\n")
+    (target / "repo-signing-keys.json").write_text("{}\n")
+    if fail("MIGRATION_FAIL"):
+        raise SystemExit(1)
+elif name == "sqlite3" or name == "tar" or name == "chown" or name == "chmod":
+    pass
+elif name == "cmp":
+    if Path(args[-2]).read_bytes() != Path(args[-1]).read_bytes():
+        raise SystemExit(1)
+elif name == "mktemp":
+    template = args[-1]
+    path = Path(template.replace("XXXXXX", "fixture"))
+    path.parent.mkdir(parents=True, exist_ok=True)
+    if "-d" in args:
+        path.mkdir(exist_ok=True)
+    else:
+        path.touch()
+    print(path)
+elif name == "readlink":
+    print(os.environ["OLD_SYSTEM"])
+elif name == "rm":
+    for arg in args:
+        if arg.startswith("-"):
+            continue
+        p = Path(arg)
+        if p.is_dir():
+            shutil.rmtree(p, ignore_errors=True)
+        else:
+            p.unlink(missing_ok=True)
+elif name == "rmdir":
+    Path(args[-1]).rmdir()
+elif name == "mv":
+    shutil.move(args[-2], args[-1])
+elif name == "touch":
+    Path(args[-1]).touch()
+else:
+    Path(os.environ["UNKNOWN_LOG"]).open("a").write(name + "\n")
+    raise SystemExit(127)
+'''
+
+
+class UpgradeKnotTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = Path(tempfile.mkdtemp(prefix="upgrade-knot-test-"))
+        self.fakebin = self.tmp / "bin"
+        self.fakebin.mkdir()
+        self.log = self.tmp / "commands.log"
+        self.unknown = self.tmp / "unknown.log"
+        self.key_scope = self.tmp / "key-scope.log"
+        dispatcher = self.fakebin / "fake-command.py"
+        dispatcher.write_text(DISPATCHER)
+        dispatcher.chmod(dispatcher.stat().st_mode | stat.S_IXUSR)
+        names = "id hostname flock systemctl nix-store sha256sum gzip grep git curl jq ssh-keygen ssh-keyscan nix-env agenix knot-migrate sqlite3 tar chown chmod cmp mktemp readlink rm rmdir mv touch".split()
+        for name in names:
+            (self.fakebin / name).symlink_to(dispatcher)
+
+        self.stage = self.tmp / "stage"
+        self.stage.mkdir()
+        self.store = self.tmp / "nix" / "store"
+        self.system = self.store / "fixture-nixos-system-asusmininix-test"
+        self.tools = self.store / "fixture-knot-migration-tools"
+        (self.system / "bin").mkdir(parents=True)
+        (self.system / "sw" / "bin").mkdir(parents=True)
+        (self.system / "etc" / "systemd" / "system").mkdir(parents=True)
+        (self.system / "etc" / "knot").mkdir()
+        (self.system / "sw" / "bin" / "knot-server").write_text(
+            f"#!/bin/sh\ntest \"$KNOT_MASTER_KEY\" = \"{'B' * 43}=\" || exit 1\nprintf validated > {self.key_scope}\n"
+        )
+        (self.system / "sw" / "bin" / "knot-server").chmod(0o755)
+        switch = self.system / "bin" / "switch-to-configuration"
+        switch.write_text("#!/bin/sh\nif [ \"${FAIL_ACTIVATION:-0}\" = 1 ]; then exit 1; fi\nexit 0\n")
+        switch.chmod(0o755)
+        gate = f"ConditionPathExists={self.state if hasattr(self, 'state') else self.tmp / 'state'}/.upgrade-verified\n"
+        (self.system / "etc" / "systemd" / "system" / "auto-update.timer").write_text(gate)
+        (self.system / "etc" / "systemd" / "system" / "auto-update.service").write_text(gate)
+        (self.tools / "bin").mkdir(parents=True)
+        (self.stage / "system-path").write_text(str(self.system) + "\n")
+        (self.stage / "tools-path").write_text(str(self.tools) + "\n")
+        export = self.stage / "upgrade.export.gz"
+        export.write_bytes(b"fixture export")
+        (self.stage / "upgrade.export.gz.sha256").write_text(hashlib.sha256(export.read_bytes()).hexdigest() + "  upgrade.export.gz\n")
+
+        self.repo = self.tmp / "repo"
+        (self.repo / "secrets").mkdir(parents=True)
+        (self.repo / "secrets" / "knot-env.age").write_text("not a secret used by fixture\n")
+        self.source = self.tmp / "git"
+        self.source.mkdir()
+        (self.source / "knotserver.db").write_text("fixture db\n")
+        self.ssh = self.tmp / "ssh_host_ed25519_key"
+        self.ssh.write_text("fixture private key\n")
+        (self.ssh.parent / (self.ssh.name + ".pub")).write_text("fixture public key\n")
+        self.state = self.tmp / "state"
+        self.agenix_target = self.tmp / "agenix" / "knot-env"
+        self.agenix_target.parent.mkdir()
+        self.agenix_target.write_text("KNOT_MASTER_KEY=" + "B" * 43 + "=\n")
+        self.old = self.store / "old-system"
+        (self.old / "bin").mkdir(parents=True)
+        (self.old / "bin" / "switch-to-configuration").write_text("#!/bin/sh\nexit 0\n")
+        (self.old / "bin" / "switch-to-configuration").chmod(0o755)
+        (self.old / "etc" / "systemd" / "system").mkdir(parents=True)
+        (self.old / "etc" / "systemd" / "system" / "auto-update.timer").write_text(
+            f"ConditionPathExists={self.state}/.upgrade-verified\n"
+        )
+        (self.old / "etc" / "systemd" / "system" / "auto-update.service").write_text(
+            f"ConditionPathExists={self.state}/.upgrade-verified\n"
+        )
+        current = self.tmp / "current-system"
+        current.symlink_to(self.old)
+
+        replacements = {
+            "/nix/store": self.store,
+            "/home/anish/tangled-v1.20.0-preflight": self.stage,
+            "/etc/commonscomputing-nix": self.repo,
+            "/var/lib/knot": self.state,
+            "/home/git": self.source,
+            "/var/backups": self.tmp / "backups",
+            "/nix/var/nix/profiles/system": self.tmp / "profile",
+            "/run/current-system": current,
+            "/etc/ssh/ssh_host_ed25519_key": self.ssh,
+            "/run/lock/commonscomputing-update.lock": self.tmp / "lock",
+            "/run/knot-upgrade.XXXXXX": self.tmp / "runtime.XXXXXX",
+            "/var/lib/knot-upgrade-system": self.tmp / "system-root",
+            "/var/lib/knot-upgrade-tools": self.tmp / "tools-root",
+            "/run/agenix/knot-env": self.agenix_target,
+        }
+        text = SCRIPT.read_text()
+        for old, new in replacements.items():
+            text = text.replace(old, str(new))
+        self.script = self.tmp / "upgrade-knot.sh"
+        self.script.write_text(text)
+        self.env = os.environ.copy()
+        self.key_leak = self.tmp / "key-leak.log"
+        self.env.update({"PATH": str(self.fakebin) + ":" + self.env["PATH"], "FAKE_LOG": str(self.log), "UNKNOWN_LOG": str(self.unknown), "KEY_LEAK_LOG": str(self.key_leak), "OLD_SYSTEM": str(self.old), "AGENIX_TARGET": str(self.agenix_target)})
+
+    def tearDown(self):
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def run_upgrade(self, *args, **flags):
+        env = self.env.copy()
+        env.update({k: str(v) for k, v in flags.items()})
+        return subprocess.run([BASH, str(self.script), *args], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
+
+    def commands(self):
+        return self.log.read_text().splitlines() if self.log.exists() else []
+
+    def assert_no_unknown_tools(self):
+        self.assertFalse(self.unknown.exists(), self.unknown.read_text() if self.unknown.exists() else "")
+
+    def test_happy_path_order_and_secret_cleanup(self):
+        result = self.run_upgrade("--yes")
+        self.assertEqual(result.returncode, 0, result.stderr)
+        commands = self.commands()
+        self.assertLess(next(i for i, x in enumerate(commands) if x.startswith("systemctl stop auto-update.timer")), next(i for i, x in enumerate(commands) if x.startswith("knot-migrate ") and "--dry-run" not in x))
+        timer_start = next(i for i, x in enumerate(commands) if x == "systemctl start auto-update.timer")
+        health_end = max(i for i, x in enumerate(commands) if x.startswith("curl ") and i < timer_start)
+        self.assertGreater(timer_start, health_end)
+        self.assertTrue((self.state / ".migration-complete").exists())
+        self.assertFalse(self.tmp.joinpath("runtime.fixture", "knot.env").exists())
+        self.assertFalse(self.key_leak.exists())
+        self.assertTrue(self.key_scope.exists())
+        self.assert_no_unknown_tools()
+
+    def test_dryrun_failure_does_not_stop_go_or_timer(self):
+        result = self.run_upgrade("--yes", DRYRUN_FAIL=1)
+        self.assertNotEqual(result.returncode, 0)
+        commands = self.commands()
+        self.assertFalse(any(x == "systemctl stop knot.service" for x in commands))
+        self.assertFalse(any(x == "systemctl stop auto-update.timer" for x in commands))
+        self.assertFalse(self.tmp.joinpath("runtime.fixture", "knot.env").exists())
+        self.assert_no_unknown_tools()
+
+    def test_migration_failure_quarantines_target_and_restarts_go(self):
+        result = self.run_upgrade("--yes", MIGRATION_FAIL=1)
+        self.assertNotEqual(result.returncode, 0)
+        self.assertTrue((self.tmp / "backups" / "knot-cutover.fixture" / "failed-target").exists())
+        self.assertFalse(self.tmp.joinpath("runtime.fixture", "knot.env").exists())
+        commands = self.commands()
+        self.assertIn("systemctl start knot.service", commands)
+        self.assert_no_unknown_tools()
+
+    def test_activation_failure_keeps_new_state_and_stops_services(self):
+        result = self.run_upgrade("--yes", FAIL_ACTIVATION=1)
+        self.assertNotEqual(result.returncode, 0)
+        commands = self.commands()
+        sets = [x for x in commands if x.startswith("nix-env --profile")]
+        self.assertEqual(len(sets), 1)
+        self.assertFalse(any(x == "systemctl start knot.service" for x in commands))
+        self.assertIn("systemctl stop auto-update.timer", commands)
+        self.assertTrue(self.state.exists())
+        self.assertFalse((self.state / ".migration-complete").exists())
+        self.assert_no_unknown_tools()
+
+    def test_health_failure_keeps_new_state_and_stops_services(self):
+        result = self.run_upgrade("--yes", HEALTH_FAIL=1)
+        self.assertNotEqual(result.returncode, 0)
+        commands = self.commands()
+        self.assertEqual(sum(x.startswith("nix-env --profile") for x in commands), 1)
+        self.assertNotIn("systemctl start knot.service", commands)
+        self.assertIn("systemctl stop knot-rs.service", commands)
+        self.assertTrue(self.state.exists())
+        self.assert_no_unknown_tools()
+
+    def test_completed_marker_skips_migration(self):
+        self.state.mkdir()
+        (self.state / ".migration-complete").touch()
+        result = self.run_upgrade()
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertFalse(any(x.startswith("knot-migrate") for x in self.commands()))
+        self.assertTrue(any(x == "systemctl start auto-update.timer" for x in self.commands()))
+        self.assert_no_unknown_tools()
+
+    def test_completed_marker_rejects_mismatched_live_gate_without_touching_state(self):
+        self.state.mkdir()
+        (self.state / ".migration-complete").touch()
+        before = (self.state / ".upgrade-verified").exists()
+        result = self.run_upgrade(LIVE_GATE_FAIL=1)
+        self.assertNotEqual(result.returncode, 0)
+        self.assertEqual((self.state / ".upgrade-verified").exists(), before)
+        self.assertFalse(any(x == "systemctl start auto-update.timer" for x in self.commands()))
+        self.assert_no_unknown_tools()
+
+    def test_invalid_key_fails_before_downtime(self):
+        result = self.run_upgrade("--yes", BAD_KEY=1)
+        self.assertNotEqual(result.returncode, 0)
+        commands = self.commands()
+        self.assertFalse(any(x == "systemctl stop knot.service" for x in commands))
+        self.assertFalse(any(x == "systemctl stop auto-update.timer" for x in commands))
+        self.assert_no_unknown_tools()
+
+    def test_missing_staged_service_gate_fails_before_downtime(self):
+        service = self.system / "etc" / "systemd" / "system" / "auto-update.service"
+        service.write_text("[Unit]\n")
+        result = self.run_upgrade("--yes")
+        self.assertNotEqual(result.returncode, 0)
+        self.assertFalse(any(x == "systemctl stop knot.service" for x in self.commands()))
+        self.assertFalse(self.state.exists())
+        self.assert_no_unknown_tools()
+
+    def test_git_preflight_failure_has_no_downtime(self):
+        result = self.run_upgrade("--yes", GIT_PREFLIGHT_FAIL=1)
+        self.assertNotEqual(result.returncode, 0)
+        commands = self.commands()
+        self.assertFalse(any(x == "systemctl stop knot.service" for x in commands))
+        self.assertFalse(any(x == "systemctl stop auto-update.timer" for x in commands))
+        self.assert_no_unknown_tools()
+
+    def test_invalid_store_imports_export(self):
+        result = self.run_upgrade("--yes", INVALID_STORE=1)
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertTrue(any(x.startswith("nix-store --import") for x in self.commands()))
+        self.assert_no_unknown_tools()
+
+
+if __name__ == "__main__":
+    unittest.main()