aynish/commoncomputing-nix
1
1 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

asusmini: script Tangled cutover and resume checked updates

Browse Source
This commit is contained in:
Anish Lakhwara committed 2026-11-01 15:40:12 -07:00
1 parent 0138dd376c
commit e3d4ae696d
8 files changed
+987 -314
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+6 -6
View File
@@ -84,16 +84,16 @@ sudo nixos-rebuild switch --flake .#asusmini
#### Auto-Update
The timer is currently disabled for the Tangled migration. Tangled is pinned
to v1.20.0 and uses its own locked dependencies so the tested binaries can
be copied to the server without recompilation. Re-enable updates deliberately
only after cutover and verification.
The timer is gated until `scripts/upgrade-knot.sh` completes the Tangled
migration and smoke tests. It then resumes weekly updates, including Tangled.
The initial lock selects tested v1.20.0 packages with their own dependencies
so the cutover reuses copied binaries without recompilation.
A systemd timer runs weekly to:
1. Pull latest git changes
2. Run `nix flake update`
3. Commit and push flake.lock changes
4. Run `nixos-rebuild switch` with automatic rollback on failure
3. Build, activate, and health-check the new configuration
4. Commit and push flake.lock only after successful validation
**Setup required:**
```bash
TANGLED-MIGRATION.md
+94 -271
View File
@@ -1,306 +1,129 @@
# Tangled v1.20.0 Preflight
# Tangled Upgrade
Status: local and server smoke tests passed. The operator completed the
real-data dry-run successfully: 23 repositories, 16 members, 12 collaborator
grants, zero skips, and no ACL/table disagreement. The master key has an
off-host backup. Nothing has been deployed or migrated.
The initial cutover uses tested v1.20.0 packages. `flake.lock` still selects
commit `2096d06d938e680297adeadd71b8320c9f3491d4`; the input URL tracks upstream
again so subsequent weekly updates can advance Tangled. Upstream's own nixpkgs
is retained to reuse the built release binaries.
The operator created `secrets/knot-env.age` on asusmini. The encrypted file
and its recipient rule have been synchronized into this checkout; the key
has not been decrypted here. The new configuration wires it into agenix at
`/run/agenix/knot-env`, root-owned with mode `0400`.
## Run The Script
## Release And Build Outputs
Verified upstream tag: `v1.20.0`, commit
`2096d06d938e680297adeadd71b8320c9f3491d4`.
Built on the workstation using the release's own locked inputs, not this
repository's `nixpkgs` override:
| Package | Exact Store Path |
| --- | --- |
| Rust knot | `/nix/store/7wv0kkflzpihyzm9k7mahxdci97r8k0h-knot-server-1.20.0` |
| Migrator | `/nix/store/h3d5c7wad0y6nmlkllrjq95g15j6zgpw-knot-migrate-1.20.0` |
| Spindle | `/nix/store/id8ma5qaj5q6msfjw2llppsz7w46jy9i-spindle-1.20.0` |
These outputs have local GC roots under `/tmp/opencode/`. Do not delete them
until the outputs have durable roots on the server.
## Completed Checks
- All three packages built successfully.
- Rust knot configuration validation and HTTP health passed at v1.20.0.
- Knot describe/contact, DID document, and SSH host-key matching passed.
- Unauthenticated repository creation was rejected with 401/403.
- Knot signing identity and SSH identity persisted across restart.
- Spindle started with disposable state; health, DID, and contact passed.
- Synthetic Go-database migration passed both dry-run and copy migration.
- Synthetic migration preserved the source database and bare repository.
- A temporary NixOS configuration evaluated with Rust knot and spindle at
1.20.0, Go knot disabled, and Rust `wantedBy = []`.
- Uploaded the closure archive and test scripts to the server and verified
that the archive checksum matches the local copy.
The upstream v1.20.0 spindle source hardcodes `wireVersion = "v1.16.1"` in
`spindle/xrpc/org_tangled_spindle_describe_spindle.go`. Its describe endpoint
is not a reliable indicator of the package release; verify the store path.
These are startup and migration smoke tests, not a full CI workflow test or
an authenticated end-to-end clone/push test against migrated production data.
## Server Artifacts
SSH: `anish@pds.commonscomputer.com`.
Directory: `/home/anish/tangled-v1.20.0-preflight`.
Archive: `tangled-v1.20.0-closure.export.gz` (137 MiB compressed), containing
the package closures plus Python/OpenSSL for the test harness.
SHA-256:
`aa50f7edf3f2e4721f27abc86e4cdd6476d66eeed3dd013ddc5252dcc228a4d3`.
The plain `nix copy --to ssh-ng://anish@pds.commonscomputer.com` attempt was
rejected because the build outputs are unsigned and only root is trusted.
No signature policy or trusted-user configuration was changed.
## Network Plan
Administrative OpenSSH remains on port **22**. Rust knot uses **2223**;
the upstream module opens that port in the NixOS firewall. Public port 2222
already belongs to the separate Gitea service and must not be repurposed.
The Ansible Caddy template already proxies all requests, including WebSocket
upgrades, from `knot.commonscomputer.com` to `192.168.1.249:5555`. No path-specific
`/events` rule needs replacing. Knot trusts forwarded addresses only from
the Caddy LXC, `192.168.1.11/32`.
WAN forwarding is not managed by either repository. Before cutover, either
arrange external TCP 2223 -> `192.168.1.249:2223`, or use the existing Tailscale
address `100.122.40.27:2223`. SSH Git clients must specify the new port. The
old port-22 Git SSH URLs will not work after removing the Go module; do not
redirect public port 22 if that is your only administrative access path.
## Stage The Built System (No Downtime)
The system build and migration-tools bundle are exported separately from
the earlier smoke archive. The staging directory contains `system-path`,
`tools-path`, `upgrade.export.gz`, and `upgrade.export.gz.sha256`.
As root on asusmini, before stopping anything:
Pull the latest configuration on asusmini, then run:
```sh
sudo -i
sudo bash /etc/commonscomputing-nix/scripts/upgrade-knot.sh
```
In that root Bash session:
The script rehearses first, prints the migration report, and asks you to type
`MIGRATE` before stopping Go knot. Keep another administrative SSH session open,
back up the master key off-host, and ask users to pause writes during cutover.
`--yes` skips the approval prompt. Running the script authorizes migration and
deployment, not merely a smoke test.
Root's Git identity must be able to fetch and push to `origin`. The script
checks `ls-remote` and `push --dry-run` before downtime. Origin is the separate
Gitea service on port 2222, not this knot.
If encrypted-secret edits prevent a fast-forward pull, preserve only those
edits first; leave unrelated DNS changes alone:
```sh
set -euo pipefail
test "$EUID" -eq 0
cd /home/anish/tangled-v1.20.0-preflight
sha256sum --check upgrade.export.gz.sha256
gzip -dc upgrade.export.gz | nix-store --import --option require-sigs false
system=$(cat system-path)
tools=$(cat tools-path)
nix-store --realise "$system" --add-root /var/lib/knot-upgrade-system
nix-store --realise "$tools" --add-root /var/lib/knot-upgrade-tools
test -x "$system/bin/switch-to-configuration"
test -x "$tools/bin/knot-migrate"
```
The archive contains only paths missing from the server at staging time.
Do not garbage-collect the server store between staging and activation.
The signature exception applies to this one root-authorized import, not to
persistent Nix configuration. Importing and rooting paths do not activate them.
The encrypted secret and recipient rule are already committed upstream.
The server currently has the same edits locally, including a staged new file.
Preserve those edits in a path-limited stash before pulling; do not create
another secret commit and diverge the server branch again:
```sh
git -C /etc/commonscomputing-nix stash push \
sudo git -C /etc/commonscomputing-nix stash push \
-m 'pre-cutover encrypted secret backup' -- secrets/knot-env.age secrets/secrets.nix
git -C /etc/commonscomputing-nix pull --ff-only
sha256sum /etc/commonscomputing-nix/secrets/knot-env.age
sudo git -C /etc/commonscomputing-nix pull --ff-only
```
The encrypted file's SHA-256 should remain
The encrypted secret is already committed; do not pop an identical stash over
it. Its SHA-256 is
`b26f36d098a337a98217aa4e73d2f39eb4e8864337dfcface937093f5557f0ac`.
Keep the stash until cutover is verified; do not pop it over the incoming
identical secret. The path-limited stash leaves unrelated DNS edits alone.
Activation below uses the exact built path, not a new flake evaluation/update.
## Decrypt And Rehearse With The Persistent Key
## Happy Path
Continue in the same root Bash session. Do not enable shell tracing (`set -x`).
1. Lock out concurrent updates and validate the host, staging files, and Git access.
2. Import the prebuilt system/tools if missing, after checking the archive checksum.
3. Decrypt the persistent key into root-only runtime storage and rehearse.
4. Obtain downtime approval; stop the timer and Go knot.
5. Back up the database, repositories, SSH identity, and old NixOS generation.
6. Copy-migrate repositories/access grants while preserving all original source data.
7. Validate the imported identity/configuration and install the prebuilt system.
8. Set ownership, check agenix supplied the same key, and start Rust knot.
9. Check local/public APIs, service state, SSH fingerprint, and Caddy WebSocket upgrade.
10. Mark verification complete, start the weekly timer, and request an indexer crawl.
```sh
export PATH="$tools/bin:$PATH"
umask 077
runtime=$(mktemp -d /run/knot-migration.XXXXXX)
trap 'rm -f "$runtime/knot.env"; rmdir "$runtime"; unset KNOT_MASTER_KEY' EXIT
agenix -d /etc/commonscomputing-nix/secrets/knot-env.age \
-i /etc/ssh/ssh_host_ed25519_key > "$runtime/knot.env"
chmod 0400 "$runtime/knot.env"
set -a
. "$runtime/knot.env"
set +a
test -n "${KNOT_MASTER_KEY:-}"
Source data stays in `/home/git`; Rust state lives in `/var/lib/knot`. The
previous rehearsal found 23 repositories, 16 members, 12 collaborator grants,
zero skips, and no ACL/table disagreement. Check the fresh report for changes.
migration_args=(
--source-db /home/git/knotserver.db
--source-repos /home/git
--hostname knot.commonscomputer.com
--host-key /etc/ssh/ssh_host_ed25519_key
--plc-url https://plc.directory
--contact-email pds@email.commonscomputer.com
--master-key-env KNOT_MASTER_KEY
--object-format sha1
--target /var/lib/knot
)
knot-migrate "${migration_args[@]}" --dry-run
```
Backups are root-only under `/var/backups/knot-cutover.*`. They contain private
key material, including `repo-signing-keys.json`; back them up encrypted off-host.
The master key is not printed and is scoped to migration/config validation.
The runtime plaintext file is removed on ordinary success/failure and signals;
SIGKILL may leave it root-readable in `/run` until cleanup or reboot.
Check the expected 23 repositories, 16 members, 12 grants, zero skips, and
no ownership drift. SQLite may create WAL/SHM files for read-only access.
This uses the real backed-up key, not the disposable preflight script's key.
If it fails, stop here; production is still running.
## Networking
## Cut Over (Downtime Starts Here)
Administrative SSH stays on **22**. Rust knot Git SSH uses **2223**, opened by
the NixOS module. Use `100.122.40.27:2223` over Tailscale, or arrange external
TCP 2223 forwarding to `192.168.1.249:2223`. Existing port-22 Git URLs need an
explicit port or client SSH configuration. Public 2222 belongs to Gitea and
must not be repurposed. The script does not change router forwarding or sshd.
Run only after explicitly approving downtime and confirming the SSH plan.
Keep a second administrative SSH connection open. Do not reboot midway.
Caddy proxies all paths and WebSocket upgrades to `192.168.1.249:5555`; Knot
trusts forwarded addresses only from `192.168.1.11/32`. The script checks
`/xrpc/com.atproto.sync.subscribeRepos` through Caddy. It cannot test an
authenticated user push without their credentials: after success, test a real
clone/push/delete of a temporary branch over port 2223.
```sh
test ! -e /var/lib/knot/.migration-complete
test ! -e /var/lib/knot/sealed-keys
old_system=$(readlink -f /run/current-system)
backup=/var/backups/knot-cutover-$(date +%Y%m%dT%H%M%S)
install -d -m 0700 "$backup"
printf '%s\n' "$old_system" > "$backup/old-system"
nix-store --realise "$old_system" --add-root "$backup/system"
## Automatic Updates
systemctl stop knot.service
sqlite3 'file:/home/git/knotserver.db?mode=ro' ".backup '$backup/knotserver.db'"
tar --acls --xattrs --numeric-owner -C / -czf "$backup/go-knot.tar.gz" \
home/git etc/ssh/ssh_host_ed25519_key etc/ssh/ssh_host_ed25519_key.pub
knot-migrate "${migration_args[@]}"
```
Both timer and service require `/var/lib/knot/.upgrade-verified`. The script
creates it only after successful smoke tests, then starts the timer. A separate
migration marker prevents Rust knot from initializing an empty identity.
Inspect the migration report before continuing. The default is **copy** mode;
never add `--consume-source`. Source repositories/database remain in place.
On any error, do not create the marker or start Rust knot. The old service
can be restarted while investigating (`systemctl start knot.service`). A
failed migration may leave partial target state; inspect it and quarantine
the entire target before a retry. Never blindly reuse it or delete the source.
Weekly updates fast-forward the configuration, update inputs, build, activate,
and health-check **before** committing/pushing `flake.lock`. Preexisting staged
changes, lockfile edits, or Nix configuration edits cause refusal. Unrelated
unstaged DNS edits remain untouched. Git runs noninteractively; update and
migration jobs share a process lock.
```sh
test -s /var/lib/knot/sealed-keys
test -f /var/lib/knot/sealed-keys
test -s /var/lib/knot/ssh_host_key
test -f /var/lib/knot/ssh_host_key
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
ssh-keygen -y -f /var/lib/knot/ssh_host_key | ssh-keygen -lf -
```
A failed build leaves the live generation unchanged and restores the script's
own lockfile changes. After activation is attempted, failure stops the application
services and timer and retains state/lockfile; old binaries are not automatically
run against possibly forward-migrated data. Publication failure retains the
healthy deployed generation and local changes; reconcile them with origin
before the next update.
The two fingerprints must match. Then validate the actual deployment config:
Future versions may compile on asusmini unless supplied by a binary cache.
Only the initial v1.20.0 cutover is guaranteed to reuse staged binaries.
```sh
/nix/store/7wv0kkflzpihyzm9k7mahxdci97r8k0h-knot-server-1.20.0/bin/knot-server \
--config "$system/etc/knot/config.toml" validate
if [ -f /var/lib/knot/repo-signing-keys.json ]; then
mv /var/lib/knot/repo-signing-keys.json "$backup/"
fi
```
## Failure And Retry
`repo-signing-keys.json` contains plaintext repository private keys. Keep the
backup directory root-only and back it up encrypted off-host. Do not commit it.
Before activation, failed migration targets are quarantined in the backup
directory and unchanged Go knot is restarted. After activation is attempted,
the script stops Rust knot/updates but preserves migrated state: blind rollback
could lose accepted writes or use an incompatible format. Investigate before
reopening writes. Runtime timer masks do not survive reboot; do not reboot an
old generation unattended during recovery.
Install the prebuilt system. The absent marker keeps Rust stopped while
activation creates the `knot` service user and decrypts the agenix secret:
A completed rerun verifies the live Rust service and resumes the gated timer;
it never migrates again or reinstalls an old staged generation. An incomplete
target without a completion marker is refused. Power loss or SIGKILL midway
requires inspection/recovery, not a blind rerun.
```sh
nix-env --profile /nix/var/nix/profiles/system --set "$system"
"$system/bin/switch-to-configuration" switch
chown -R knot:knot /var/lib/knot
chmod 0600 /var/lib/knot/ssh_host_key /var/lib/knot/sealed-keys
test -s /run/agenix/knot-env
cmp --silent "$runtime/knot.env" /run/agenix/knot-env
touch /var/lib/knot/.migration-complete
if ! systemctl start knot-rs.service || \
! curl --fail --silent --show-error --retry 10 --retry-connrefused \
--retry-delay 1 http://127.0.0.1:5555/xrpc/_health; then
rm -f /var/lib/knot/.migration-complete
systemctl stop knot-rs.service
exit 1
fi
rm -f "$runtime/knot.env"
rmdir "$runtime"
unset KNOT_MASTER_KEY
trap - EXIT
```
## Staging And Tests
Stop on any activation error rather than continuing to create the marker.
The marker allows the service to start normally on subsequent boots.
Auto-update remains disabled until deliberately re-enabled after verification.
`/home/anish/tangled-v1.20.0-preflight` contains `system-path`, `tools-path`,
`upgrade.export.gz`, and its checksum. The archive contains paths missing at
staging time; avoid garbage collection before import. Import alone activates
nothing. The root-authorized import accepts unsigned local products only for
that command, without changing global Nix trust settings.
## Final Checks
Local/server runtime and synthetic-migration smoke tests passed earlier. Mocked
orchestration tests cover success, preflight refusal, migration failure,
post-activation failure, secret cleanup, idempotence, and publication ordering.
They do not simulate a full boot or replace the on-host checks.
```sh
systemctl is-active knot-rs spindle sshd
systemctl is-active knot.service || true
journalctl -u knot-rs -n 80 --no-pager
curl --fail http://127.0.0.1:5555/xrpc/_health
curl --fail https://knot.commonscomputer.com/xrpc/_health
curl --fail https://knot.commonscomputer.com/xrpc/org.tangled.knot.describeKnot
curl --fail https://knot.commonscomputer.com/.well-known/did.json
ssh-keyscan -p 2223 -t ed25519 127.0.0.1 2>/dev/null | ssh-keygen -lf -
```
Expect v1.20.0, the original knot hostname/DID and contact, and the original
SSH fingerprint. From a client, clone a known migrated repository, push a
temporary branch using port 2223, then delete that branch. Check membership
and collaborator access in the web UI. Test WebSocket upgrade for
`/xrpc/com.atproto.sync.subscribeRepos` through Caddy; a successful HTTP health
request alone does not prove the event stream works.
Only after verification, request an indexer crawl:
```sh
curl --fail -X POST https://index.tangled.network/xrpc/com.atproto.sync.requestCrawl \
-H 'Content-Type: application/json' \
-d '{"hostname":"knot.commonscomputer.com"}'
```
## Roll Back If Needed
Before activation, just restart `knot.service`; the source was copied, not
moved. After activation, use the preserved generation:
```sh
systemctl stop knot-rs.service
rm -f /var/lib/knot/.migration-complete
old_system=$(cat "$backup/old-system")
nix-env --profile /nix/var/nix/profiles/system --set "$old_system"
"$old_system/bin/switch-to-configuration" switch
systemctl stop auto-update.timer
systemctl mask --runtime auto-update.timer
systemctl start knot.service
```
The old generation may re-enable the timer, hence the explicit stop and
runtime mask. The mask does not survive reboot: do not reboot the rolled-back
generation until a declarative configuration with the timer disabled is ready.
Retain all backups and migrated state. Writes made after cutover are not copied
back to the Go database by rollback; coordinate with users before reopening
writes. Never overwrite the source database unless recovery actually needs it.
At the last check, production knot and spindle were active, the auto-update
timer was inactive, and `/run/current-system` was unchanged:
`/nix/store/sv9r0h1xs6fsfjvbqv2vwslj42wm08v4-nixos-system-asusmininix-26.11.20261027.31ecbac`.
Upstream's v1.20.0 spindle still reports hardcoded wire version `v1.16.1` in
its describe endpoint; identify its actual package by the service store path.
## Upstream Guides
flake.lock
Generated
-3
View File
@@ -521,9 +521,6 @@
"url": "https://tangled.org/@tangled.org/core"
},
"original": {
"ref": "refs/tags/v1.20.0",
"rev": "2096d06d938e680297adeadd71b8320c9f3491d4",
"shallow": true,
"type": "git",
"url": "https://tangled.org/@tangled.org/core"
}
flake.nix
+5 -2
View File
@@ -4,8 +4,9 @@
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
tangled = {
# Keep the release's own package set so deployment reuses tested binaries.
url = "git+https://tangled.org/@tangled.org/core?ref=refs/tags/v1.20.0&rev=2096d06d938e680297adeadd71b8320c9f3491d4&shallow=1";
# flake.lock initially selects tested v1.20.0; weekly updates track upstream.
# Keep upstream's own nixpkgs so copied release binaries remain reusable.
url = "git+https://tangled.org/@tangled.org/core";
};
agenix = {
url = "github:ryantm/agenix";
@@ -57,6 +58,8 @@
pkgs.sqlite
pkgs.openssl
pkgs.git
pkgs.curl
pkgs.jq
];
};
hosts/asusmini/auto-update.nix
+86 -32
View File
@@ -1,57 +1,116 @@
{ config, pkgs, ... }:
{ pkgs, ... }:
{
systemd.services.auto-update = {
description = "Auto-update NixOS configuration";
path = with pkgs; [
coreutils
curl
git
jq
nix
openssh
util-linux
];
serviceConfig = {
Type = "oneshot";
User = "root";
WorkingDirectory = "/etc/commonscomputing-nix";
TimeoutStartSec = "6h";
};
restartIfChanged = false;
unitConfig.ConditionPathExists = "/var/lib/knot/.upgrade-verified";
unitConfig.X-StopOnRemoval = false;
environment = {
GIT_TERMINAL_PROMPT = "0";
GIT_SSH_COMMAND = "${pkgs.openssh}/bin/ssh -o BatchMode=yes -o ConnectTimeout=15";
};
script = ''
set -e
set -euo pipefail
exec 9>/run/lock/commonscomputing-update.lock
flock --nonblock 9 || exit 0
echo "Pulling latest changes..."
git pull
lock_snapshot="$(${pkgs.coreutils}/bin/mktemp)"
snapshot_ready=0
activated=0
validated=0
system_profile=/nix/var/nix/profiles/system
echo "Updating flake inputs..."
nix flake update
cleanup() {
status=$?
trap - EXIT
set +e
if [ "$status" -ne 0 ] && [ "$snapshot_ready" -eq 1 ] && [ "$activated" -eq 0 ]; then
${pkgs.coreutils}/bin/cp "$lock_snapshot" flake.lock
fi
if [ "$status" -ne 0 ] && [ "$activated" -eq 1 ] && [ "$validated" -eq 0 ]; then
# New binaries may already have migrated persistent state. Never
# blindly roll back their data format or discard accepted writes.
rm -f /var/lib/knot/.upgrade-verified
${pkgs.systemd}/bin/systemctl stop auto-update.timer
${pkgs.systemd}/bin/systemctl stop knot-rs.service spindle.service
echo "Activation/health failed: application services and updates stopped; inspect state and retained flake.lock" >&2
elif [ "$status" -ne 0 ] && [ "$validated" -eq 1 ]; then
echo "System is healthy, but Git publication failed; retain/reconcile the local lockfile or commit before retrying" >&2
fi
${pkgs.coreutils}/bin/rm -f "$lock_snapshot"
exit "$status"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
# Tangled stays at the release pinned in flake.nix. Upgrade it deliberately
# after migration/testing rather than tracking master automatically.
# Check if there are changes to commit
if ! git diff --quiet flake.lock; then
echo "Committing flake.lock updates..."
git add flake.lock
git commit -m "auto-update: flake inputs $(date -Iseconds)"
echo "Pushing changes..."
git push
else
echo "No flake.lock changes to commit"
fi
echo "Rebuilding system..."
if ! ${config.system.build.nixos-rebuild}/bin/nixos-rebuild switch --flake .#asusmini; then
echo "Build/switch failed, staying on current generation"
if ! git diff --cached --quiet; then
echo "Refusing to run with staged changes"
exit 1
fi
if ! git diff --quiet -- flake.lock; then
echo "Refusing to run with preexisting flake.lock changes"
exit 1
fi
if ! git diff --quiet -- '*.nix'; then
echo "Refusing to deploy uncommitted Nix configuration changes"
exit 1
fi
git pull --ff-only
git push --dry-run
${pkgs.coreutils}/bin/cp flake.lock "$lock_snapshot"
snapshot_ready=1
nix flake update
echo "Building NixOS toplevel..."
toplevel="$(${pkgs.nix}/bin/nix build .#nixosConfigurations.asusmini.config.system.build.toplevel --no-link --print-out-paths)"
echo "Activating new system..."
activated=1
${pkgs.nix}/bin/nix-env -p "$system_profile" --set "$toplevel"
"$toplevel/bin/switch-to-configuration" switch
${pkgs.systemd}/bin/systemctl is-active --quiet knot-rs
${pkgs.systemd}/bin/systemctl is-active --quiet spindle
${pkgs.systemd}/bin/systemctl is-active --quiet sshd
${pkgs.curl}/bin/curl --fail --silent --show-error --max-time 10 --retry 5 --retry-connrefused --retry-delay 1 http://127.0.0.1:5555/xrpc/_health | jq -e '.version | length > 0'
${pkgs.curl}/bin/curl --fail --silent --show-error --max-time 10 --retry 5 --retry-connrefused --retry-delay 1 http://127.0.0.1:6555/_health | jq -e '.status == "ok"'
${pkgs.curl}/bin/curl --fail --silent --show-error --max-time 15 https://knot.commonscomputer.com/xrpc/org.tangled.knot.describeKnot | jq -e '.did == "did:web:knot.commonscomputer.com" and .contact.email == "pds@email.commonscomputer.com"'
validated=1
snapshot_ready=0
if ! git diff --quiet -- flake.lock; then
git add flake.lock
git commit -m "auto-update: flake inputs $(${pkgs.coreutils}/bin/date -Iseconds)"
fi
git push
echo "Auto-update completed successfully"
'';
};
systemd.timers.auto-update = {
# Re-enable deliberately after the supervised Tangled migration is verified.
enable = false;
enable = true;
description = "Auto-update timer";
wantedBy = [ "timers.target" ];
timerConfig = {
@@ -59,11 +118,6 @@
Persistent = true; # Run on boot if missed
RandomizedDelaySec = "1h"; # Add some randomness
};
unitConfig.ConditionPathExists = "/var/lib/knot/.upgrade-verified";
};
# TODO: Set up SSH key for git push access
# Options:
# 1. Deploy key with write access to the repo
# 2. Generate SSH key on server and add to GitHub
# Command to generate: ssh-keygen -t ed25519 -f /root/.ssh/commons-nix-deploy -N ""
}
scripts/upgrade-knot.sh
+226
View File
@@ -0,0 +1,226 @@
#!/usr/bin/env bash
set -euo pipefail
unset KNOT_MASTER_KEY
# These paths are deliberately fixed to this host and its rehearsed migration.
stage=/home/anish/tangled-v1.20.0-preflight
repo=/etc/commonscomputing-nix
state=/var/lib/knot
source_dir=/home/git
backup_dir=/var/backups
profile=/nix/var/nix/profiles/system
current_system=/run/current-system
ssh_key=/etc/ssh/ssh_host_ed25519_key
lock_file=/run/lock/commonscomputing-update.lock
runtime_template=/run/knot-upgrade.XXXXXX
system_root=/var/lib/knot-upgrade-system
tools_root=/var/lib/knot-upgrade-tools
if [ "$(id -u)" != 0 ] || [ "$(hostname)" != asusmininix ]; then
printf 'Run this script as root on asusmininix.\n' >&2
exit 1
fi
if [ "$#" -gt 1 ] || { [ "$#" -eq 1 ] && [ "$1" != --yes ]; }; then
printf 'Usage: bash scripts/upgrade-knot.sh [--yes]\n' >&2
exit 1
fi
exec 9>"$lock_file"
flock --nonblock 9 || { printf 'Another update is running.\n' >&2; exit 1; }
if systemctl is-active --quiet auto-update.service; then
printf 'Wait for auto-update.service to finish before migrating.\n' >&2
exit 1
fi
system=$(<"$stage/system-path")
tools=$(<"$stage/tools-path")
case "$system" in /nix/store/*-nixos-system-asusmininix-*) ;; *) exit 1 ;; esac
case "$tools" in /nix/store/*-knot-migration-tools) ;; *) exit 1 ;; esac
if ! nix-store --check-validity "$system" "$tools"; then
read -r expected _ < "$stage/upgrade.export.gz.sha256"
actual=$(sha256sum "$stage/upgrade.export.gz")
test "${actual%% *}" = "$expected"
gzip -dc "$stage/upgrade.export.gz" | nix-store --import --option require-sigs false
fi
nix-store --realise "$system" --add-root "$system_root" >/dev/null
nix-store --realise "$tools" --add-root "$tools_root" >/dev/null
export PATH="$tools/bin:$PATH"
test -x "$system/bin/switch-to-configuration"
test -x "$system/sw/bin/knot-server"
# An older staged build intentionally disabled the timer; do not install it.
grep -Fq 'ConditionPathExists=/var/lib/knot/.upgrade-verified' \
"$system/etc/systemd/system/auto-update.timer"
grep -Fq 'ConditionPathExists=/var/lib/knot/.upgrade-verified' \
"$system/etc/systemd/system/auto-update.service"
smoke() {
for unit in knot-rs.service spindle.service sshd.service; do
systemctl is-active --quiet "$unit"
done
curl --fail --silent --show-error --max-time 10 --retry 5 --retry-connrefused \
--retry-delay 1 http://127.0.0.1:5555/xrpc/_health | jq -e '.version | length > 0'
curl --fail --silent --show-error --max-time 10 --retry 5 --retry-connrefused \
--retry-delay 1 http://127.0.0.1:6555/_health | jq -e '.status == "ok"'
curl --fail --silent --show-error --max-time 15 \
https://knot.commonscomputer.com/xrpc/org.tangled.knot.describeKnot \
| jq -e '.did == "did:web:knot.commonscomputer.com" and .contact.email == "pds@email.commonscomputer.com"'
read -r _ expected_key _ < <(ssh-keygen -lf "$ssh_key.pub")
scanned=$(ssh-keyscan -T 5 -p 2223 -t ed25519 127.0.0.1 2>/dev/null)
read -r _ actual_key _ < <(printf '%s\n' "$scanned" | ssh-keygen -lf -)
test "$expected_key" = "$actual_key"
# The upgraded endpoint is a long-lived WebSocket: timeout after the handshake.
status=$(curl --silent --max-time 3 --output /dev/null --write-out '%{http_code}' \
-H 'Connection: Upgrade' -H 'Upgrade: websocket' -H 'Sec-WebSocket-Version: 13' \
-H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
https://knot.commonscomputer.com/xrpc/com.atproto.sync.subscribeRepos || true)
test "$status" = 101
}
export GIT_TERMINAL_PROMPT=0
export GIT_SSH_COMMAND='ssh -o BatchMode=yes -o ConnectTimeout=15'
git -C "$repo" ls-remote --exit-code origin HEAD >/dev/null
git -C "$repo" push --dry-run origin HEAD:main
git -C "$repo" var GIT_AUTHOR_IDENT >/dev/null
if ! git -C "$repo" diff --cached --quiet || \
! git -C "$repo" diff --quiet -- '*.nix' flake.lock; then
printf 'Commit/stash staged changes and Nix/lockfile edits before cutover.\n' >&2
exit 1
fi
if [ -e "$state/.migration-complete" ]; then
printf 'Migration marker already exists; checking the live Rust service only.\n'
live_system=$(readlink -f "$current_system")
grep -Fq 'ConditionPathExists=/var/lib/knot/.upgrade-verified' \
"$live_system/etc/systemd/system/auto-update.timer"
grep -Fq 'ConditionPathExists=/var/lib/knot/.upgrade-verified' \
"$live_system/etc/systemd/system/auto-update.service"
smoke
touch "$state/.upgrade-verified"
systemctl unmask --runtime auto-update.timer
systemctl start auto-update.timer
systemctl is-active --quiet auto-update.timer
printf 'Already migrated; smoke tests passed and automatic updates are active.\n'
exit 0
fi
if [ -e "$state" ]; then
printf '%s exists without a completed migration. Inspect/quarantine it before retrying.\n' "$state" >&2
exit 1
fi
systemctl is-active --quiet knot.service
test -r "$source_dir/knotserver.db"
test -s "$ssh_key"
old_system=$(readlink -f "$current_system")
runtime=""
backup=""
downtime=0
activation_attempted=0
migration_started=0
verified=0
cleanup() {
result=$?
trap - EXIT INT TERM
set +e
if [ "$result" -ne 0 ] && [ "$downtime" -eq 1 ] && [ "$verified" -eq 0 ]; then
systemctl stop auto-update.timer
systemctl mask --runtime auto-update.timer
systemctl stop knot-rs.service
rm -f "$state/.migration-complete" "$state/.upgrade-verified"
if [ "$activation_attempted" -eq 1 ]; then
printf 'Activation was attempted: state may have changed. Updates/Rust knot are stopped.\n' >&2
printf 'Do NOT blindly roll back; inspect %s and %s before reopening writes.\n' "$state" "$backup" >&2
else
printf 'Migration failed before activation; restarting the unchanged Go knot.\n' >&2
if [ "$migration_started" -eq 1 ] && [ -e "$state" ]; then
mv "$state" "$backup/failed-target"
fi
if ! systemctl start knot.service; then
printf 'URGENT: Go knot could not be restarted; inspect %s.\n' "$backup" >&2
fi
fi
printf 'Backups retained in %s. Timer runtime mask does not survive reboot.\n' "$backup" >&2
fi
if [ -n "$runtime" ]; then
rm -f "$runtime/knot.env"
rmdir "$runtime"
fi
unset master_key
exit "$result"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
umask 077
runtime=$(mktemp -d "$runtime_template")
agenix -d "$repo/secrets/knot-env.age" -i "$ssh_key" > "$runtime/knot.env"
chmod 0400 "$runtime/knot.env"
master_key=""
export -n master_key
while IFS= read -r line || [ -n "$line" ]; do
case "$line" in
KNOT_MASTER_KEY=*) master_key=${line#KNOT_MASTER_KEY=} ;;
''|\#*) ;;
*) printf 'Unexpected entry in knot-env.age: expected only KNOT_MASTER_KEY.\n' >&2; exit 1 ;;
esac
done < "$runtime/knot.env"
if ! [[ "$master_key" =~ ^[A-Za-z0-9+/]{43}=$ ]]; then
printf 'KNOT_MASTER_KEY must be an unquoted base64 encoding of 32 random bytes.\n' >&2
exit 1
fi
migration_args=(
--source-db "$source_dir/knotserver.db" --source-repos "$source_dir"
--hostname knot.commonscomputer.com --host-key "$ssh_key"
--plc-url https://plc.directory --contact-email pds@email.commonscomputer.com
--master-key-env KNOT_MASTER_KEY --object-format sha1 --target "$state"
)
KNOT_MASTER_KEY="$master_key" knot-migrate "${migration_args[@]}" --dry-run
printf '\nThis will stop Go knot, back it up, COPY its data, and install the staged Rust system.\n'
printf 'Git SSH moves to port 2223; administrative SSH stays on 22. Use Tailscale or arrange WAN forwarding.\n'
printf 'Confirm the master key is backed up off-host and keep a second administrative session open.\n'
if [ "${1:-}" != --yes ]; then
read -r -p 'Type MIGRATE to approve downtime: ' approval
test "$approval" = MIGRATE
fi
backup=$(mktemp -d "$backup_dir/knot-cutover.XXXXXX")
printf '%s\n' "$old_system" > "$backup/old-system"
nix-store --realise "$old_system" --add-root "$backup/system" >/dev/null
systemctl stop auto-update.timer
downtime=1
systemctl stop knot.service
sqlite3 "file:$source_dir/knotserver.db?mode=ro" ".backup '$backup/knotserver.db'"
tar --acls --xattrs --numeric-owner -czf "$backup/go-knot.tar.gz" \
"$source_dir" "$ssh_key" "$ssh_key.pub"
migration_started=1
KNOT_MASTER_KEY="$master_key" knot-migrate "${migration_args[@]}"
test -f "$state/ssh_host_key" && test -s "$state/ssh_host_key"
test -f "$state/sealed-keys" && test -s "$state/sealed-keys"
read -r _ expected_key _ < <(ssh-keygen -lf "$ssh_key.pub")
read -r _ imported_key _ < <(ssh-keygen -lf "$state/ssh_host_key")
test "$expected_key" = "$imported_key"
KNOT_MASTER_KEY="$master_key" "$system/sw/bin/knot-server" --config "$system/etc/knot/config.toml" validate
if [ -f "$state/repo-signing-keys.json" ]; then
mv "$state/repo-signing-keys.json" "$backup/"
fi
activation_attempted=1
nix-env --profile "$profile" --set "$system"
"$system/bin/switch-to-configuration" switch
chown -R knot:knot "$state"
chmod 0600 "$state/ssh_host_key" "$state/sealed-keys"
cmp --silent "$runtime/knot.env" /run/agenix/knot-env
touch "$state/.migration-complete"
systemctl start knot-rs.service
smoke
verified=1
touch "$state/.upgrade-verified"
systemctl unmask --runtime auto-update.timer
systemctl start auto-update.timer
systemctl is-active --quiet auto-update.timer
if ! curl --fail --silent --show-error --max-time 15 -X POST \
https://index.tangled.network/xrpc/com.atproto.sync.requestCrawl \
-H 'Content-Type: application/json' -d '{"hostname":"knot.commonscomputer.com"}'; then
printf 'Warning: request the indexer crawl again later; the local migration passed.\n' >&2
fi
printf '\nSUCCESS: Rust knot and spindle are healthy; weekly automatic updates are active.\n'
printf 'Backups: %s (includes plaintext repository signing keys; back up encrypted off-host).\n' "$backup"
printf 'Next: test an authenticated clone/push over Git SSH port 2223.\n'
tests/test_auto_update.py
+203
View File
@@ -0,0 +1,203 @@
import re
import os
import shutil
import stat
import subprocess
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
NIX_FILE = ROOT / "hosts/asusmini/auto-update.nix"
BASH = shutil.which("bash") or "/run/current-system/sw/bin/bash"
FAKE = r'''#!/usr/bin/env python3
import os
import shutil
import sys
from pathlib import Path
name = Path(sys.argv[0]).name
args = sys.argv[1:]
log = Path(os.environ["FAKE_LOG"])
with log.open("a") as f:
f.write(name + " " + " ".join(args) + "\n")
def fail(flag):
return os.environ.get(flag) == "1"
if name == "flock":
pass
elif name == "mktemp":
p = Path(os.environ["TMP_ROOT"]) / "lock-snapshot"
p.touch()
print(p)
elif name == "cp":
shutil.copyfile(args[-2], args[-1])
elif name == "rm":
for arg in args:
if not arg.startswith("-"):
Path(arg).unlink(missing_ok=True)
elif name == "date":
print("fixture-date")
elif name == "git":
operation = " ".join(args)
if "diff --cached --quiet" in operation and fail("STAGED_CHANGES"):
raise SystemExit(1)
if "diff --quiet -- *.nix" in operation and fail("NIX_CHANGES"):
raise SystemExit(1)
if "diff --quiet -- flake.lock" in operation:
raise SystemExit(0 if not Path("flake.lock").read_text().startswith("updated") else 1)
if args[:1] == ["pull"]:
if fail("PULL_FAIL"):
raise SystemExit(1)
Path("flake.lock").write_text("pulled lock\n")
elif args[:1] == ["push"] and "--dry-run" not in args and fail("PUBLISH_FAIL"):
raise SystemExit(1)
elif args[:1] == ["commit"]:
if fail("COMMIT_FAIL"):
raise SystemExit(1)
Path("flake.lock").write_text("committed lock\n")
elif name == "nix":
if args[:2] == ["flake", "update"]:
Path("flake.lock").write_text("updated lock\n")
elif args[:1] == ["build"]:
if fail("BUILD_FAIL"):
raise SystemExit(1)
print(os.environ["TOPLEVEL"])
elif name == "nix-env":
if fail("ENV_SET_FAIL"):
raise SystemExit(1)
elif name == "systemctl":
if args[:2] == ["is-active", "--quiet"] and fail("HEALTH_FAIL"):
raise SystemExit(1)
if args[:1] == ["stop"] and fail("STOP_FAIL"):
raise SystemExit(1)
elif name == "curl":
if fail("HEALTH_FAIL"):
raise SystemExit(1)
elif name == "jq":
sys.stdin.read()
if fail("HEALTH_FAIL"):
raise SystemExit(1)
else:
Path(os.environ["UNKNOWN_LOG"]).open("a").write(name + "\n")
raise SystemExit(127)
'''
class AutoUpdateTests(unittest.TestCase):
def setUp(self):
self.tmp = Path(tempfile.mkdtemp(prefix="auto-update-test-"))
self.bin = self.tmp / "bin"
self.bin.mkdir()
self.log = self.tmp / "commands.log"
self.unknown = self.tmp / "unknown.log"
fake = self.bin / "fake.py"
fake.write_text(FAKE)
fake.chmod(fake.stat().st_mode | stat.S_IXUSR)
for name in "flock mktemp cp rm date git nix nix-env systemctl curl jq".split():
(self.bin / name).symlink_to(fake)
self.repo = self.tmp / "repo"
self.repo.mkdir()
(self.repo / "flake.lock").write_text("initial lock\n")
self.toplevel = self.tmp / "toplevel"
(self.toplevel / "bin").mkdir(parents=True)
switch = self.toplevel / "bin" / "switch-to-configuration"
switch.write_text("#!/bin/sh\nif [ \"${ACTIVATE_FAIL:-0}\" = 1 ]; then exit 1; fi\nexit 0\n")
switch.chmod(0o755)
self.state = self.tmp / "state"
(self.state).mkdir()
(self.state / ".upgrade-verified").touch()
script = NIX_FILE.read_text()
body = script.split("script = ''", 1)[1].split("'';", 1)[0]
body = re.sub(r"\$\{pkgs\.[^}]+\}/bin/", "", body)
replacements = {
"/run/lock/commonscomputing-update.lock": self.tmp / "update.lock",
"/var/lib/knot/.upgrade-verified": self.state / ".upgrade-verified",
"/nix/var/nix/profiles/system": self.tmp / "profile",
}
for old, new in replacements.items():
body = body.replace(old, str(new))
self.script = self.tmp / "auto-update.sh"
self.script.write_text("#!/usr/bin/env bash\n" + body)
self.script.chmod(0o755)
self.env = os.environ.copy()
self.env.update({"PATH": str(self.bin) + ":" + self.env["PATH"], "FAKE_LOG": str(self.log), "UNKNOWN_LOG": str(self.unknown), "TMP_ROOT": str(self.tmp), "TOPLEVEL": str(self.toplevel)})
def tearDown(self):
shutil.rmtree(self.tmp, ignore_errors=True)
def run_update(self, **flags):
env = self.env.copy()
env.update({k: str(v) for k, v in flags.items()})
return subprocess.run([BASH, str(self.script)], cwd=self.repo, env=env, text=True, capture_output=True)
def commands(self):
return self.log.read_text().splitlines() if self.log.exists() else []
def assert_no_unknown(self):
self.assertFalse(self.unknown.exists(), self.unknown.read_text() if self.unknown.exists() else "")
def test_build_failure_restores_post_pull_lock_without_activation_or_commit(self):
result = self.run_update(BUILD_FAIL=1)
self.assertNotEqual(result.returncode, 0)
self.assertEqual((self.repo / "flake.lock").read_text(), "pulled lock\n")
commands = self.commands()
self.assertFalse(any(x.startswith("nix-env") for x in commands))
self.assertFalse(any(x.startswith("git commit") for x in commands))
self.assert_no_unknown()
def test_activation_failure_pauses_updates_without_rollback_or_publish(self):
result = self.run_update(ACTIVATE_FAIL=1)
self.assertNotEqual(result.returncode, 0)
commands = self.commands()
self.assertIn("systemctl stop auto-update.timer", commands)
self.assertEqual(sum(x.startswith("nix-env") for x in commands), 1)
self.assertFalse(any(x.startswith("git commit") or x == "git push" for x in commands))
self.assertTrue((self.state / ".upgrade-verified").exists() is False)
self.assert_no_unknown()
def test_health_failure_pauses_updates_without_rollback_or_publish(self):
result = self.run_update(HEALTH_FAIL=1)
self.assertNotEqual(result.returncode, 0)
commands = self.commands()
self.assertIn("systemctl stop auto-update.timer", commands)
self.assertEqual(sum(x.startswith("nix-env") for x in commands), 1)
self.assertFalse(any(x.startswith("git commit") or x == "git push" for x in commands))
self.assert_no_unknown()
def test_happy_path_commits_and_pushes_after_health(self):
result = self.run_update()
self.assertEqual(result.returncode, 0, result.stderr)
commands = self.commands()
health = max(i for i, x in enumerate(commands) if x.startswith("curl "))
commit = next(i for i, x in enumerate(commands) if x.startswith("git commit"))
push = next(i for i, x in enumerate(commands) if x == "git push")
self.assertLess(health, commit)
self.assertLess(commit, push)
self.assert_no_unknown()
def test_publication_failure_retains_deployed_lock_without_rollback(self):
result = self.run_update(PUBLISH_FAIL=1)
self.assertNotEqual(result.returncode, 0)
self.assertEqual((self.repo / "flake.lock").read_text(), "committed lock\n")
commands = self.commands()
self.assertFalse(any(x.startswith("nix-env") and i > commands.index("git push") for i, x in enumerate(commands)))
self.assert_no_unknown()
def test_staged_and_nix_edits_refuse_before_pull(self):
for flag in ("STAGED_CHANGES", "NIX_CHANGES"):
with self.subTest(flag=flag):
result = self.run_update(**{flag: 1})
self.assertNotEqual(result.returncode, 0)
self.assertFalse(any(x.startswith("git pull") for x in self.commands()))
self.assert_no_unknown()
if __name__ == "__main__":
unittest.main()
tests/test_upgrade_knot.py
+367
View File
@@ -0,0 +1,367 @@
import hashlib
import os
import shutil
import stat
import subprocess
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SCRIPT = ROOT / "scripts" / "upgrade-knot.sh"
BASH = shutil.which("bash") or "/run/current-system/sw/bin/bash"
DISPATCHER = r'''#!/usr/bin/env python3
import hashlib
import os
import shutil
import sys
from pathlib import Path
name = Path(sys.argv[0]).name
args = sys.argv[1:]
log = Path(os.environ["FAKE_LOG"])
with log.open("a") as f:
f.write(name + " " + " ".join(args) + "\n")
if os.environ.get("KNOT_MASTER_KEY") and name not in {"knot-migrate", "knot-server"}:
Path(os.environ["KEY_LEAK_LOG"]).write_text(name + "\n")
def fail(flag):
return os.environ.get(flag) == "1"
if name == "id":
print("0")
elif name == "hostname":
print("asusmininix")
elif name == "flock":
pass
elif name == "systemctl":
if args[:2] == ["is-active", "--quiet"]:
unit = args[2]
if unit == "auto-update.service":
raise SystemExit(0 if fail("AUTO_UPDATE_ACTIVE") else 1)
if unit == "knot-rs.service" and fail("RUST_INACTIVE"):
raise SystemExit(1)
raise SystemExit(0)
if args[:1] == ["start"] and "auto-update.timer" in args and fail("TIMER_START_FAIL"):
raise SystemExit(1)
elif name == "nix-store":
if args[:1] == ["--check-validity"]:
raise SystemExit(1 if fail("INVALID_STORE") else 0)
if args[:1] == ["--import"]:
sys.stdin.buffer.read()
elif name == "sha256sum":
data = Path(args[0]).read_bytes()
print(hashlib.sha256(data).hexdigest() + " " + args[0])
elif name == "gzip":
try:
os.write(1, b"fixture export")
except OSError:
pass
elif name == "grep":
path = Path(args[-1])
if not path.is_file() or args[1] not in path.read_text():
raise SystemExit(1)
if fail("LIVE_GATE_FAIL") and path.name in {"auto-update.timer", "auto-update.service"}:
raise SystemExit(1)
elif name == "git":
if fail("GIT_PREFLIGHT_FAIL"):
raise SystemExit(1)
elif name == "curl":
if "--write-out" in args:
print("101")
elif "requestCrawl" in " ".join(args) and fail("CRAWL_FAIL"):
raise SystemExit(1)
elif "5555" in " ".join(args) and fail("HEALTH_FAIL"):
print("{}")
else:
print("{}")
elif name == "jq":
sys.stdin.read()
if fail("HEALTH_FAIL") and "version" in " ".join(args):
raise SystemExit(1)
elif name == "ssh-keygen":
try:
os.write(1, b"256 TESTKEY fixture\n")
except OSError:
pass
os._exit(0)
elif name == "ssh-keyscan":
try:
os.write(1, b"127.0.0.1 ssh-ed25519 TESTKEY\n")
except OSError:
pass
os._exit(0)
elif name == "nix-env":
pass
elif name == "agenix":
value = (b"KNOT_MASTER_KEY=" + (b"B" * 43 + b"=\n" if not fail("BAD_KEY") else b"bad\n"))
sys.stdout.buffer.write(value)
target = os.environ.get("AGENIX_TARGET")
if target:
Path(target).write_bytes(value)
elif name == "knot-migrate":
if os.environ.get("KNOT_MASTER_KEY") != "B" * 43 + "=":
raise SystemExit(1)
if "--dry-run" in args:
raise SystemExit(1 if fail("DRYRUN_FAIL") else 0)
target = Path(args[args.index("--target") + 1])
target.mkdir(parents=True, exist_ok=True)
(target / "ssh_host_key").write_text("fixture-key\n")
(target / "sealed-keys").write_text("fixture-sealed\n")
(target / "repo-signing-keys.json").write_text("{}\n")
if fail("MIGRATION_FAIL"):
raise SystemExit(1)
elif name == "sqlite3" or name == "tar" or name == "chown" or name == "chmod":
pass
elif name == "cmp":
if Path(args[-2]).read_bytes() != Path(args[-1]).read_bytes():
raise SystemExit(1)
elif name == "mktemp":
template = args[-1]
path = Path(template.replace("XXXXXX", "fixture"))
path.parent.mkdir(parents=True, exist_ok=True)
if "-d" in args:
path.mkdir(exist_ok=True)
else:
path.touch()
print(path)
elif name == "readlink":
print(os.environ["OLD_SYSTEM"])
elif name == "rm":
for arg in args:
if arg.startswith("-"):
continue
p = Path(arg)
if p.is_dir():
shutil.rmtree(p, ignore_errors=True)
else:
p.unlink(missing_ok=True)
elif name == "rmdir":
Path(args[-1]).rmdir()
elif name == "mv":
shutil.move(args[-2], args[-1])
elif name == "touch":
Path(args[-1]).touch()
else:
Path(os.environ["UNKNOWN_LOG"]).open("a").write(name + "\n")
raise SystemExit(127)
'''
class UpgradeKnotTests(unittest.TestCase):
def setUp(self):
self.tmp = Path(tempfile.mkdtemp(prefix="upgrade-knot-test-"))
self.fakebin = self.tmp / "bin"
self.fakebin.mkdir()
self.log = self.tmp / "commands.log"
self.unknown = self.tmp / "unknown.log"
self.key_scope = self.tmp / "key-scope.log"
dispatcher = self.fakebin / "fake-command.py"
dispatcher.write_text(DISPATCHER)
dispatcher.chmod(dispatcher.stat().st_mode | stat.S_IXUSR)
names = "id hostname flock systemctl nix-store sha256sum gzip grep git curl jq ssh-keygen ssh-keyscan nix-env agenix knot-migrate sqlite3 tar chown chmod cmp mktemp readlink rm rmdir mv touch".split()
for name in names:
(self.fakebin / name).symlink_to(dispatcher)
self.stage = self.tmp / "stage"
self.stage.mkdir()
self.store = self.tmp / "nix" / "store"
self.system = self.store / "fixture-nixos-system-asusmininix-test"
self.tools = self.store / "fixture-knot-migration-tools"
(self.system / "bin").mkdir(parents=True)
(self.system / "sw" / "bin").mkdir(parents=True)
(self.system / "etc" / "systemd" / "system").mkdir(parents=True)
(self.system / "etc" / "knot").mkdir()
(self.system / "sw" / "bin" / "knot-server").write_text(
f"#!/bin/sh\ntest \"$KNOT_MASTER_KEY\" = \"{'B' * 43}=\" || exit 1\nprintf validated > {self.key_scope}\n"
)
(self.system / "sw" / "bin" / "knot-server").chmod(0o755)
switch = self.system / "bin" / "switch-to-configuration"
switch.write_text("#!/bin/sh\nif [ \"${FAIL_ACTIVATION:-0}\" = 1 ]; then exit 1; fi\nexit 0\n")
switch.chmod(0o755)
gate = f"ConditionPathExists={self.state if hasattr(self, 'state') else self.tmp / 'state'}/.upgrade-verified\n"
(self.system / "etc" / "systemd" / "system" / "auto-update.timer").write_text(gate)
(self.system / "etc" / "systemd" / "system" / "auto-update.service").write_text(gate)
(self.tools / "bin").mkdir(parents=True)
(self.stage / "system-path").write_text(str(self.system) + "\n")
(self.stage / "tools-path").write_text(str(self.tools) + "\n")
export = self.stage / "upgrade.export.gz"
export.write_bytes(b"fixture export")
(self.stage / "upgrade.export.gz.sha256").write_text(hashlib.sha256(export.read_bytes()).hexdigest() + " upgrade.export.gz\n")
self.repo = self.tmp / "repo"
(self.repo / "secrets").mkdir(parents=True)
(self.repo / "secrets" / "knot-env.age").write_text("not a secret used by fixture\n")
self.source = self.tmp / "git"
self.source.mkdir()
(self.source / "knotserver.db").write_text("fixture db\n")
self.ssh = self.tmp / "ssh_host_ed25519_key"
self.ssh.write_text("fixture private key\n")
(self.ssh.parent / (self.ssh.name + ".pub")).write_text("fixture public key\n")
self.state = self.tmp / "state"
self.agenix_target = self.tmp / "agenix" / "knot-env"
self.agenix_target.parent.mkdir()
self.agenix_target.write_text("KNOT_MASTER_KEY=" + "B" * 43 + "=\n")
self.old = self.store / "old-system"
(self.old / "bin").mkdir(parents=True)
(self.old / "bin" / "switch-to-configuration").write_text("#!/bin/sh\nexit 0\n")
(self.old / "bin" / "switch-to-configuration").chmod(0o755)
(self.old / "etc" / "systemd" / "system").mkdir(parents=True)
(self.old / "etc" / "systemd" / "system" / "auto-update.timer").write_text(
f"ConditionPathExists={self.state}/.upgrade-verified\n"
)
(self.old / "etc" / "systemd" / "system" / "auto-update.service").write_text(
f"ConditionPathExists={self.state}/.upgrade-verified\n"
)
current = self.tmp / "current-system"
current.symlink_to(self.old)
replacements = {
"/nix/store": self.store,
"/home/anish/tangled-v1.20.0-preflight": self.stage,
"/etc/commonscomputing-nix": self.repo,
"/var/lib/knot": self.state,
"/home/git": self.source,
"/var/backups": self.tmp / "backups",
"/nix/var/nix/profiles/system": self.tmp / "profile",
"/run/current-system": current,
"/etc/ssh/ssh_host_ed25519_key": self.ssh,
"/run/lock/commonscomputing-update.lock": self.tmp / "lock",
"/run/knot-upgrade.XXXXXX": self.tmp / "runtime.XXXXXX",
"/var/lib/knot-upgrade-system": self.tmp / "system-root",
"/var/lib/knot-upgrade-tools": self.tmp / "tools-root",
"/run/agenix/knot-env": self.agenix_target,
}
text = SCRIPT.read_text()
for old, new in replacements.items():
text = text.replace(old, str(new))
self.script = self.tmp / "upgrade-knot.sh"
self.script.write_text(text)
self.env = os.environ.copy()
self.key_leak = self.tmp / "key-leak.log"
self.env.update({"PATH": str(self.fakebin) + ":" + self.env["PATH"], "FAKE_LOG": str(self.log), "UNKNOWN_LOG": str(self.unknown), "KEY_LEAK_LOG": str(self.key_leak), "OLD_SYSTEM": str(self.old), "AGENIX_TARGET": str(self.agenix_target)})
def tearDown(self):
shutil.rmtree(self.tmp, ignore_errors=True)
def run_upgrade(self, *args, **flags):
env = self.env.copy()
env.update({k: str(v) for k, v in flags.items()})
return subprocess.run([BASH, str(self.script), *args], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
def commands(self):
return self.log.read_text().splitlines() if self.log.exists() else []
def assert_no_unknown_tools(self):
self.assertFalse(self.unknown.exists(), self.unknown.read_text() if self.unknown.exists() else "")
def test_happy_path_order_and_secret_cleanup(self):
result = self.run_upgrade("--yes")
self.assertEqual(result.returncode, 0, result.stderr)
commands = self.commands()
self.assertLess(next(i for i, x in enumerate(commands) if x.startswith("systemctl stop auto-update.timer")), next(i for i, x in enumerate(commands) if x.startswith("knot-migrate ") and "--dry-run" not in x))
timer_start = next(i for i, x in enumerate(commands) if x == "systemctl start auto-update.timer")
health_end = max(i for i, x in enumerate(commands) if x.startswith("curl ") and i < timer_start)
self.assertGreater(timer_start, health_end)
self.assertTrue((self.state / ".migration-complete").exists())
self.assertFalse(self.tmp.joinpath("runtime.fixture", "knot.env").exists())
self.assertFalse(self.key_leak.exists())
self.assertTrue(self.key_scope.exists())
self.assert_no_unknown_tools()
def test_dryrun_failure_does_not_stop_go_or_timer(self):
result = self.run_upgrade("--yes", DRYRUN_FAIL=1)
self.assertNotEqual(result.returncode, 0)
commands = self.commands()
self.assertFalse(any(x == "systemctl stop knot.service" for x in commands))
self.assertFalse(any(x == "systemctl stop auto-update.timer" for x in commands))
self.assertFalse(self.tmp.joinpath("runtime.fixture", "knot.env").exists())
self.assert_no_unknown_tools()
def test_migration_failure_quarantines_target_and_restarts_go(self):
result = self.run_upgrade("--yes", MIGRATION_FAIL=1)
self.assertNotEqual(result.returncode, 0)
self.assertTrue((self.tmp / "backups" / "knot-cutover.fixture" / "failed-target").exists())
self.assertFalse(self.tmp.joinpath("runtime.fixture", "knot.env").exists())
commands = self.commands()
self.assertIn("systemctl start knot.service", commands)
self.assert_no_unknown_tools()
def test_activation_failure_keeps_new_state_and_stops_services(self):
result = self.run_upgrade("--yes", FAIL_ACTIVATION=1)
self.assertNotEqual(result.returncode, 0)
commands = self.commands()
sets = [x for x in commands if x.startswith("nix-env --profile")]
self.assertEqual(len(sets), 1)
self.assertFalse(any(x == "systemctl start knot.service" for x in commands))
self.assertIn("systemctl stop auto-update.timer", commands)
self.assertTrue(self.state.exists())
self.assertFalse((self.state / ".migration-complete").exists())
self.assert_no_unknown_tools()
def test_health_failure_keeps_new_state_and_stops_services(self):
result = self.run_upgrade("--yes", HEALTH_FAIL=1)
self.assertNotEqual(result.returncode, 0)
commands = self.commands()
self.assertEqual(sum(x.startswith("nix-env --profile") for x in commands), 1)
self.assertNotIn("systemctl start knot.service", commands)
self.assertIn("systemctl stop knot-rs.service", commands)
self.assertTrue(self.state.exists())
self.assert_no_unknown_tools()
def test_completed_marker_skips_migration(self):
self.state.mkdir()
(self.state / ".migration-complete").touch()
result = self.run_upgrade()
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(any(x.startswith("knot-migrate") for x in self.commands()))
self.assertTrue(any(x == "systemctl start auto-update.timer" for x in self.commands()))
self.assert_no_unknown_tools()
def test_completed_marker_rejects_mismatched_live_gate_without_touching_state(self):
self.state.mkdir()
(self.state / ".migration-complete").touch()
before = (self.state / ".upgrade-verified").exists()
result = self.run_upgrade(LIVE_GATE_FAIL=1)
self.assertNotEqual(result.returncode, 0)
self.assertEqual((self.state / ".upgrade-verified").exists(), before)
self.assertFalse(any(x == "systemctl start auto-update.timer" for x in self.commands()))
self.assert_no_unknown_tools()
def test_invalid_key_fails_before_downtime(self):
result = self.run_upgrade("--yes", BAD_KEY=1)
self.assertNotEqual(result.returncode, 0)
commands = self.commands()
self.assertFalse(any(x == "systemctl stop knot.service" for x in commands))
self.assertFalse(any(x == "systemctl stop auto-update.timer" for x in commands))
self.assert_no_unknown_tools()
def test_missing_staged_service_gate_fails_before_downtime(self):
service = self.system / "etc" / "systemd" / "system" / "auto-update.service"
service.write_text("[Unit]\n")
result = self.run_upgrade("--yes")
self.assertNotEqual(result.returncode, 0)
self.assertFalse(any(x == "systemctl stop knot.service" for x in self.commands()))
self.assertFalse(self.state.exists())
self.assert_no_unknown_tools()
def test_git_preflight_failure_has_no_downtime(self):
result = self.run_upgrade("--yes", GIT_PREFLIGHT_FAIL=1)
self.assertNotEqual(result.returncode, 0)
commands = self.commands()
self.assertFalse(any(x == "systemctl stop knot.service" for x in commands))
self.assertFalse(any(x == "systemctl stop auto-update.timer" for x in commands))
self.assert_no_unknown_tools()
def test_invalid_store_imports_export(self):
result = self.run_upgrade("--yes", INVALID_STORE=1)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertTrue(any(x.startswith("nix-store --import") for x in self.commands()))
self.assert_no_unknown_tools()
if __name__ == "__main__":
unittest.main()