132 lines
6.4 KiB
Markdown
132 lines
6.4 KiB
Markdown
# Tangled Upgrade
|
|
|
|
The initial cutover uses tested v1.20.0 packages. `flake.lock` still selects
|
|
commit `2096d06d938e680297adeadd71b8320c9f3491d4`; the input URL tracks upstream
|
|
again so subsequent weekly updates can advance Tangled. Upstream's own nixpkgs
|
|
is retained to reuse the built release binaries.
|
|
|
|
## Run The Script
|
|
|
|
Pull the latest configuration on asusmini, then run:
|
|
|
|
```sh
|
|
sudo bash /etc/commonscomputing-nix/scripts/upgrade-knot.sh
|
|
```
|
|
|
|
The script rehearses first, prints the migration report, and asks you to type
|
|
`MIGRATE` before stopping Go knot. Keep another administrative SSH session open,
|
|
back up the master key off-host, and ask users to pause writes during cutover.
|
|
`--yes` skips the approval prompt. Running the script authorizes migration and
|
|
deployment, not merely a smoke test.
|
|
|
|
Root's Git identity must be able to fetch and push to `origin`. The script
|
|
checks `ls-remote` and `push --dry-run` before downtime. Origin is the separate
|
|
Gitea service on port 2222, not this knot.
|
|
|
|
If encrypted-secret edits prevent a fast-forward pull, preserve only those
|
|
edits first; leave unrelated DNS changes alone:
|
|
|
|
```sh
|
|
sudo git -C /etc/commonscomputing-nix stash push \
|
|
-m 'pre-cutover encrypted secret backup' -- secrets/knot-env.age secrets/secrets.nix
|
|
sudo git -C /etc/commonscomputing-nix pull --ff-only
|
|
```
|
|
|
|
The encrypted secret is already committed; do not pop an identical stash over
|
|
it. Its SHA-256 is
|
|
`b26f36d098a337a98217aa4e73d2f39eb4e8864337dfcface937093f5557f0ac`.
|
|
|
|
## Happy Path
|
|
|
|
1. Lock out concurrent updates and validate the host, staging files, and Git access.
|
|
2. Import the prebuilt system/tools if missing, after checking the archive checksum.
|
|
3. Decrypt the persistent key into root-only runtime storage and rehearse.
|
|
4. Obtain downtime approval; stop the timer and Go knot.
|
|
5. Back up the database, repositories, SSH identity, and old NixOS generation.
|
|
6. Copy-migrate repositories/access grants while preserving all original source data.
|
|
7. Validate the imported identity/configuration and install the prebuilt system.
|
|
8. Set ownership, check agenix supplied the same key, and start Rust knot.
|
|
9. Check local/public APIs, service state, SSH fingerprint, and Caddy WebSocket upgrade.
|
|
10. Mark verification complete, start the weekly timer, and request an indexer crawl.
|
|
|
|
Source data stays in `/home/git`; Rust state lives in `/var/lib/knot`. The
|
|
previous rehearsal found 23 repositories, 16 members, 12 collaborator grants,
|
|
zero skips, and no ACL/table disagreement. Check the fresh report for changes.
|
|
|
|
Backups are root-only under `/var/backups/knot-cutover.*`. They contain private
|
|
key material, including `repo-signing-keys.json`; back them up encrypted off-host.
|
|
The master key is not printed and is scoped to migration/config validation.
|
|
The runtime plaintext file is removed on ordinary success/failure and signals;
|
|
SIGKILL may leave it root-readable in `/run` until cleanup or reboot.
|
|
|
|
## Networking
|
|
|
|
Administrative SSH stays on **22**. Rust knot Git SSH uses **2223**, opened by
|
|
the NixOS module. Use `100.122.40.27:2223` over Tailscale, or arrange external
|
|
TCP 2223 forwarding to `192.168.1.249:2223`. Existing port-22 Git URLs need an
|
|
explicit port or client SSH configuration. Public 2222 belongs to Gitea and
|
|
must not be repurposed. The script does not change router forwarding or sshd.
|
|
|
|
Caddy proxies all paths and WebSocket upgrades to `192.168.1.249:5555`; Knot
|
|
trusts forwarded addresses only from `192.168.1.11/32`. The script checks
|
|
`/xrpc/com.atproto.sync.subscribeRepos` through Caddy. It cannot test an
|
|
authenticated user push without their credentials: after success, test a real
|
|
clone/push/delete of a temporary branch over port 2223.
|
|
|
|
## Automatic Updates
|
|
|
|
Both timer and service require `/var/lib/knot/.upgrade-verified`. The script
|
|
creates it only after successful smoke tests, then starts the timer. A separate
|
|
migration marker prevents Rust knot from initializing an empty identity.
|
|
|
|
Weekly updates fast-forward the configuration, update inputs, build, activate,
|
|
and health-check **before** committing/pushing `flake.lock`. Preexisting staged
|
|
changes, lockfile edits, or Nix configuration edits cause refusal. Unrelated
|
|
unstaged DNS edits remain untouched. Git runs noninteractively; update and
|
|
migration jobs share a process lock.
|
|
|
|
A failed build leaves the live generation unchanged and restores the script's
|
|
own lockfile changes. After activation is attempted, failure stops the application
|
|
services and timer and retains state/lockfile; old binaries are not automatically
|
|
run against possibly forward-migrated data. Publication failure retains the
|
|
healthy deployed generation and local changes; reconcile them with origin
|
|
before the next update.
|
|
|
|
Future versions may compile on asusmini unless supplied by a binary cache.
|
|
Only the initial v1.20.0 cutover is guaranteed to reuse staged binaries.
|
|
|
|
## Failure And Retry
|
|
|
|
Before activation, failed migration targets are quarantined in the backup
|
|
directory and unchanged Go knot is restarted. After activation is attempted,
|
|
the script stops Rust knot/updates but preserves migrated state: blind rollback
|
|
could lose accepted writes or use an incompatible format. Investigate before
|
|
reopening writes. Runtime timer masks do not survive reboot; do not reboot an
|
|
old generation unattended during recovery.
|
|
|
|
A completed rerun verifies the live Rust service and resumes the gated timer;
|
|
it never migrates again or reinstalls an old staged generation. An incomplete
|
|
target without a completion marker is refused. Power loss or SIGKILL midway
|
|
requires inspection/recovery, not a blind rerun.
|
|
|
|
## Staging And Tests
|
|
|
|
`/home/anish/tangled-v1.20.0-preflight` contains `system-path`, `tools-path`,
|
|
`upgrade.export.gz`, and its checksum. The archive contains paths missing at
|
|
staging time; avoid garbage collection before import. Import alone activates
|
|
nothing. The root-authorized import accepts unsigned local products only for
|
|
that command, without changing global Nix trust settings.
|
|
|
|
Local/server runtime and synthetic-migration smoke tests passed earlier. Mocked
|
|
orchestration tests cover success, preflight refusal, migration failure,
|
|
post-activation failure, secret cleanup, idempotence, and publication ordering.
|
|
They do not simulate a full boot or replace the on-host checks.
|
|
|
|
Upstream's v1.20.0 spindle still reports hardcoded wire version `v1.16.1` in
|
|
its describe endpoint; identify its actual package by the service store path.
|
|
|
|
## Upstream Guides
|
|
|
|
- https://tangled.org/docs/upgrading/knot-2/
|
|
- https://tangled.org/docs/upgrading/knots-and-spindles/#upgrading-to-v1200
|