Release runbook — private train to Artifact Registry

Status: operating procedure of the private train (WP-RN-07, spec 15 §4.2; record docs/specs/platform-gaps/phase1/rn-07-release.md). Audience: release owners of a5c-ai/fabric. Model reference: docs/release-pipeline.md; versioning posture: docs/VERSIONING.md; registry/credentials: docs/user-guide/private-install.md. Historical (public-train, superseded): docs/release-recovery-runbook.md.

**Registry mutations happen only in GitHub Actions through Workload Identity

Federation.** Nothing in this runbook asks a human to run npm publish or

npm dist-tag add from a laptop: a maintainer holds roles/artifactregistry.reader

at most, the CI service account alone holds writer

(docs/specs/platform-gaps/phase0/ar-provisioning.md). Every command marked

*local* is read-only or a dry run.

**Every gh workflow run in this runbook names --ref.** Without --ref, gh dispatches

the copy of the workflow file on the repository's **default branch** (main). Until PR #4

reaches main **and staging**, the copies of publish-packages-from-tag.yml and

fix-npm-tags.yml on **both** branches are the legacy public-train files (the same blob,

44112b847, for fix-npm-tags.yml: registry-url: https://registry.npmjs.org/,

NODE_AUTH_TOKEN: secrets.NPM_TOKEN, babysitter/** tags, workflow_dispatch default

dry_run=false) — a dispatch bound to them is a dispatch against the public registry, and

--ref staging is itself the legacy copy until the merge.

--ref <release branch or fabric/ tag> runs the file of that ref instead, which is correct

only once that ref carries this branch's file. Do not dispatch fix-npm-tags.yml until PR #4 is on staging and main and M7 is verified. It is disabled_manually until Gate B

re-enables it after the merge (section 0a, review-2 M8). The --ref rule is pinned by

scripts/__tests__/private-release-train.test.mjs ("every gh workflow run in the runbooks

names its --ref"), which also asserts the sentence above. Review M3 / review-2 M8, 2026-10-04.

0. Invariants the train encodes

0a. Gate-B prerequisites and state (2026-10-04)

1. Normal train (push to `staging`, `main` or `develop`)

Workflow: .github/workflows/publish-npm-private.yml (job graph below). Nothing to run by hand; the steps are listed so a reviewer can follow a run.

resolve_release          resolve ONE identity (version, dist-tag, fabric/ tag, registry, candidate tag)
                         verify-manifests on the checked-in tree; publish-isolation guards (G1-G3)
verify_release_artifacts release-artifact-verifier.yml: all 43 packages, no tolerance (FIX-011)
publish_candidates       needs + checks the verifier result; WIF auth -> google-artifactregistry-auth
                         release plan written + validated; sync-workspace-versions + verify-manifests
                         + publish-isolation-guard over the synchronized tree; preflight channel assertion
                         build the train in derived dependency waves; verify SDK package health
                         publish every package under candidate-<version> (FABRIC_RELEASE_PUBLISH=1)
                         assert every candidate dist-tag == version
published_consumer_validation   live-stack-published.yml with the EXACT version (section 3)
promote_release_channel  release-promotion.cjs promote --evidence validation.json (the only dist-tag move)
                         gated on the EXACT-VERSION consumer job + the recorded verdict (callee outputs
                         consumer_result / promotable); the live-model lanes are informational unless the
                         variable FABRIC_LIVE_LANES_GATE=required (owner decision 2026-10-05, section 0a)
assert_release_channel_tags     every package's channel tag == version (FIX-001 final assertion)
create_release_tag       release-tags.yml: fabric/<branch>/v<version>, provenance fabric-release-*

What to check after a green run (read-only). Prerequisite: re-authenticate first (section 6; M6 verified 2026-10-04) — the run's uploaded artifacts are the CI-side evidence of the same facts.

# the identity the run resolved (also in the run's step summary)
SHORT_SHA=$(git rev-parse --short=12 <released commit>)
node scripts/release-version.cjs resolve --branch staging --sha "$SHORT_SHA"
# channel state on Artifact Registry over ALL 43 packages (reader credential, section 6)
node scripts/release-version.cjs assert-channel-tags --version <version> --dist-tag staging --mode final
# the tag exists here and nowhere else
git ls-remote --tags origin "refs/tags/fabric/staging/v<version>"
git ls-remote --tags https://github.com/a5c-ai/babysitter.git | grep -c "fabric/"   # must be 0

2. Dry-run commands (local, nothing written anywhere)

# 1. the identity the train would use for this commit on each channel
SHORT_SHA=$(git rev-parse --short=12 HEAD)
node scripts/release-version.cjs resolve --branch main    --sha "$SHORT_SHA"
node scripts/release-version.cjs resolve --branch staging --sha "$SHORT_SHA"
node scripts/release-version.cjs verify  --branch main    --sha "$SHORT_SHA" --version "$(node -p "require('./package.json').version")"

# 2. the checked-in train is lockstep with exact internal pins (G2(d))
node scripts/release-version.cjs verify-manifests --version "$(node -p "require('./package.json').version")"

# 3. publish-isolation (G1-G3), public-push guard, metadata/coverage gate
node scripts/ci/publish-isolation-guard.cjs --self-test
node scripts/ci/public-push-guard.cjs --self-test
npm run verify:metadata

# 4. the derived build/publication waves
node scripts/release-matrix.cjs --group all-publishable --format waves

# 5. release tooling + workflow-shape suites (FIX-001/005/010/011 + WP-RN-07)
npm run test:release-tooling

# 6. rehearse the publish helper for one package: npm publish --dry-run under the
#    candidate tag, nothing written. --dry-run queries Artifact Registry for the
#    package's own existence (READER credential, section 6 — re-authenticate first) and
#    accepts an exact internal pin that a workspace of this checkout carries at
#    that version ("published by an earlier wave of this train"); any other pin
#    is still looked up on the registry. --rehearsal (implies --dry-run) issues
#    NO registry query at all, so it needs no credential — this is what the CI
#    rehearsal job runs for every package.
GITHUB_REF_NAME=main node scripts/publish-package-from-tag.mjs \
  --workspace=@fabricstack/trust --release-version="$(node -p "require('./package.json').version")" --skip-build --dry-run
GITHUB_REF_NAME=main node scripts/publish-package-from-tag.mjs \
  --workspace=@fabricstack/harness-policy --release-version="$(node -p "require('./package.json').version")" --skip-build --rehearsal

# 7. the full packed-artifact matrix (slow; what the release gate runs)
node scripts/verify-release-artifacts.mjs --report-dir artifacts/release-verifier --exactness

Whole-train rehearsal in CI — two forms:

1. **Before the merge (every PR):** the release-train-rehearsal job of ci.yml (section 0a) executes the train's publish_candidates sequence end-to-end for the staging identity of the PR head, with --rehearsal in place of the publication opt-in, and proves it reached the last of the 13 waves with all 43 packages dry-run-published and no publish command without --dry-run. The registry-facing steps (preflight channel assertion, candidate dist-tag assertion) and the packed-artifact gate are logged as skipped there: the verifier runs as its own PR workflow (release-artifact-verifier.yml), the two assertions run in the train. 2. **After the merge:** gh workflow run publish-npm-private.yml -R a5c-ai/fabric --ref staging with the default dry_run=true runs every job up to and including npm publish --dry-run for all 43 packages (the helper's --dry-run satisfies exact internal pins from the synchronized checkout, so an empty registry no longer stops wave 2), then stops before validation/promotion/tagging. A dispatch publishes only when dry_run is explicitly false.

**The first real run of publish-npm-private.yml is the staging merge itself** (a push to staging runs with dry_run=false); it cannot be dispatched before the file is on a branch GitHub registers it from. Gate B therefore relies on form 1 plus the verifier's 43/43 as the rehearsal evidence (docs/specs/platform-gaps/phase1/rn-07-release.md §6).

3. Published-consumer validation (what gates promotion)

.github/workflows/live-stack-published.yml, called with the exact version and the candidate dist-tag, never a channel:

(@fabricstack:registry=<Artifact Registry> — written by scripts/verify-published-release.mjs), all 43 packages installed at @<version>, every root and exported runtime subpath imported, every declared bin smoked; then the spec 15 §3.1 global-install smoke (WP-RN-06 AC2): npm i -g @fabricstack/orchestrator@V @fabricstack/harness-cli@V with per-call scope routing (--@fabricstack:registry=<AR>), no EEXIST, exactly the orchestrator bin; then @fabricstack/harness-hooks-cli@V @fabricstack/harness-sdk@V → exactly harness harness-hooks orchestrator; each answers --version.

agent-runtime-cli, harness-hooks-cli) installed globally from Artifact Registry at the exact version, asserted with release-promotion.cjs assert-installed, then the live stack.

subpath-import, bin-smoke, live-stack); release-promotion.cjs promote refuses without it.

RECORDED and every *gating* check green. live-stack is gating only when the evidence was recorded with --live-lanes-gate required, which record_validation derives from the repository/organization variable FABRIC_LIVE_LANES_GATE (unset or informational → consumer-only gate, the lanes' failure is listed under informationalFailures; required → the full gate; the script's own default is required). The caller's promote_release_channel evaluates after a red lane (!cancelled()) and checks, explicitly, the resolver and publish results, the consumer job's own result (consumer_result == 'success'), the recorded verdict (promotable == 'true') and, under required, the called workflow's aggregate result. No lane is continue-on-error. Condition to restore the full gate: a valid OPENAI_API_KEY (section 7.2).

• published_consumer: fresh project outside the repository with its own .npmrc
• live-stack lanes: the §3.1 CLIs (orchestrator, orchestrator-sdk, harness-sdk,
• record_validation writes validation.json (checks package-install, root-import,
• **Gate (owner decision 2026-10-05, "Consumer-only gate for the first train")**: every check must be

Local equivalent of the consumer half (reader credential, section 6 — re-authenticate first; the published-consumer-validation artifact is the CI-side evidence):

node scripts/verify-published-release.mjs --version <version> --report-dir artifacts/published-consumer

4. Recovering a half-published train

Symptom: publish_candidates failed in wave N. Waves < N are on Artifact Registry under candidate-<version>; **wave N itself is partially published** — its helpers run in parallel and wait collects every sibling before the step exits, so the members that succeeded are on the registry and the failed ones are not; waves > N are missing; **no channel tag has moved** and **no release tag exists** (both happen after validation). Nothing is broken for consumers: channels still resolve the previous release (or nothing, on a first train).

**First staging train (7.0.1-staging.<sha>) — abort is fix-forward only once staging has been promoted.** Before promotion there is nothing to roll back: the candidate versions are orphaned and no channel points at them. After promotion there is no earlier version on Artifact Registry for --set staging=<previous-version> (section 5; the script refuses a version that any package lacks), so the only abort is a new commit → a new train that moves staging forward again. Consumers of the brand-new @fabricstack scope are unaffected either way: nothing is pinned to it yet, and latest never moves on a staging train.

4.1 Find out what is on the registry (local, read-only)

Re-authenticate first (section 6). The uploaded publish-npm-private-<version> artifact (one helper log per package: published … / already exists … / the failure) is the same inventory, read from CI.

V=<version>
AR=$(node -p "require('./package.json').fabric.privateRegistry")
# which packages have the exact version (prints MISSING for the unpublished half)
node scripts/release-matrix.cjs --group all-publishable --format workspaces | while read -r p; do
  if npm view "$p@$V" version --registry "$AR" >/dev/null 2>&1; then echo "PUBLISHED $p"; else echo "MISSING $p"; fi
done
# candidate dist-tag state over the whole inventory (final mode fails while any package is missing — that is the diagnosis)
node scripts/release-version.cjs assert-channel-tags --version "$V" --dist-tag "$(node scripts/release-promotion.cjs candidate-tag --version "$V")" --mode final

4.2 Decide

publish_candidates job on the same run** (gh run rerun <run-id> -R a5c-ai/fabric --failed). It is idempotent — the helper only ensures the candidate dist-tag of a version that already exists and publishes the missing ones — and the same run then continues into validation, promotion, assertion and the tag, so nothing is re-derived. Do **not** push an empty commit: the branch train resolves a *new* version for a new commit. Use 4.3 when the run itself is dead (expired, cancelled, or failed a second time) — it acts on an immutable tag.

publishes a complete train. The orphaned candidate-<old> versions stay installable for diagnosis and are never promoted; nothing needs deleting (channels never pointed at them).

• **Cause was transient** (registry hiccup, runner died): **first re-run the failed
• **Cause was a package defect**: fix it on the branch; the next push resolves a new version and

4.3 Recover from the immutable tag (`publish-packages-from-tag.yml`) — APPROVAL REQUIRED

Recovery acts on a fabric/<branch>/v<version> tag so the tree is immutable. The branch train only tags after promotion, so for a half-published train the tag does not exist yet: create it locally **at the released commit**, push it, and let the tag workflow do the rest.

# at the exact commit the failed run built (check the run's "Release identity" summary)
git checkout <commit>
SHORT_SHA=$(git rev-parse --short=12 HEAD)
node scripts/release-version.cjs verify --branch staging --sha "$SHORT_SHA" --version "$V"   # must agree
# local, idempotent, annotated, provenance source=manual (so the tag workflow knows it may publish)
node scripts/release-version.cjs ensure-tag --version "$V" --branch staging --commit "$(git rev-parse HEAD)" --source manual
node scripts/release-version.cjs tag-source --tag "fabric/staging/v$V"      # published_by_release_workflow=false
# APPROVAL REQUIRED: pushing the tag starts the recovery publication
git push origin "refs/tags/fabric/staging/v$V"
# or, for an existing tag — --ref is mandatory (see the note at the top): dispatch from the
# tag itself or from `staging`/`main`, never from a feature branch whose name contains `/`
# (the helper refuses any `/`-shaped ref outside fabric/<branch>/v<version> as a channel source)
gh workflow run publish-packages-from-tag.yml -R a5c-ai/fabric --ref "fabric/staging/v$V" -f tag="fabric/staging/v$V"

What the recovery lane does (all automatic, all idempotent):

1. release_identity: derives version/channel/registry from the tag; refuses a tag the branch train created itself (published_by_release_workflow=true) unless manually dispatched. 2. verify_release_artifacts: the strict matrix against the tag's tree. 3. publish_non_plugin_packages: WIF auth, release plan from the tag, sync + verify every manifest, preflight channel assertion, build in derived waves, publish in derived waves under candidate-<version>. **Packages whose exact version already exists are not re-published — the helper only ensures their candidate dist-tag** (scripts/publish-package-from-tag.mjs), so the already-published half is a no-op and the missing half completes. Then every candidate dist-tag is asserted. 4. published_consumer_validation → promote_release_channel → channel assertion, exactly as the branch train. Re-running the recovery for the same tag after a second failure is safe for the same reason.

Idempotency guarantees, by mechanism: isPublished (E404-only "absent" detection; any other registry answer aborts rather than guesses), ensure-tag (never re-points a tag), assert-channel-tags --mode preflight (never moves a channel backward), release-promotion.cjs promote (refuses without evidence naming this exact version).

4.4 After recovery

Local, read-only (re-authenticate first, section 6); the recovery run's assert_release_channel_tags job is the CI-side evidence.

node scripts/release-version.cjs assert-channel-tags --version "$V" --dist-tag staging --mode final
git ls-remote --tags origin "refs/tags/fabric/staging/v$V"

5. Repairing dist-tags (`fix-npm-tags.yml`) — APPROVAL REQUIRED for `dry_run=false`

Script: scripts/fix-broken-latest-tags.mjs, inventory-derived, private registry named on every call. Two modes, both dry runs unless --fix / dry_run=false:

# audit: latest -> channel prerelease, channel tag -> empty artifact, missing latest; proposes the last healthy release
node scripts/fix-broken-latest-tags.mjs
node scripts/fix-broken-latest-tags.mjs --json
# explicit repair of one dist-tag for EVERY package at one exact version (refuses if any package lacks that version)
node scripts/fix-broken-latest-tags.mjs --set staging=<version>
# narrow
node scripts/fix-broken-latest-tags.mjs --package @fabricstack/catalog

In CI (--ref mandatory, see the note at the top). **Until PR #4 is merged, the copies of this workflow on main AND staging are the legacy npmjs file** (identical blob 44112b847): workflow_dispatch input dry_run defaults to false, checkout ref: staging, registry-url: https://registry.npmjs.org/, NODE_AUTH_TOKEN: secrets.NPM_TOKEN, and the non-dry branch runs scripts/fix-broken-latest-tags.mjs --fix over the public @a5c-ai/* packages — so --ref staging is only correct after the merge, and a dispatch with no --ref, with --ref main, or with --ref staging before the merge would be a live public dist-tag mutation (the forms below carry -f mode=…, which the legacy inputs reject with 422, but a bare dispatch would not be rejected). Do not dispatch fix-npm-tags.yml until PR #4 is on staging and main and M7 is verified. Mitigation in force: the workflow is disabled_manually (re-disabled by the orchestrator 2026-10-04, review-2 M8) and is re-enabled by the orchestrator as part of Gate B, right after the merge to staging and after M7's verification (section 0a). Once re-enabled: gh workflow run fix-npm-tags.yml -R a5c-ai/fabric --ref staging -f mode=audit (dry run), then gh workflow run fix-npm-tags.yml -R a5c-ai/fabric --ref staging -f mode=audit -f dry_run=false once the plan is approved; gh workflow run fix-npm-tags.yml -R a5c-ai/fabric --ref staging -f mode=set -f set_spec=latest=<version> for the explicit repair. The audit mode needs no human registry credential (the workflow authenticates through WIF), so it is the diagnosis tool whenever a human token is stale. A channel rollback is the same operation with the previously validated version: --set latest=<previous-version> — publication is immutable, the dist-tag is the only reversible surface. Never unpublish; recover forward. **On the FIRST staging train there is no <previous-version> on Artifact Registry** (the script refuses a version any package lacks), so a post-promotion abort of that train is fix-forward only — a new commit, a new train (section 4).

6. Credentials for the read-only local commands

**Review M6 — verified 2026-10-04 (section 0a).** On 2026-10-03 the human credential of tal@a5c.ai received 403 Forbidden from Artifact Registry on every npm read (npm view of an existing and of a made-up package alike, and a direct bearer-token GET), although the phase-0 record lists the account as roles/artifactregistry.reader. The cause was an expired access token: with a fresh gcloud token the orchestrator ran the verification below and all of it passed (the policy shows roles/artifactregistry.reader for user:tal@a5c.ai; npm view @fabricstack/smoke → 0.0.0-smoke.4; the direct bearer GET → 200). **Prerequisite kept: re-authenticate first** — run the google-artifactregistry-auth line before any local read-only command in this runbook, and again when a read starts answering 401/403. The verification, re-runnable at any time:

gcloud artifacts repositories get-iam-policy fabric-npm --location=us-central1 --format=json   # roles/artifactregistry.reader for user:tal@a5c.ai
npx -y google-artifactregistry-auth --repo-config=.npmrc --credential-config="$HOME/.npmrc"
npm view @fabricstack/smoke --@fabricstack:registry=https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/   # must succeed

No workflow depends on the human credential: all of them authenticate as the CI service account through WIF, which published in ar-smoke run 36454074285. If the three lines above ever fail again, the run's uploaded artifacts carry the same facts and every *local* read-only command in this runbook falls back to CI-only until the credential is repaired.

gcloud auth login            # once; tal@a5c.ai holds roles/artifactregistry.reader on fabric-npm
npx -y google-artifactregistry-auth --repo-config=.npmrc --credential-config="$HOME/.npmrc"
npm ping --registry https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/
git diff --exit-code -- .npmrc   # the committed file never gains a credential line

Tokens live about an hour; rerun the helper when npm view starts answering 401/403. A 403 on a package you expect to exist means the active gcloud account lacks the reader binding (gcloud auth list), not that the package is absent.

7. Follow-ups attached to the first publications

7.1 Pin the newly published agent-runtime hooks leaf in the hooks CLI — REQUIRED FOLLOW-UP

@fabricstack/harness-hooks-agent-runtime has never been published, so packages/adapters/hooks/cli/package.json deliberately does not pin it yet (//deferred-dependency, FIX-005): an exact pin on an unpublished version would fail the clean-consumer verification and the publish helper's exact-internal-dependency gate. Immediately after the first train that publishes it:

# AFTER @fabricstack/harness-hooks-agent-runtime@$RELEASE_VERSION exists on Artifact Registry:
npm view "@fabricstack/harness-hooks-agent-runtime@$RELEASE_VERSION" version --registry "$AR"   # must succeed
# add the exact pin next to the other hooks leaves, drop the //deferred-dependency note
node -e "const fs=require('fs');const p='packages/adapters/hooks/cli/package.json';const m=JSON.parse(fs.readFileSync(p));m.dependencies['@fabricstack/harness-hooks-agent-runtime']=m.version;delete m['//deferred-dependency'];fs.writeFileSync(p,JSON.stringify(m,null,2)+'\n')"
npm install --package-lock-only --legacy-peer-deps
node scripts/release-version.cjs verify-manifests --version "$(node -p "require('./package.json').version")"
npm run test:release-tooling   # the FIX-005 test flips to guarding the pin

7.2 Pending owner actions (recorded, not automated)

37279009004)**: the secret is delivered to fabric (visibility all, non-empty in the job) and rejected by OpenAI. pi lane, agent-output.txt: OpenAI API error (401): {"message":"Incorrect API key provided: AcuYB8hq************…JuS3. …","type":"invalid_request_error","code":"invalid_api_key"}; codex lane (same credential through codex 0.160.0's websocket transport): 401 Unauthorized: Missing bearer or basic authentication in header, url: https://api.openai.com/v1/responses. Every default live lane is openai/gpt-5.5, so no lane can pass until the secret is replaced. Then set the variable FABRIC_LIVE_LANES_GATE=required to restore the full gate (section 0a). Verification: a green Published (…, codex/gpt-5.5, non-interactive) lane on the next train, and validation.json with liveLanesGate: "required" and informationalFailures: [].

before its Gate-B re-enable** (review-2 M8 moved it forward from "before Gate B", 2026-10-04): restrict the organization secret NPM_TOKEN to the repositories that publish to npmjs (fabric unselected): docs/specs/platform-gaps/phase0/g3-npm-token-restriction.md. Fabric's branch workflows no longer reference it, but the legacy copies on main and staging still do (which is what makes --ref mandatory above and why fix-npm-tags.yml stays disabled until Gate B), and the secret is currently visible to all repositories (gh api orgs/a5c-ai/actions/secrets --jq '.secrets[] | "\(.name) \(.visibility)"' → NPM_TOKEN all). Verification: gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN --jq .visibility → selected, and gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN/repositories --jq '.repositories[].name' without fabric.

fresh token (section 6); re-authenticate before local reads.

restricted membership): docs/specs/platform-gaps/phase0/g4-npmjs-org-runbook.md; node scripts/ci/publish-isolation-guard.cjs --g4-status prints the current state.

• **Organization secret OPENAI_API_KEY is invalid — owner action (found 2026-10-05, publish run
• **G3 / review M7 — owner action, required BEFORE anyone dispatches fix-npm-tags.yml, i.e.
• **Review M6 — verified 2026-10-04**: the human reader binding on fabric-npm works with a
• **G4** — register the fabricstack organization on public npmjs (publish nothing, 2FA,

8. Public-namespace invariant checks (run after any release activity)

# the public repository was not written to
gh api repos/a5c-ai/babysitter --jq .pushed_at                     # unchanged
git ls-remote --tags https://github.com/a5c-ai/babysitter.git | grep -c "fabric/"   # 0
git ls-remote --tags https://github.com/a5c-ai/babysitter.git | wc -l               # unchanged count
# nothing under the private scope exists on the public registry (404 everywhere)
for p in orchestrator orchestrator-sdk catalog harness-sdk; do
  printf "%s " "@fabricstack/$p"; curl -s -o /dev/null -w "%{http_code}\n" "https://registry.npmjs.org/@fabricstack%2f$p"
done
# fabric's tags are only fabric/<branch>/v*
git ls-remote --tags origin | grep -v "refs/tags/fabric/" | grep -c "babysitter/"   # 0
# guards
node scripts/ci/publish-isolation-guard.cjs --self-test && node scripts/ci/public-push-guard.cjs --self-test