{
  "id": "page:docs-release-runbook",
  "_kind": "Page",
  "_file": "wiki/docs/release-runbook.md",
  "_cluster": "wiki",
  "attributes": {
    "nodeKind": "Page",
    "sourcePath": "docs/RELEASE-RUNBOOK.md",
    "sourceKind": "repo-docs",
    "title": "Release runbook — private train to Artifact Registry",
    "displayName": "Release runbook — private train to Artifact Registry",
    "slug": "docs/release-runbook",
    "articlePath": "wiki/docs/RELEASE-RUNBOOK.md",
    "article": "\n# Release runbook — private train to Artifact Registry\n\nStatus: operating procedure of the private train (WP-RN-07, spec 15 §4.2;\nrecord `docs/specs/platform-gaps/phase1/rn-07-release.md`).\nAudience: release owners of `a5c-ai/fabric`.\nModel reference: [docs/release-pipeline.md](./release-pipeline.md); versioning posture:\n[docs/VERSIONING.md](./VERSIONING.md); registry/credentials:\n[docs/user-guide/private-install.md](pathname:///docs/user-guide/private-install).\nHistorical (public-train, superseded): [docs/release-recovery-runbook.md](pathname:///docs/release-recovery-runbook).\n\n> **Registry mutations happen only in GitHub Actions through Workload Identity\n> Federation.** Nothing in this runbook asks a human to run `npm publish` or\n> `npm dist-tag add` from a laptop: a maintainer holds `roles/artifactregistry.reader`\n> at most, the CI service account alone holds `writer`\n> (`docs/specs/platform-gaps/phase0/ar-provisioning.md`). Every command marked\n> *local* is read-only or a dry run.\n\n> **Every `gh workflow run` in this runbook names `--ref`.** Without `--ref`, `gh` dispatches\n> the copy of the workflow file on the repository's **default branch** (`main`). Until PR #4\n> reaches `main` **and `staging`**, the copies of `publish-packages-from-tag.yml` and\n> `fix-npm-tags.yml` on **both** branches are the legacy public-train files (the same blob,\n> `44112b847`, for `fix-npm-tags.yml`: `registry-url: https://registry.npmjs.org/`,\n> `NODE_AUTH_TOKEN: secrets.NPM_TOKEN`, `babysitter/**` tags, `workflow_dispatch` default\n> `dry_run=false`) — a dispatch bound to them is a dispatch against the public registry, and\n> `--ref staging` is itself the legacy copy until the merge.\n> `--ref <release branch or fabric/ tag>` runs the file of that ref instead, which is correct\n> only once that ref carries this branch's file. Do not dispatch `fix-npm-tags.yml` until PR #4 is on `staging` and `main` and M7 is verified. It is `disabled_manually` until Gate B\n> re-enables it after the merge (section 0a, review-2 M8). The `--ref` rule is pinned by\n> `scripts/__tests__/private-release-train.test.mjs` (\"every `gh workflow run` in the runbooks\n> names its --ref\"), which also asserts the sentence above. Review M3 / review-2 M8, 2026-10-04.\n\n## 0. Invariants the train encodes\n\n| Invariant | Mechanism |\n|---|---|\n| One registry | every `@fabricstack/*` manifest pins `publishConfig.registry` to `https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/` (G1); the resolver emits the same `registry`; the publish helper names it explicitly on every `npm` call and refuses a manifest that disagrees; no workflow mentions the public registry host (G2(c)); no npmjs credential exists (G3) |\n| One version | `scripts/release-version.cjs resolve` runs exactly once per train (`resolve_release`); every later job validates or reads that identity (`verify`, `release-version.json`); nothing re-derives it |\n| Channels | `main` → `latest` as `X.Y.Z`; `staging` → `staging` and `develop` → `develop` as commit-pinned `X.Y.(Z+1)-<branch>.<sha>` prereleases (unchanged from the public train, spec 15 §4.1) |\n| Gate before publish | the strict 43-package packed-artifact matrix (`release-artifact-verifier.yml`, `allow_known_failures: 'false'`) must succeed before `publish_candidates` runs |\n| Candidate first | publication writes only `candidate-<version>`; the channel moves in `promote_release_channel` after the published-consumer validation of that exact version |\n| Tag last | `fabric/<branch>/v<version>` is created on `a5c-ai/fabric` only after promotion and the full-inventory channel assertion; the tag is immutable and idempotent (`ensure-tag`) |\n| Public namespace untouched | the public repository keeps `babysitter/<branch>/v*`; a `babysitter/*` tag is rejected by the resolver and the helper; the public-push guard refuses any write to `a5c-ai/babysitter` |\n| Version shape of the 7.0.0 train | the staging train publishes **`7.0.1-staging.<sha>`** under dist-tag `staging` and tags **`fabric/staging/v7.0.1-staging.<sha>`**; the exact **`7.0.0`** (`latest`, tag `fabric/main/v7.0.0`) is the later main (promotion) train. The resolver rejects `fabric/staging/v7.0.0` by design and its message names both shapes. Roadmap M1 criterion 1 / WP-RM-02 AC1 / risk 7 were amended to this on 2026-10-04 (owner decision: \"Spec 15 wins\") |\n\n## 0a. Gate-B prerequisites and state (2026-10-04)\n\n| Item | State | How to verify |\n|---|---|---|\n| Live-model lanes gate — **owner decision 2026-10-05, \"Consumer-only gate for the first train\"** | Promotion of the first private train depends on the EXACT-VERSION `published_consumer` job (install/import/bin smoke of all 43 + the §3.1 global-install smoke) and the recorded verdict; the live-model lanes still run, upload their artifacts and are recorded in `validation.json` (`live-stack` check, `liveLanesGate`, `informationalFailures`) but do not block while the repository/organization **variable** `FABRIC_LIVE_LANES_GATE` is unset or `informational`. Setting it to `required` restores the full FIX-010 gate (promotion then also needs the called workflow's aggregate result green). Owner's stated condition for flipping it back: a valid `OPENAI_API_KEY` organization secret (section 7.2). Background: run 37279009004 — consumer job green, every `openai/gpt-5.5` lane red with `invalid_api_key`. Note: a run's overall conclusion stays `failure` while a lane is red even when it promoted and tagged — read the `promote_release_channel` / `assert_release_channel_tags` / `create_release_tag` jobs, not the run badge. | `gh api repos/a5c-ai/fabric/actions/variables/FABRIC_LIVE_LANES_GATE --jq .value` (404 = unset = informational); the run's `Record Published-Consumer Validation` log (`::notice::… informational check(s) live-stack did not succeed; not blocking …`); `validation.json` in the `published-consumer-validation` artifact |\n| Whole-train rehearsal before the merge (review M1) | **done, in CI on every PR**: `ci.yml` job `release-train-rehearsal` runs the train's `publish_candidates` sequence for the staging identity of the PR head — resolve, release plan, sync + verify every manifest, wave build, wave `publish --rehearsal` (= `npm publish --dry-run` through the helper, zero registry traffic) — and asserts it reached the last wave with all 43 packages dry-run-published and no publish command without `--dry-run`. Run URL recorded in `docs/specs/platform-gaps/phase1/rn-07-release.md` §6 | `gh run view <ci run> -R a5c-ai/fabric --json jobs --jq '.jobs[] \\| select(.name \\| startswith(\"Release train rehearsal\")) \\| .conclusion'` → `success` |\n| First real run of `publish-npm-private.yml` | **is the staging merge of PR #4** (a push to `staging` sets `dry_run=false`). The workflow is not registered on any branch GitHub can dispatch from until that push, so no `workflow_dispatch` rehearsal of it can precede the merge; the rehearsal evidence is the job above plus the verifier's 43/43 | `gh api repos/a5c-ai/fabric/actions/workflows/publish-npm-private.yml` → 404 until the merge |\n| Workflows the train calls or this runbook dispatches (review M2, amended by review-2 M8) | **enabled 2026-10-04** by the owner at the pre-Gate-B interview (\"Enable all four now\"): `live-stack-published.yml`, `release-tags.yml`, `publish-packages-from-tag.yml` are `active`, as is `release-artifact-verifier.yml`. **`fix-npm-tags.yml` was re-DISABLED by the orchestrator the same day** (workflow 352401959, `disabled_manually`, 0 runs ever) and is re-enabled by the orchestrator immediately after PR #4 merges to `staging`, as part of Gate B, once M7 is verified. Why: no *trigger* fires pre-merge (the legacy copies on `main`/`staging` are `workflow_call`/dispatch-only or trigger on `babysitter/**` tags), **but a MANUAL dispatch of `fix-npm-tags.yml` without `--ref`, or with `--ref main`, or with `--ref staging` before the merge, would run the legacy npmjs copy** (identical blob `44112b847` on both branches) whose `dry_run` input defaults to `false` — a live `--fix` against the public `@a5c-ai/*` dist-tags with the org-wide `NPM_TOKEN`. Do not dispatch `fix-npm-tags.yml` until PR #4 is on `staging` and `main` and M7 is verified. This is also why every dispatch here carries `--ref` | `gh api repos/a5c-ai/fabric/actions/workflows --paginate --jq '.workflows[] \\| \"\\(.state) \\(.path)\"'` → `active` for the four above, `disabled_manually` for `fix-npm-tags.yml` until Gate B; `gh api repos/a5c-ai/fabric/actions/workflows/352401959/runs --jq .total_count` → `0` |\n| Human reader binding on `fabric-npm` (review M6) | **verified 2026-10-04** by the orchestrator with a fresh `gcloud` token: the IAM policy shows `roles/artifactregistry.reader` for `user:tal@a5c.ai`, `npm view @fabricstack/smoke` through `google-artifactregistry-auth` printed `0.0.0-smoke.4`, and a direct bearer-token `GET` answered `200`; the 2026-10-03 E403 was an expired token, not a missing binding. **Prerequisite kept: re-authenticate first** (section 6; tokens live about an hour). The *local* read-only commands in sections 1, 2.6, 3, 4.1, 4.4 and 6 are usable after that; the run's uploaded artifacts (`publish-npm-private-<version>`, `published-consumer-validation`, `release-train-rehearsal-<version>`) remain the CI-side evidence of the same facts | `gcloud artifacts repositories get-iam-policy fabric-npm --location=us-central1 --format=json` shows `roles/artifactregistry.reader` for `user:tal@a5c.ai`, **and** `npx -y google-artifactregistry-auth --repo-config=.npmrc --credential-config=\"$HOME/.npmrc\" && npm view @fabricstack/smoke --@fabricstack:registry=https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/` → `0.0.0-smoke.4` (both passed 2026-10-04) |\n| Organization secret `NPM_TOKEN` restricted away from fabric (G3, review M7) | **owner action, required BEFORE anyone dispatches `fix-npm-tags.yml` — i.e. before its Gate-B re-enable** (review-2 M8 moved it forward from \"before Gate B\"; still `all` on 2026-10-04): restrict the secret's repository selection in the GitHub UI to the public-publishing repositories; fabric unselected | `gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN --jq .visibility` → `selected`, and `gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN/repositories --jq '.repositories[].name'` does not list `fabric` |\n\n## 1. Normal train (push to `staging`, `main` or `develop`)\n\nWorkflow: `.github/workflows/publish-npm-private.yml` (job graph below). Nothing to run by\nhand; the steps are listed so a reviewer can follow a run.\n\n```\nresolve_release          resolve ONE identity (version, dist-tag, fabric/ tag, registry, candidate tag)\n                         verify-manifests on the checked-in tree; publish-isolation guards (G1-G3)\nverify_release_artifacts release-artifact-verifier.yml: all 43 packages, no tolerance (FIX-011)\npublish_candidates       needs + checks the verifier result; WIF auth -> google-artifactregistry-auth\n                         release plan written + validated; sync-workspace-versions + verify-manifests\n                         + publish-isolation-guard over the synchronized tree; preflight channel assertion\n                         build the train in derived dependency waves; verify SDK package health\n                         publish every package under candidate-<version> (FABRIC_RELEASE_PUBLISH=1)\n                         assert every candidate dist-tag == version\npublished_consumer_validation   live-stack-published.yml with the EXACT version (section 3)\npromote_release_channel  release-promotion.cjs promote --evidence validation.json (the only dist-tag move)\n                         gated on the EXACT-VERSION consumer job + the recorded verdict (callee outputs\n                         consumer_result / promotable); the live-model lanes are informational unless the\n                         variable FABRIC_LIVE_LANES_GATE=required (owner decision 2026-10-05, section 0a)\nassert_release_channel_tags     every package's channel tag == version (FIX-001 final assertion)\ncreate_release_tag       release-tags.yml: fabric/<branch>/v<version>, provenance fabric-release-*\n```\n\nWhat to check after a green run (read-only). Prerequisite: re-authenticate first (section 6;\nM6 verified 2026-10-04) — the run's uploaded artifacts are the CI-side evidence of the same\nfacts.\n\n```bash\n# the identity the run resolved (also in the run's step summary)\nSHORT_SHA=$(git rev-parse --short=12 <released commit>)\nnode scripts/release-version.cjs resolve --branch staging --sha \"$SHORT_SHA\"\n# channel state on Artifact Registry over ALL 43 packages (reader credential, section 6)\nnode scripts/release-version.cjs assert-channel-tags --version <version> --dist-tag staging --mode final\n# the tag exists here and nowhere else\ngit ls-remote --tags origin \"refs/tags/fabric/staging/v<version>\"\ngit ls-remote --tags https://github.com/a5c-ai/babysitter.git | grep -c \"fabric/\"   # must be 0\n```\n\n## 2. Dry-run commands (local, nothing written anywhere)\n\n```bash\n# 1. the identity the train would use for this commit on each channel\nSHORT_SHA=$(git rev-parse --short=12 HEAD)\nnode scripts/release-version.cjs resolve --branch main    --sha \"$SHORT_SHA\"\nnode scripts/release-version.cjs resolve --branch staging --sha \"$SHORT_SHA\"\nnode scripts/release-version.cjs verify  --branch main    --sha \"$SHORT_SHA\" --version \"$(node -p \"require('./package.json').version\")\"\n\n# 2. the checked-in train is lockstep with exact internal pins (G2(d))\nnode scripts/release-version.cjs verify-manifests --version \"$(node -p \"require('./package.json').version\")\"\n\n# 3. publish-isolation (G1-G3), public-push guard, metadata/coverage gate\nnode scripts/ci/publish-isolation-guard.cjs --self-test\nnode scripts/ci/public-push-guard.cjs --self-test\nnpm run verify:metadata\n\n# 4. the derived build/publication waves\nnode scripts/release-matrix.cjs --group all-publishable --format waves\n\n# 5. release tooling + workflow-shape suites (FIX-001/005/010/011 + WP-RN-07)\nnpm run test:release-tooling\n\n# 6. rehearse the publish helper for one package: npm publish --dry-run under the\n#    candidate tag, nothing written. --dry-run queries Artifact Registry for the\n#    package's own existence (READER credential, section 6 — re-authenticate first) and\n#    accepts an exact internal pin that a workspace of this checkout carries at\n#    that version (\"published by an earlier wave of this train\"); any other pin\n#    is still looked up on the registry. --rehearsal (implies --dry-run) issues\n#    NO registry query at all, so it needs no credential — this is what the CI\n#    rehearsal job runs for every package.\nGITHUB_REF_NAME=main node scripts/publish-package-from-tag.mjs \\\n  --workspace=@fabricstack/trust --release-version=\"$(node -p \"require('./package.json').version\")\" --skip-build --dry-run\nGITHUB_REF_NAME=main node scripts/publish-package-from-tag.mjs \\\n  --workspace=@fabricstack/harness-policy --release-version=\"$(node -p \"require('./package.json').version\")\" --skip-build --rehearsal\n\n# 7. the full packed-artifact matrix (slow; what the release gate runs)\nnode scripts/verify-release-artifacts.mjs --report-dir artifacts/release-verifier --exactness\n```\n\nWhole-train rehearsal in CI — two forms:\n\n1. **Before the merge (every PR):** the `release-train-rehearsal` job of `ci.yml` (section 0a)\n   executes the train's `publish_candidates` sequence end-to-end for the staging identity of the\n   PR head, with `--rehearsal` in place of the publication opt-in, and proves it reached the last\n   of the 13 waves with all 43 packages dry-run-published and no publish command without\n   `--dry-run`. The registry-facing steps (preflight channel assertion, candidate dist-tag\n   assertion) and the packed-artifact gate are logged as skipped there: the verifier runs as its\n   own PR workflow (`release-artifact-verifier.yml`), the two assertions run in the train.\n2. **After the merge:** `gh workflow run publish-npm-private.yml -R a5c-ai/fabric --ref staging`\n   with the default `dry_run=true` runs every job up to and including `npm publish --dry-run` for\n   all 43 packages (the helper's `--dry-run` satisfies exact internal pins from the synchronized\n   checkout, so an empty registry no longer stops wave 2), then stops before\n   validation/promotion/tagging. A dispatch publishes only when `dry_run` is explicitly `false`.\n\n**The first real run of `publish-npm-private.yml` is the staging merge itself** (a push to\n`staging` runs with `dry_run=false`); it cannot be dispatched before the file is on a branch\nGitHub registers it from. Gate B therefore relies on form 1 plus the verifier's 43/43 as the\nrehearsal evidence (`docs/specs/platform-gaps/phase1/rn-07-release.md` §6).\n\n## 3. Published-consumer validation (what gates promotion)\n\n`.github/workflows/live-stack-published.yml`, called with the exact version and the candidate\ndist-tag, never a channel:\n\n- `published_consumer`: fresh project outside the repository with its own `.npmrc`\n  (`@fabricstack:registry=<Artifact Registry>` — written by\n  `scripts/verify-published-release.mjs`), all 43 packages installed at `@<version>`, every root\n  and exported runtime subpath imported, every declared bin smoked; then the spec 15 §3.1\n  global-install smoke (WP-RN-06 AC2): `npm i -g @fabricstack/orchestrator@V @fabricstack/harness-cli@V`\n  with per-call scope routing (`--@fabricstack:registry=<AR>`), no `EEXIST`, exactly the\n  `orchestrator` bin; then `@fabricstack/harness-hooks-cli@V @fabricstack/harness-sdk@V` → exactly\n  `harness harness-hooks orchestrator`; each answers `--version`.\n- live-stack lanes: the §3.1 CLIs (`orchestrator`, `orchestrator-sdk`, `harness-sdk`,\n  `agent-runtime-cli`, `harness-hooks-cli`) installed globally from Artifact Registry at the exact\n  version, asserted with `release-promotion.cjs assert-installed`, then the live stack.\n- `record_validation` writes `validation.json` (checks `package-install`, `root-import`,\n  `subpath-import`, `bin-smoke`, `live-stack`); `release-promotion.cjs promote` refuses without it.\n- **Gate (owner decision 2026-10-05, \"Consumer-only gate for the first train\")**: every check must be\n  RECORDED and every *gating* check green. `live-stack` is gating only when the evidence was recorded\n  with `--live-lanes-gate required`, which `record_validation` derives from the repository/organization\n  variable `FABRIC_LIVE_LANES_GATE` (unset or `informational` → consumer-only gate, the lanes' failure is\n  listed under `informationalFailures`; `required` → the full gate; the script's own default is\n  `required`). The caller's `promote_release_channel` evaluates after a red lane (`!cancelled()`) and\n  checks, explicitly, the resolver and publish results, the consumer job's own result\n  (`consumer_result == 'success'`), the recorded verdict (`promotable == 'true'`) and, under `required`,\n  the called workflow's aggregate result. No lane is `continue-on-error`. Condition to restore the full\n  gate: a valid `OPENAI_API_KEY` (section 7.2).\n\nLocal equivalent of the consumer half (reader credential, section 6 — re-authenticate first;\nthe `published-consumer-validation` artifact is the CI-side evidence):\n\n```bash\nnode scripts/verify-published-release.mjs --version <version> --report-dir artifacts/published-consumer\n```\n\n## 4. Recovering a half-published train\n\nSymptom: `publish_candidates` failed in wave N. Waves `< N` are on Artifact Registry under\n`candidate-<version>`; **wave N itself is partially published** — its helpers run in parallel\nand `wait` collects every sibling before the step exits, so the members that succeeded are on\nthe registry and the failed ones are not; waves `> N` are missing; **no channel tag has moved**\nand **no release tag exists** (both happen after validation). Nothing is broken for consumers:\nchannels still resolve the previous release (or nothing, on a first train).\n\n**First staging train (`7.0.1-staging.<sha>`) — abort is fix-forward only once `staging` has\nbeen promoted.** Before promotion there is nothing to roll back: the candidate versions are\norphaned and no channel points at them. After promotion there is no earlier version on\nArtifact Registry for `--set staging=<previous-version>` (section 5; the script refuses a version\nthat any package lacks), so the only abort is a new commit → a new train that moves `staging`\nforward again. Consumers of the brand-new `@fabricstack` scope are unaffected either way:\nnothing is pinned to it yet, and `latest` never moves on a staging train.\n\n### 4.1 Find out what is on the registry (local, read-only)\n\nRe-authenticate first (section 6). The uploaded `publish-npm-private-<version>` artifact (one\nhelper log per package: `published …` / `already exists …` / the failure) is the same\ninventory, read from CI.\n\n```bash\nV=<version>\nAR=$(node -p \"require('./package.json').fabric.privateRegistry\")\n# which packages have the exact version (prints MISSING for the unpublished half)\nnode scripts/release-matrix.cjs --group all-publishable --format workspaces | while read -r p; do\n  if npm view \"$p@$V\" version --registry \"$AR\" >/dev/null 2>&1; then echo \"PUBLISHED $p\"; else echo \"MISSING $p\"; fi\ndone\n# candidate dist-tag state over the whole inventory (final mode fails while any package is missing — that is the diagnosis)\nnode scripts/release-version.cjs assert-channel-tags --version \"$V\" --dist-tag \"$(node scripts/release-promotion.cjs candidate-tag --version \"$V\")\" --mode final\n```\n\n### 4.2 Decide\n\n- **Cause was transient** (registry hiccup, runner died): **first re-run the failed\n  `publish_candidates` job on the same run** (`gh run rerun <run-id> -R a5c-ai/fabric\n  --failed`). It is idempotent — the helper only ensures the candidate dist-tag of a version\n  that already exists and publishes the missing ones — and the same run then continues into\n  validation, promotion, assertion and the tag, so nothing is re-derived. Do **not** push an\n  empty commit: the branch train resolves a *new* version for a new commit. Use 4.3 when the run\n  itself is dead (expired, cancelled, or failed a second time) — it acts on an immutable tag.\n- **Cause was a package defect**: fix it on the branch; the next push resolves a new version and\n  publishes a complete train. The orphaned `candidate-<old>` versions stay installable for\n  diagnosis and are never promoted; nothing needs deleting (channels never pointed at them).\n\n### 4.3 Recover from the immutable tag (`publish-packages-from-tag.yml`) — APPROVAL REQUIRED\n\nRecovery acts on a `fabric/<branch>/v<version>` tag so the tree is immutable. The branch train\nonly tags after promotion, so for a half-published train the tag does not exist yet: create it\nlocally **at the released commit**, push it, and let the tag workflow do the rest.\n\n```bash\n# at the exact commit the failed run built (check the run's \"Release identity\" summary)\ngit checkout <commit>\nSHORT_SHA=$(git rev-parse --short=12 HEAD)\nnode scripts/release-version.cjs verify --branch staging --sha \"$SHORT_SHA\" --version \"$V\"   # must agree\n# local, idempotent, annotated, provenance source=manual (so the tag workflow knows it may publish)\nnode scripts/release-version.cjs ensure-tag --version \"$V\" --branch staging --commit \"$(git rev-parse HEAD)\" --source manual\nnode scripts/release-version.cjs tag-source --tag \"fabric/staging/v$V\"      # published_by_release_workflow=false\n# APPROVAL REQUIRED: pushing the tag starts the recovery publication\ngit push origin \"refs/tags/fabric/staging/v$V\"\n# or, for an existing tag — --ref is mandatory (see the note at the top): dispatch from the\n# tag itself or from `staging`/`main`, never from a feature branch whose name contains `/`\n# (the helper refuses any `/`-shaped ref outside fabric/<branch>/v<version> as a channel source)\ngh workflow run publish-packages-from-tag.yml -R a5c-ai/fabric --ref \"fabric/staging/v$V\" -f tag=\"fabric/staging/v$V\"\n```\n\nWhat the recovery lane does (all automatic, all idempotent):\n\n1. `release_identity`: derives version/channel/registry from the tag; refuses a tag the branch\n   train created itself (`published_by_release_workflow=true`) unless manually dispatched.\n2. `verify_release_artifacts`: the strict matrix against the tag's tree.\n3. `publish_non_plugin_packages`: WIF auth, release plan from the tag, sync + verify every\n   manifest, preflight channel assertion, build in derived waves, publish in derived waves under\n   `candidate-<version>`. **Packages whose exact version already exists are not re-published —\n   the helper only ensures their candidate dist-tag** (`scripts/publish-package-from-tag.mjs`),\n   so the already-published half is a no-op and the missing half completes. Then every\n   candidate dist-tag is asserted.\n4. `published_consumer_validation` → `promote_release_channel` → channel assertion, exactly as\n   the branch train. Re-running the recovery for the same tag after a second failure is safe for\n   the same reason.\n\nIdempotency guarantees, by mechanism: `isPublished` (E404-only \"absent\" detection; any other\nregistry answer aborts rather than guesses), `ensure-tag` (never re-points a tag),\n`assert-channel-tags --mode preflight` (never moves a channel backward),\n`release-promotion.cjs promote` (refuses without evidence naming this exact version).\n\n### 4.4 After recovery\n\nLocal, read-only (re-authenticate first, section 6); the recovery run's\n`assert_release_channel_tags` job is the CI-side evidence.\n\n```bash\nnode scripts/release-version.cjs assert-channel-tags --version \"$V\" --dist-tag staging --mode final\ngit ls-remote --tags origin \"refs/tags/fabric/staging/v$V\"\n```\n\n## 5. Repairing dist-tags (`fix-npm-tags.yml`) — APPROVAL REQUIRED for `dry_run=false`\n\nScript: `scripts/fix-broken-latest-tags.mjs`, inventory-derived, private registry named on\nevery call. Two modes, both dry runs unless `--fix` / `dry_run=false`:\n\n```bash\n# audit: latest -> channel prerelease, channel tag -> empty artifact, missing latest; proposes the last healthy release\nnode scripts/fix-broken-latest-tags.mjs\nnode scripts/fix-broken-latest-tags.mjs --json\n# explicit repair of one dist-tag for EVERY package at one exact version (refuses if any package lacks that version)\nnode scripts/fix-broken-latest-tags.mjs --set staging=<version>\n# narrow\nnode scripts/fix-broken-latest-tags.mjs --package @fabricstack/catalog\n```\n\nIn CI (`--ref` mandatory, see the note at the top). **Until PR #4 is merged, the copies of this\nworkflow on `main` AND `staging` are the legacy npmjs file** (identical blob `44112b847`):\n`workflow_dispatch` input `dry_run` defaults to `false`, `checkout ref: staging`,\n`registry-url: https://registry.npmjs.org/`, `NODE_AUTH_TOKEN: secrets.NPM_TOKEN`, and the\nnon-dry branch runs `scripts/fix-broken-latest-tags.mjs --fix` over the public `@a5c-ai/*`\npackages — so `--ref staging` is only correct after the merge, and a dispatch with no `--ref`,\nwith `--ref main`, or with `--ref staging` before the merge would be a live public dist-tag\nmutation (the forms below carry `-f mode=…`, which the legacy inputs reject with 422, but a\nbare dispatch would not be rejected). Do not dispatch `fix-npm-tags.yml` until PR #4 is on `staging` and `main` and M7 is verified. Mitigation in force: the\nworkflow is `disabled_manually` (re-disabled by the orchestrator 2026-10-04, review-2 M8) and is\nre-enabled by the orchestrator as part of Gate B, right after the merge to `staging` and after\nM7's verification (section 0a). Once re-enabled:\n`gh workflow run fix-npm-tags.yml -R a5c-ai/fabric --ref staging -f mode=audit` (dry run), then\n`gh workflow run fix-npm-tags.yml -R a5c-ai/fabric --ref staging -f mode=audit -f dry_run=false`\nonce the plan is approved;\n`gh workflow run fix-npm-tags.yml -R a5c-ai/fabric --ref staging -f mode=set -f set_spec=latest=<version>`\nfor the explicit repair. The audit mode needs no human registry credential (the workflow\nauthenticates through WIF), so it is the diagnosis tool whenever a human token is stale. A\nchannel rollback is the same operation with the previously validated version:\n`--set latest=<previous-version>` — publication is immutable, the dist-tag is the only\nreversible surface. Never unpublish; recover forward. **On the FIRST staging train there is no\n`<previous-version>` on Artifact Registry** (the script refuses a version any package lacks), so\na post-promotion abort of that train is fix-forward only — a new commit, a new train (section 4).\n\n## 6. Credentials for the read-only local commands\n\n**Review M6 — verified 2026-10-04 (section 0a).** On 2026-10-03 the human credential of\n`tal@a5c.ai` received `403 Forbidden` from Artifact Registry on every npm read (`npm view` of an\nexisting and of a made-up package alike, and a direct bearer-token `GET`), although the phase-0\nrecord lists the account as `roles/artifactregistry.reader`. The cause was an expired access\ntoken: with a fresh `gcloud` token the orchestrator ran the verification below and all of it\npassed (the policy shows `roles/artifactregistry.reader` for `user:tal@a5c.ai`;\n`npm view @fabricstack/smoke` → `0.0.0-smoke.4`; the direct bearer `GET` → `200`).\n**Prerequisite kept: re-authenticate first** — run the `google-artifactregistry-auth` line\nbefore any local read-only command in this runbook, and again when a read starts answering\n401/403. The verification, re-runnable at any time:\n\n```bash\ngcloud artifacts repositories get-iam-policy fabric-npm --location=us-central1 --format=json   # roles/artifactregistry.reader for user:tal@a5c.ai\nnpx -y google-artifactregistry-auth --repo-config=.npmrc --credential-config=\"$HOME/.npmrc\"\nnpm view @fabricstack/smoke --@fabricstack:registry=https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/   # must succeed\n```\n\nNo workflow depends on the human credential: all of them authenticate as the CI service account\nthrough WIF, which published in `ar-smoke` run 36454074285. If the three lines above ever fail\nagain, the run's uploaded artifacts carry the same facts and every *local* read-only command in\nthis runbook falls back to CI-only until the credential is repaired.\n\n```bash\ngcloud auth login            # once; tal@a5c.ai holds roles/artifactregistry.reader on fabric-npm\nnpx -y google-artifactregistry-auth --repo-config=.npmrc --credential-config=\"$HOME/.npmrc\"\nnpm ping --registry https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/\ngit diff --exit-code -- .npmrc   # the committed file never gains a credential line\n```\n\nTokens live about an hour; rerun the helper when `npm view` starts answering 401/403. A 403 on a\npackage you expect to exist means the active gcloud account lacks the reader binding\n(`gcloud auth list`), not that the package is absent.\n\n## 7. Follow-ups attached to the first publications\n\n### 7.1 Pin the newly published agent-runtime hooks leaf in the hooks CLI — REQUIRED FOLLOW-UP\n\n`@fabricstack/harness-hooks-agent-runtime` has never been published, so\n`packages/adapters/hooks/cli/package.json` deliberately does not pin it yet\n(`//deferred-dependency`, FIX-005): an exact pin on an unpublished version would fail the\nclean-consumer verification and the publish helper's exact-internal-dependency gate. Immediately\nafter the first train that publishes it:\n\n```bash\n# AFTER @fabricstack/harness-hooks-agent-runtime@$RELEASE_VERSION exists on Artifact Registry:\nnpm view \"@fabricstack/harness-hooks-agent-runtime@$RELEASE_VERSION\" version --registry \"$AR\"   # must succeed\n# add the exact pin next to the other hooks leaves, drop the //deferred-dependency note\nnode -e \"const fs=require('fs');const p='packages/adapters/hooks/cli/package.json';const m=JSON.parse(fs.readFileSync(p));m.dependencies['@fabricstack/harness-hooks-agent-runtime']=m.version;delete m['//deferred-dependency'];fs.writeFileSync(p,JSON.stringify(m,null,2)+'\\n')\"\nnpm install --package-lock-only --legacy-peer-deps\nnode scripts/release-version.cjs verify-manifests --version \"$(node -p \"require('./package.json').version\")\"\nnpm run test:release-tooling   # the FIX-005 test flips to guarding the pin\n```\n\n### 7.2 Pending owner actions (recorded, not automated)\n\n- **Organization secret `OPENAI_API_KEY` is invalid — owner action (found 2026-10-05, publish run\n  37279009004)**: the secret is delivered to fabric (visibility `all`, non-empty in the job) and\n  rejected by OpenAI. pi lane, `agent-output.txt`:\n  `OpenAI API error (401): {\"message\":\"Incorrect API key provided: AcuYB8hq************…JuS3. …\",\"type\":\"invalid_request_error\",\"code\":\"invalid_api_key\"}`;\n  codex lane (same credential through codex 0.160.0's websocket transport):\n  `401 Unauthorized: Missing bearer or basic authentication in header, url: https://api.openai.com/v1/responses`.\n  Every default live lane is `openai/gpt-5.5`, so no lane can pass until the secret is replaced. Then set\n  the variable `FABRIC_LIVE_LANES_GATE=required` to restore the full gate (section 0a). Verification: a\n  green `Published (…, codex/gpt-5.5, non-interactive)` lane on the next train, and `validation.json`\n  with `liveLanesGate: \"required\"` and `informationalFailures: []`.\n- **G3 / review M7 — owner action, required BEFORE anyone dispatches `fix-npm-tags.yml`, i.e.\n  before its Gate-B re-enable** (review-2 M8 moved it forward from \"before Gate B\", 2026-10-04):\n  restrict the organization secret `NPM_TOKEN` to the repositories that publish to npmjs (fabric\n  unselected): `docs/specs/platform-gaps/phase0/g3-npm-token-restriction.md`. Fabric's branch\n  workflows no longer reference it, but the legacy copies on `main` and `staging` still do (which\n  is what makes `--ref` mandatory above and why `fix-npm-tags.yml` stays disabled until Gate B),\n  and the secret is currently visible to all repositories (`gh api orgs/a5c-ai/actions/secrets --jq '.secrets[] | \"\\(.name) \\(.visibility)\"'`\n  → `NPM_TOKEN all`). Verification: `gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN --jq .visibility`\n  → `selected`, and `gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN/repositories --jq '.repositories[].name'`\n  without `fabric`.\n- **Review M6 — verified 2026-10-04**: the human reader binding on `fabric-npm` works with a\n  fresh token (section 6); re-authenticate before local reads.\n- **G4** — register the `fabricstack` organization on public npmjs (publish nothing, 2FA,\n  restricted membership): `docs/specs/platform-gaps/phase0/g4-npmjs-org-runbook.md`;\n  `node scripts/ci/publish-isolation-guard.cjs --g4-status` prints the current state.\n\n## 8. Public-namespace invariant checks (run after any release activity)\n\n```bash\n# the public repository was not written to\ngh api repos/a5c-ai/babysitter --jq .pushed_at                     # unchanged\ngit ls-remote --tags https://github.com/a5c-ai/babysitter.git | grep -c \"fabric/\"   # 0\ngit ls-remote --tags https://github.com/a5c-ai/babysitter.git | wc -l               # unchanged count\n# nothing under the private scope exists on the public registry (404 everywhere)\nfor p in orchestrator orchestrator-sdk catalog harness-sdk; do\n  printf \"%s \" \"@fabricstack/$p\"; curl -s -o /dev/null -w \"%{http_code}\\n\" \"https://registry.npmjs.org/@fabricstack%2f$p\"\ndone\n# fabric's tags are only fabric/<branch>/v*\ngit ls-remote --tags origin | grep -v \"refs/tags/fabric/\" | grep -c \"babysitter/\"   # 0\n# guards\nnode scripts/ci/publish-isolation-guard.cjs --self-test && node scripts/ci/public-push-guard.cjs --self-test\n```\n",
    "documents": []
  },
  "outgoingEdges": [],
  "incomingEdges": [
    {
      "from": "page:docs",
      "to": "page:docs-release-runbook",
      "kind": "contains_page"
    }
  ]
}