II.
Page overview
Reference · livepage:docs-release-runbook
Release runbook — private train to Artifact Registry overview
Inspect the raw attributes, linked wiki pages, and inbound or outbound graph edges for page:docs-release-runbook.
Attributes
nodeKind
Page
sourcePath
docs/RELEASE-RUNBOOK.md
sourceKind
repo-docs
title
Release runbook — private train to Artifact Registry
displayName
Release runbook — private train to Artifact Registry
slug
docs/release-runbook
articlePath
wiki/docs/RELEASE-RUNBOOK.md
article
# Release runbook — private train to Artifact Registry
Status: operating procedure of the private train (WP-RN-07, spec 15 §4.2;
record `docs/specs/platform-gaps/phase1/rn-07-release.md`).
Audience: release owners of `a5c-ai/fabric`.
Model reference: [docs/release-pipeline.md](./release-pipeline.md); versioning posture:
[docs/VERSIONING.md](./VERSIONING.md); registry/credentials:
[docs/user-guide/private-install.md](pathname:///docs/user-guide/private-install).
Historical (public-train, superseded): [docs/release-recovery-runbook.md](pathname:///docs/release-recovery-runbook).
> **Registry mutations happen only in GitHub Actions through Workload Identity
> Federation.** Nothing in this runbook asks a human to run `npm publish` or
> `npm dist-tag add` from a laptop: a maintainer holds `roles/artifactregistry.reader`
> at most, the CI service account alone holds `writer`
> (`docs/specs/platform-gaps/phase0/ar-provisioning.md`). Every command marked
> *local* is read-only or a dry run.
> **Every `gh workflow run` in this runbook names `--ref`.** Without `--ref`, `gh` dispatches
> the copy of the workflow file on the repository's **default branch** (`main`). Until PR #4
> reaches `main` **and `staging`**, the copies of `publish-packages-from-tag.yml` and
> `fix-npm-tags.yml` on **both** branches are the legacy public-train files (the same blob,
> `44112b847`, for `fix-npm-tags.yml`: `registry-url: https://registry.npmjs.org/`,
> `NODE_AUTH_TOKEN: secrets.NPM_TOKEN`, `babysitter/**` tags, `workflow_dispatch` default
> `dry_run=false`) — a dispatch bound to them is a dispatch against the public registry, and
> `--ref staging` is itself the legacy copy until the merge.
> `--ref <release branch or fabric/ tag>` runs the file of that ref instead, which is correct
> only once that ref carries this branch's file. Do not dispatch `fix-npm-tags.yml` until PR #4 is on `staging` and `main` and M7 is verified. It is `disabled_manually` until Gate B
> re-enables it after the merge (section 0a, review-2 M8). The `--ref` rule is pinned by
> `scripts/__tests__/private-release-train.test.mjs` ("every `gh workflow run` in the runbooks
> names its --ref"), which also asserts the sentence above. Review M3 / review-2 M8, 2026-10-04.
## 0. Invariants the train encodes
| Invariant | Mechanism |
|---|---|
| One registry | every `@fabricstack/*` manifest pins `publishConfig.registry` to `https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/` (G1); the resolver emits the same `registry`; the publish helper names it explicitly on every `npm` call and refuses a manifest that disagrees; no workflow mentions the public registry host (G2(c)); no npmjs credential exists (G3) |
| One version | `scripts/release-version.cjs resolve` runs exactly once per train (`resolve_release`); every later job validates or reads that identity (`verify`, `release-version.json`); nothing re-derives it |
| Channels | `main` → `latest` as `X.Y.Z`; `staging` → `staging` and `develop` → `develop` as commit-pinned `X.Y.(Z+1)-<branch>.<sha>` prereleases (unchanged from the public train, spec 15 §4.1) |
| Gate before publish | the strict 43-package packed-artifact matrix (`release-artifact-verifier.yml`, `allow_known_failures: 'false'`) must succeed before `publish_candidates` runs |
| Candidate first | publication writes only `candidate-<version>`; the channel moves in `promote_release_channel` after the published-consumer validation of that exact version |
| Tag last | `fabric/<branch>/v<version>` is created on `a5c-ai/fabric` only after promotion and the full-inventory channel assertion; the tag is immutable and idempotent (`ensure-tag`) |
| Public namespace untouched | the public repository keeps `babysitter/<branch>/v*`; a `babysitter/*` tag is rejected by the resolver and the helper; the public-push guard refuses any write to `a5c-ai/babysitter` |
| Version shape of the 7.0.0 train | the staging train publishes **`7.0.1-staging.<sha>`** under dist-tag `staging` and tags **`fabric/staging/v7.0.1-staging.<sha>`**; the exact **`7.0.0`** (`latest`, tag `fabric/main/v7.0.0`) is the later main (promotion) train. The resolver rejects `fabric/staging/v7.0.0` by design and its message names both shapes. Roadmap M1 criterion 1 / WP-RM-02 AC1 / risk 7 were amended to this on 2026-10-04 (owner decision: "Spec 15 wins") |
## 0a. Gate-B prerequisites and state (2026-10-04)
| Item | State | How to verify |
|---|---|---|
| Live-model lanes gate — **owner decision 2026-10-05, "Consumer-only gate for the first train"** | Promotion of the first private train depends on the EXACT-VERSION `published_consumer` job (install/import/bin smoke of all 43 + the §3.1 global-install smoke) and the recorded verdict; the live-model lanes still run, upload their artifacts and are recorded in `validation.json` (`live-stack` check, `liveLanesGate`, `informationalFailures`) but do not block while the repository/organization **variable** `FABRIC_LIVE_LANES_GATE` is unset or `informational`. Setting it to `required` restores the full FIX-010 gate (promotion then also needs the called workflow's aggregate result green). Owner's stated condition for flipping it back: a valid `OPENAI_API_KEY` organization secret (section 7.2). Background: run 37279009004 — consumer job green, every `openai/gpt-5.5` lane red with `invalid_api_key`. Note: a run's overall conclusion stays `failure` while a lane is red even when it promoted and tagged — read the `promote_release_channel` / `assert_release_channel_tags` / `create_release_tag` jobs, not the run badge. | `gh api repos/a5c-ai/fabric/actions/variables/FABRIC_LIVE_LANES_GATE --jq .value` (404 = unset = informational); the run's `Record Published-Consumer Validation` log (`::notice::… informational check(s) live-stack did not succeed; not blocking …`); `validation.json` in the `published-consumer-validation` artifact |
| Whole-train rehearsal before the merge (review M1) | **done, in CI on every PR**: `ci.yml` job `release-train-rehearsal` runs the train's `publish_candidates` sequence for the staging identity of the PR head — resolve, release plan, sync + verify every manifest, wave build, wave `publish --rehearsal` (= `npm publish --dry-run` through the helper, zero registry traffic) — and asserts it reached the last wave with all 43 packages dry-run-published and no publish command without `--dry-run`. Run URL recorded in `docs/specs/platform-gaps/phase1/rn-07-release.md` §6 | `gh run view <ci run> -R a5c-ai/fabric --json jobs --jq '.jobs[] \| select(.name \| startswith("Release train rehearsal")) \| .conclusion'` → `success` |
| First real run of `publish-npm-private.yml` | **is the staging merge of PR #4** (a push to `staging` sets `dry_run=false`). The workflow is not registered on any branch GitHub can dispatch from until that push, so no `workflow_dispatch` rehearsal of it can precede the merge; the rehearsal evidence is the job above plus the verifier's 43/43 | `gh api repos/a5c-ai/fabric/actions/workflows/publish-npm-private.yml` → 404 until the merge |
| Workflows the train calls or this runbook dispatches (review M2, amended by review-2 M8) | **enabled 2026-10-04** by the owner at the pre-Gate-B interview ("Enable all four now"): `live-stack-published.yml`, `release-tags.yml`, `publish-packages-from-tag.yml` are `active`, as is `release-artifact-verifier.yml`. **`fix-npm-tags.yml` was re-DISABLED by the orchestrator the same day** (workflow 352401959, `disabled_manually`, 0 runs ever) and is re-enabled by the orchestrator immediately after PR #4 merges to `staging`, as part of Gate B, once M7 is verified. Why: no *trigger* fires pre-merge (the legacy copies on `main`/`staging` are `workflow_call`/dispatch-only or trigger on `babysitter/**` tags), **but a MANUAL dispatch of `fix-npm-tags.yml` without `--ref`, or with `--ref main`, or with `--ref staging` before the merge, would run the legacy npmjs copy** (identical blob `44112b847` on both branches) whose `dry_run` input defaults to `false` — a live `--fix` against the public `@a5c-ai/*` dist-tags with the org-wide `NPM_TOKEN`. Do not dispatch `fix-npm-tags.yml` until PR #4 is on `staging` and `main` and M7 is verified. This is also why every dispatch here carries `--ref` | `gh api repos/a5c-ai/fabric/actions/workflows --paginate --jq '.workflows[] \| "\(.state) \(.path)"'` → `active` for the four above, `disabled_manually` for `fix-npm-tags.yml` until Gate B; `gh api repos/a5c-ai/fabric/actions/workflows/352401959/runs --jq .total_count` → `0` |
| Human reader binding on `fabric-npm` (review M6) | **verified 2026-10-04** by the orchestrator with a fresh `gcloud` token: the IAM policy shows `roles/artifactregistry.reader` for `user:tal@a5c.ai`, `npm view @fabricstack/smoke` through `google-artifactregistry-auth` printed `0.0.0-smoke.4`, and a direct bearer-token `GET` answered `200`; the 2026-10-03 E403 was an expired token, not a missing binding. **Prerequisite kept: re-authenticate first** (section 6; tokens live about an hour). The *local* read-only commands in sections 1, 2.6, 3, 4.1, 4.4 and 6 are usable after that; the run's uploaded artifacts (`publish-npm-private-<version>`, `published-consumer-validation`, `release-train-rehearsal-<version>`) remain the CI-side evidence of the same facts | `gcloud artifacts repositories get-iam-policy fabric-npm --location=us-central1 --format=json` shows `roles/artifactregistry.reader` for `user:tal@a5c.ai`, **and** `npx -y google-artifactregistry-auth --repo-config=.npmrc --credential-config="$HOME/.npmrc" && npm view @fabricstack/smoke --@fabricstack:registry=https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/` → `0.0.0-smoke.4` (both passed 2026-10-04) |
| Organization secret `NPM_TOKEN` restricted away from fabric (G3, review M7) | **owner action, required BEFORE anyone dispatches `fix-npm-tags.yml` — i.e. before its Gate-B re-enable** (review-2 M8 moved it forward from "before Gate B"; still `all` on 2026-10-04): restrict the secret's repository selection in the GitHub UI to the public-publishing repositories; fabric unselected | `gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN --jq .visibility` → `selected`, and `gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN/repositories --jq '.repositories[].name'` does not list `fabric` |
## 1. Normal train (push to `staging`, `main` or `develop`)
Workflow: `.github/workflows/publish-npm-private.yml` (job graph below). Nothing to run by
hand; the steps are listed so a reviewer can follow a run.
```
resolve_release resolve ONE identity (version, dist-tag, fabric/ tag, registry, candidate tag)
verify-manifests on the checked-in tree; publish-isolation guards (G1-G3)
verify_release_artifacts release-artifact-verifier.yml: all 43 packages, no tolerance (FIX-011)
publish_candidates needs + checks the verifier result; WIF auth -> google-artifactregistry-auth
release plan written + validated; sync-workspace-versions + verify-manifests
+ publish-isolation-guard over the synchronized tree; preflight channel assertion
build the train in derived dependency waves; verify SDK package health
publish every package under candidate-<version> (FABRIC_RELEASE_PUBLISH=1)
assert every candidate dist-tag == version
published_consumer_validation live-stack-published.yml with the EXACT version (section 3)
promote_release_channel release-promotion.cjs promote --evidence validation.json (the only dist-tag move)
gated on the EXACT-VERSION consumer job + the recorded verdict (callee outputs
consumer_result / promotable); the live-model lanes are informational unless the
variable FABRIC_LIVE_LANES_GATE=required (owner decision 2026-10-05, section 0a)
assert_release_channel_tags every package's channel tag == version (FIX-001 final assertion)
create_release_tag release-tags.yml: fabric/<branch>/v<version>, provenance fabric-release-*
```
What to check after a green run (read-only). Prerequisite: re-authenticate first (section 6;
M6 verified 2026-10-04) — the run's uploaded artifacts are the CI-side evidence of the same
facts.
```bash
# the identity the run resolved (also in the run's step summary)
SHORT_SHA=$(git rev-parse --short=12 <released commit>)
node scripts/release-version.cjs resolve --branch staging --sha "$SHORT_SHA"
# channel state on Artifact Registry over ALL 43 packages (reader credential, section 6)
node scripts/release-version.cjs assert-channel-tags --version <version> --dist-tag staging --mode final
# the tag exists here and nowhere else
git ls-remote --tags origin "refs/tags/fabric/staging/v<version>"
git ls-remote --tags https://github.com/a5c-ai/babysitter.git | grep -c "fabric/" # must be 0
```
## 2. Dry-run commands (local, nothing written anywhere)
```bash
# 1. the identity the train would use for this commit on each channel
SHORT_SHA=$(git rev-parse --short=12 HEAD)
node scripts/release-version.cjs resolve --branch main --sha "$SHORT_SHA"
node scripts/release-version.cjs resolve --branch staging --sha "$SHORT_SHA"
node scripts/release-version.cjs verify --branch main --sha "$SHORT_SHA" --version "$(node -p "require('./package.json').version")"
# 2. the checked-in train is lockstep with exact internal pins (G2(d))
node scripts/release-version.cjs verify-manifests --version "$(node -p "require('./package.json').version")"
# 3. publish-isolation (G1-G3), public-push guard, metadata/coverage gate
node scripts/ci/publish-isolation-guard.cjs --self-test
node scripts/ci/public-push-guard.cjs --self-test
npm run verify:metadata
# 4. the derived build/publication waves
node scripts/release-matrix.cjs --group all-publishable --format waves
# 5. release tooling + workflow-shape suites (FIX-001/005/010/011 + WP-RN-07)
npm run test:release-tooling
# 6. rehearse the publish helper for one package: npm publish --dry-run under the
# candidate tag, nothing written. --dry-run queries Artifact Registry for the
# package's own existence (READER credential, section 6 — re-authenticate first) and
# accepts an exact internal pin that a workspace of this checkout carries at
# that version ("published by an earlier wave of this train"); any other pin
# is still looked up on the registry. --rehearsal (implies --dry-run) issues
# NO registry query at all, so it needs no credential — this is what the CI
# rehearsal job runs for every package.
GITHUB_REF_NAME=main node scripts/publish-package-from-tag.mjs \
--workspace=@fabricstack/trust --release-version="$(node -p "require('./package.json').version")" --skip-build --dry-run
GITHUB_REF_NAME=main node scripts/publish-package-from-tag.mjs \
--workspace=@fabricstack/harness-policy --release-version="$(node -p "require('./package.json').version")" --skip-build --rehearsal
# 7. the full packed-artifact matrix (slow; what the release gate runs)
node scripts/verify-release-artifacts.mjs --report-dir artifacts/release-verifier --exactness
```
Whole-train rehearsal in CI — two forms:
1. **Before the merge (every PR):** the `release-train-rehearsal` job of `ci.yml` (section 0a)
executes the train's `publish_candidates` sequence end-to-end for the staging identity of the
PR head, with `--rehearsal` in place of the publication opt-in, and proves it reached the last
of the 13 waves with all 43 packages dry-run-published and no publish command without
`--dry-run`. The registry-facing steps (preflight channel assertion, candidate dist-tag
assertion) and the packed-artifact gate are logged as skipped there: the verifier runs as its
own PR workflow (`release-artifact-verifier.yml`), the two assertions run in the train.
2. **After the merge:** `gh workflow run publish-npm-private.yml -R a5c-ai/fabric --ref staging`
with the default `dry_run=true` runs every job up to and including `npm publish --dry-run` for
all 43 packages (the helper's `--dry-run` satisfies exact internal pins from the synchronized
checkout, so an empty registry no longer stops wave 2), then stops before
validation/promotion/tagging. A dispatch publishes only when `dry_run` is explicitly `false`.
**The first real run of `publish-npm-private.yml` is the staging merge itself** (a push to
`staging` runs with `dry_run=false`); it cannot be dispatched before the file is on a branch
GitHub registers it from. Gate B therefore relies on form 1 plus the verifier's 43/43 as the
rehearsal evidence (`docs/specs/platform-gaps/phase1/rn-07-release.md` §6).
## 3. Published-consumer validation (what gates promotion)
`.github/workflows/live-stack-published.yml`, called with the exact version and the candidate
dist-tag, never a channel:
- `published_consumer`: fresh project outside the repository with its own `.npmrc`
(`@fabricstack:registry=<Artifact Registry>` — written by
`scripts/verify-published-release.mjs`), all 43 packages installed at `@<version>`, every root
and exported runtime subpath imported, every declared bin smoked; then the spec 15 §3.1
global-install smoke (WP-RN-06 AC2): `npm i -g @fabricstack/orchestrator@V @fabricstack/harness-cli@V`
with per-call scope routing (`--@fabricstack:registry=<AR>`), no `EEXIST`, exactly the
`orchestrator` bin; then `@fabricstack/harness-hooks-cli@V @fabricstack/harness-sdk@V` → exactly
`harness harness-hooks orchestrator`; each answers `--version`.
- live-stack lanes: the §3.1 CLIs (`orchestrator`, `orchestrator-sdk`, `harness-sdk`,
`agent-runtime-cli`, `harness-hooks-cli`) installed globally from Artifact Registry at the exact
version, asserted with `release-promotion.cjs assert-installed`, then the live stack.
- `record_validation` writes `validation.json` (checks `package-install`, `root-import`,
`subpath-import`, `bin-smoke`, `live-stack`); `release-promotion.cjs promote` refuses without it.
- **Gate (owner decision 2026-10-05, "Consumer-only gate for the first train")**: every check must be
RECORDED and every *gating* check green. `live-stack` is gating only when the evidence was recorded
with `--live-lanes-gate required`, which `record_validation` derives from the repository/organization
variable `FABRIC_LIVE_LANES_GATE` (unset or `informational` → consumer-only gate, the lanes' failure is
listed under `informationalFailures`; `required` → the full gate; the script's own default is
`required`). The caller's `promote_release_channel` evaluates after a red lane (`!cancelled()`) and
checks, explicitly, the resolver and publish results, the consumer job's own result
(`consumer_result == 'success'`), the recorded verdict (`promotable == 'true'`) and, under `required`,
the called workflow's aggregate result. No lane is `continue-on-error`. Condition to restore the full
gate: a valid `OPENAI_API_KEY` (section 7.2).
Local equivalent of the consumer half (reader credential, section 6 — re-authenticate first;
the `published-consumer-validation` artifact is the CI-side evidence):
```bash
node scripts/verify-published-release.mjs --version <version> --report-dir artifacts/published-consumer
```
## 4. Recovering a half-published train
Symptom: `publish_candidates` failed in wave N. Waves `< N` are on Artifact Registry under
`candidate-<version>`; **wave N itself is partially published** — its helpers run in parallel
and `wait` collects every sibling before the step exits, so the members that succeeded are on
the registry and the failed ones are not; waves `> N` are missing; **no channel tag has moved**
and **no release tag exists** (both happen after validation). Nothing is broken for consumers:
channels still resolve the previous release (or nothing, on a first train).
**First staging train (`7.0.1-staging.<sha>`) — abort is fix-forward only once `staging` has
been promoted.** Before promotion there is nothing to roll back: the candidate versions are
orphaned and no channel points at them. After promotion there is no earlier version on
Artifact Registry for `--set staging=<previous-version>` (section 5; the script refuses a version
that any package lacks), so the only abort is a new commit → a new train that moves `staging`
forward again. Consumers of the brand-new `@fabricstack` scope are unaffected either way:
nothing is pinned to it yet, and `latest` never moves on a staging train.
### 4.1 Find out what is on the registry (local, read-only)
Re-authenticate first (section 6). The uploaded `publish-npm-private-<version>` artifact (one
helper log per package: `published …` / `already exists …` / the failure) is the same
inventory, read from CI.
```bash
V=<version>
AR=$(node -p "require('./package.json').fabric.privateRegistry")
# which packages have the exact version (prints MISSING for the unpublished half)
node scripts/release-matrix.cjs --group all-publishable --format workspaces | while read -r p; do
if npm view "$p@$V" version --registry "$AR" >/dev/null 2>&1; then echo "PUBLISHED $p"; else echo "MISSING $p"; fi
done
# candidate dist-tag state over the whole inventory (final mode fails while any package is missing — that is the diagnosis)
node scripts/release-version.cjs assert-channel-tags --version "$V" --dist-tag "$(node scripts/release-promotion.cjs candidate-tag --version "$V")" --mode final
```
### 4.2 Decide
- **Cause was transient** (registry hiccup, runner died): **first re-run the failed
`publish_candidates` job on the same run** (`gh run rerun <run-id> -R a5c-ai/fabric
--failed`). It is idempotent — the helper only ensures the candidate dist-tag of a version
that already exists and publishes the missing ones — and the same run then continues into
validation, promotion, assertion and the tag, so nothing is re-derived. Do **not** push an
empty commit: the branch train resolves a *new* version for a new commit. Use 4.3 when the run
itself is dead (expired, cancelled, or failed a second time) — it acts on an immutable tag.
- **Cause was a package defect**: fix it on the branch; the next push resolves a new version and
publishes a complete train. The orphaned `candidate-<old>` versions stay installable for
diagnosis and are never promoted; nothing needs deleting (channels never pointed at them).
### 4.3 Recover from the immutable tag (`publish-packages-from-tag.yml`) — APPROVAL REQUIRED
Recovery acts on a `fabric/<branch>/v<version>` tag so the tree is immutable. The branch train
only tags after promotion, so for a half-published train the tag does not exist yet: create it
locally **at the released commit**, push it, and let the tag workflow do the rest.
```bash
# at the exact commit the failed run built (check the run's "Release identity" summary)
git checkout <commit>
SHORT_SHA=$(git rev-parse --short=12 HEAD)
node scripts/release-version.cjs verify --branch staging --sha "$SHORT_SHA" --version "$V" # must agree
# local, idempotent, annotated, provenance source=manual (so the tag workflow knows it may publish)
node scripts/release-version.cjs ensure-tag --version "$V" --branch staging --commit "$(git rev-parse HEAD)" --source manual
node scripts/release-version.cjs tag-source --tag "fabric/staging/v$V" # published_by_release_workflow=false
# APPROVAL REQUIRED: pushing the tag starts the recovery publication
git push origin "refs/tags/fabric/staging/v$V"
# or, for an existing tag — --ref is mandatory (see the note at the top): dispatch from the
# tag itself or from `staging`/`main`, never from a feature branch whose name contains `/`
# (the helper refuses any `/`-shaped ref outside fabric/<branch>/v<version> as a channel source)
gh workflow run publish-packages-from-tag.yml -R a5c-ai/fabric --ref "fabric/staging/v$V" -f tag="fabric/staging/v$V"
```
What the recovery lane does (all automatic, all idempotent):
1. `release_identity`: derives version/channel/registry from the tag; refuses a tag the branch
train created itself (`published_by_release_workflow=true`) unless manually dispatched.
2. `verify_release_artifacts`: the strict matrix against the tag's tree.
3. `publish_non_plugin_packages`: WIF auth, release plan from the tag, sync + verify every
manifest, preflight channel assertion, build in derived waves, publish in derived waves under
`candidate-<version>`. **Packages whose exact version already exists are not re-published —
the helper only ensures their candidate dist-tag** (`scripts/publish-package-from-tag.mjs`),
so the already-published half is a no-op and the missing half completes. Then every
candidate dist-tag is asserted.
4. `published_consumer_validation` → `promote_release_channel` → channel assertion, exactly as
the branch train. Re-running the recovery for the same tag after a second failure is safe for
the same reason.
Idempotency guarantees, by mechanism: `isPublished` (E404-only "absent" detection; any other
registry answer aborts rather than guesses), `ensure-tag` (never re-points a tag),
`assert-channel-tags --mode preflight` (never moves a channel backward),
`release-promotion.cjs promote` (refuses without evidence naming this exact version).
### 4.4 After recovery
Local, read-only (re-authenticate first, section 6); the recovery run's
`assert_release_channel_tags` job is the CI-side evidence.
```bash
node scripts/release-version.cjs assert-channel-tags --version "$V" --dist-tag staging --mode final
git ls-remote --tags origin "refs/tags/fabric/staging/v$V"
```
## 5. Repairing dist-tags (`fix-npm-tags.yml`) — APPROVAL REQUIRED for `dry_run=false`
Script: `scripts/fix-broken-latest-tags.mjs`, inventory-derived, private registry named on
every call. Two modes, both dry runs unless `--fix` / `dry_run=false`:
```bash
# audit: latest -> channel prerelease, channel tag -> empty artifact, missing latest; proposes the last healthy release
node scripts/fix-broken-latest-tags.mjs
node scripts/fix-broken-latest-tags.mjs --json
# explicit repair of one dist-tag for EVERY package at one exact version (refuses if any package lacks that version)
node scripts/fix-broken-latest-tags.mjs --set staging=<version>
# narrow
node scripts/fix-broken-latest-tags.mjs --package @fabricstack/catalog
```
In CI (`--ref` mandatory, see the note at the top). **Until PR #4 is merged, the copies of this
workflow on `main` AND `staging` are the legacy npmjs file** (identical blob `44112b847`):
`workflow_dispatch` input `dry_run` defaults to `false`, `checkout ref: staging`,
`registry-url: https://registry.npmjs.org/`, `NODE_AUTH_TOKEN: secrets.NPM_TOKEN`, and the
non-dry branch runs `scripts/fix-broken-latest-tags.mjs --fix` over the public `@a5c-ai/*`
packages — so `--ref staging` is only correct after the merge, and a dispatch with no `--ref`,
with `--ref main`, or with `--ref staging` before the merge would be a live public dist-tag
mutation (the forms below carry `-f mode=…`, which the legacy inputs reject with 422, but a
bare dispatch would not be rejected). Do not dispatch `fix-npm-tags.yml` until PR #4 is on `staging` and `main` and M7 is verified. Mitigation in force: the
workflow is `disabled_manually` (re-disabled by the orchestrator 2026-10-04, review-2 M8) and is
re-enabled by the orchestrator as part of Gate B, right after the merge to `staging` and after
M7's verification (section 0a). Once re-enabled:
`gh workflow run fix-npm-tags.yml -R a5c-ai/fabric --ref staging -f mode=audit` (dry run), then
`gh workflow run fix-npm-tags.yml -R a5c-ai/fabric --ref staging -f mode=audit -f dry_run=false`
once the plan is approved;
`gh workflow run fix-npm-tags.yml -R a5c-ai/fabric --ref staging -f mode=set -f set_spec=latest=<version>`
for the explicit repair. The audit mode needs no human registry credential (the workflow
authenticates through WIF), so it is the diagnosis tool whenever a human token is stale. A
channel rollback is the same operation with the previously validated version:
`--set latest=<previous-version>` — publication is immutable, the dist-tag is the only
reversible surface. Never unpublish; recover forward. **On the FIRST staging train there is no
`<previous-version>` on Artifact Registry** (the script refuses a version any package lacks), so
a post-promotion abort of that train is fix-forward only — a new commit, a new train (section 4).
## 6. Credentials for the read-only local commands
**Review M6 — verified 2026-10-04 (section 0a).** On 2026-10-03 the human credential of
`tal@a5c.ai` received `403 Forbidden` from Artifact Registry on every npm read (`npm view` of an
existing and of a made-up package alike, and a direct bearer-token `GET`), although the phase-0
record lists the account as `roles/artifactregistry.reader`. The cause was an expired access
token: with a fresh `gcloud` token the orchestrator ran the verification below and all of it
passed (the policy shows `roles/artifactregistry.reader` for `user:tal@a5c.ai`;
`npm view @fabricstack/smoke` → `0.0.0-smoke.4`; the direct bearer `GET` → `200`).
**Prerequisite kept: re-authenticate first** — run the `google-artifactregistry-auth` line
before any local read-only command in this runbook, and again when a read starts answering
401/403. The verification, re-runnable at any time:
```bash
gcloud artifacts repositories get-iam-policy fabric-npm --location=us-central1 --format=json # roles/artifactregistry.reader for user:tal@a5c.ai
npx -y google-artifactregistry-auth --repo-config=.npmrc --credential-config="$HOME/.npmrc"
npm view @fabricstack/smoke --@fabricstack:registry=https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/ # must succeed
```
No workflow depends on the human credential: all of them authenticate as the CI service account
through WIF, which published in `ar-smoke` run 36454074285. If the three lines above ever fail
again, the run's uploaded artifacts carry the same facts and every *local* read-only command in
this runbook falls back to CI-only until the credential is repaired.
```bash
gcloud auth login # once; tal@a5c.ai holds roles/artifactregistry.reader on fabric-npm
npx -y google-artifactregistry-auth --repo-config=.npmrc --credential-config="$HOME/.npmrc"
npm ping --registry https://us-central1-npm.pkg.dev/boot-464019/fabric-npm/
git diff --exit-code -- .npmrc # the committed file never gains a credential line
```
Tokens live about an hour; rerun the helper when `npm view` starts answering 401/403. A 403 on a
package you expect to exist means the active gcloud account lacks the reader binding
(`gcloud auth list`), not that the package is absent.
## 7. Follow-ups attached to the first publications
### 7.1 Pin the newly published agent-runtime hooks leaf in the hooks CLI — REQUIRED FOLLOW-UP
`@fabricstack/harness-hooks-agent-runtime` has never been published, so
`packages/adapters/hooks/cli/package.json` deliberately does not pin it yet
(`//deferred-dependency`, FIX-005): an exact pin on an unpublished version would fail the
clean-consumer verification and the publish helper's exact-internal-dependency gate. Immediately
after the first train that publishes it:
```bash
# AFTER @fabricstack/harness-hooks-agent-runtime@$RELEASE_VERSION exists on Artifact Registry:
npm view "@fabricstack/harness-hooks-agent-runtime@$RELEASE_VERSION" version --registry "$AR" # must succeed
# add the exact pin next to the other hooks leaves, drop the //deferred-dependency note
node -e "const fs=require('fs');const p='packages/adapters/hooks/cli/package.json';const m=JSON.parse(fs.readFileSync(p));m.dependencies['@fabricstack/harness-hooks-agent-runtime']=m.version;delete m['//deferred-dependency'];fs.writeFileSync(p,JSON.stringify(m,null,2)+'\n')"
npm install --package-lock-only --legacy-peer-deps
node scripts/release-version.cjs verify-manifests --version "$(node -p "require('./package.json').version")"
npm run test:release-tooling # the FIX-005 test flips to guarding the pin
```
### 7.2 Pending owner actions (recorded, not automated)
- **Organization secret `OPENAI_API_KEY` is invalid — owner action (found 2026-10-05, publish run
37279009004)**: the secret is delivered to fabric (visibility `all`, non-empty in the job) and
rejected by OpenAI. pi lane, `agent-output.txt`:
`OpenAI API error (401): {"message":"Incorrect API key provided: AcuYB8hq************…JuS3. …","type":"invalid_request_error","code":"invalid_api_key"}`;
codex lane (same credential through codex 0.160.0's websocket transport):
`401 Unauthorized: Missing bearer or basic authentication in header, url: https://api.openai.com/v1/responses`.
Every default live lane is `openai/gpt-5.5`, so no lane can pass until the secret is replaced. Then set
the variable `FABRIC_LIVE_LANES_GATE=required` to restore the full gate (section 0a). Verification: a
green `Published (…, codex/gpt-5.5, non-interactive)` lane on the next train, and `validation.json`
with `liveLanesGate: "required"` and `informationalFailures: []`.
- **G3 / review M7 — owner action, required BEFORE anyone dispatches `fix-npm-tags.yml`, i.e.
before its Gate-B re-enable** (review-2 M8 moved it forward from "before Gate B", 2026-10-04):
restrict the organization secret `NPM_TOKEN` to the repositories that publish to npmjs (fabric
unselected): `docs/specs/platform-gaps/phase0/g3-npm-token-restriction.md`. Fabric's branch
workflows no longer reference it, but the legacy copies on `main` and `staging` still do (which
is what makes `--ref` mandatory above and why `fix-npm-tags.yml` stays disabled until Gate B),
and the secret is currently visible to all repositories (`gh api orgs/a5c-ai/actions/secrets --jq '.secrets[] | "\(.name) \(.visibility)"'`
→ `NPM_TOKEN all`). Verification: `gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN --jq .visibility`
→ `selected`, and `gh api orgs/a5c-ai/actions/secrets/NPM_TOKEN/repositories --jq '.repositories[].name'`
without `fabric`.
- **Review M6 — verified 2026-10-04**: the human reader binding on `fabric-npm` works with a
fresh token (section 6); re-authenticate before local reads.
- **G4** — register the `fabricstack` organization on public npmjs (publish nothing, 2FA,
restricted membership): `docs/specs/platform-gaps/phase0/g4-npmjs-org-runbook.md`;
`node scripts/ci/publish-isolation-guard.cjs --g4-status` prints the current state.
## 8. Public-namespace invariant checks (run after any release activity)
```bash
# the public repository was not written to
gh api repos/a5c-ai/babysitter --jq .pushed_at # unchanged
git ls-remote --tags https://github.com/a5c-ai/babysitter.git | grep -c "fabric/" # 0
git ls-remote --tags https://github.com/a5c-ai/babysitter.git | wc -l # unchanged count
# nothing under the private scope exists on the public registry (404 everywhere)
for p in orchestrator orchestrator-sdk catalog harness-sdk; do
printf "%s " "@fabricstack/$p"; curl -s -o /dev/null -w "%{http_code}\n" "https://registry.npmjs.org/@fabricstack%2f$p"
done
# fabric's tags are only fabric/<branch>/v*
git ls-remote --tags origin | grep -v "refs/tags/fabric/" | grep -c "babysitter/" # 0
# guards
node scripts/ci/publish-isolation-guard.cjs --self-test && node scripts/ci/public-push-guard.cjs --self-test
```
documents
[]
Outgoing edges
None.
Incoming edges
contains_page1
- page:docs·PageBabysitter Docs