feat(workspace): a stage advance reaches the Work card on the write, not the next poll (ent#533)

[trinity-ability]

Summary

The Work card's pipeline steps (ent#525) refreshed only on the 12 s poll, so a stage advance could lag a full poll behind a file the agent had already written.

The file lives in the agent's own container, so the only thing that knows when it changed is the agent — the backend can only ask. That decided the fork:

• The agent server watches its own directory. A new agent_server/pipeline_state_watch.py scans ~/.trinity/pipeline-state/ once a second with os.scandir, keyed on (mtime_ns, size) per file. Steady state is one directory scan per second and no network at all — the canonical pipeline-tick writer runs every 15 minutes, so a POST is an event, not a stream. The baseline snapshot is taken silently at startup, so a restart never replays every instance on disk.
• It authenticates with its own key, not a shared secret. POST /api/agents/{agent_name}/pipeline-state/changed, Authorization: Bearer <the agent's own MCP key>, validated with track_usage=False and then authorize_heartbeat — feat: Agent heartbeat push for fast failure detection (RELIABILITY-004) #307's predicate, now guarding two routes. It is an allowlist: true only when scope == "agent" and the bound name equals the path. A user, system, connector, ops or null-scoped key, and any other agent's key, all get the same 403 with the same detail — a differential would make the route an oracle for which agents exist (Invariant security: implement safe tar extraction with symlink/hardlink validation #8).
  /api/internal/* was the wrong door twice over: its blanket router is gated on X-Internal-Secret, which is deliberately never injected into an agent container, and its one agent-key predicate (_pull_authorized) additionally requires a pull-pilot agent — a route placed there would have been silently dead for most of the fleet. /api/agents/{name}/… is also Invariant Security: Agent Credential Leakage in Execution Logs [FIXED] #15's shape for a fact about one named agent, and where both existing always-on agent self-reports already live.
• The backend publishes a thin trigger and nothing else. A pipeline_state_changed event on /ws carrying identifiers only — pipeline_id, instance_id, an opaque stage bounded to 80 printable chars, and a server-stamped changed_at. Never health, blockers, escalations, per-stage metrics, or any file body. agent_name is top-level, which is what makes ent#467 scope it to clients whose roster contains the agent; the dict literal is built in the same function that broadcasts it, as that guard's AST discovery requires. services/event_bus.py is untouched — an agent-keyed payload needs no bus change.
• CLAUDE.md Rule security: implement safe tar extraction with symlink/hardlink validation #8 is respected. Trinity learns exactly one thing — a file changed. No DAG state, no stage comparison in the backend, no transition logic, no pipeline state in the database. The read stays client_portal/work/pipeline_state.py; the store refetches through the existing access-controlled route.
• The 12 s poll stays (AC2). WORK_POLL_MS = 12000 and _ensurePolling are untouched, so a dropped, coalesced or failed notice is invisible and an old base image behaves exactly as ent#525 shipped.
• A 120 s Redis generation, because the cache is per-process. A notice bumps a per-agent generation that the 10 s cache compares alongside its TTL. Dropping the local entry alone fixes only the worker that received the notice — production runs uvicorn --workers 2, so the other worker would keep serving its own ≤10 s-old entry, and dev's single --reload worker would have passed green. The generation is read before the read, not after, so a notice landing mid-read invalidates the entry it is about to store rather than being stamped onto it as already-seen. Redis down ⇒ generation None on both sides ⇒ exactly today's TTL-only behaviour.
• No Dockerfile change. docker/base-image/Dockerfile already does COPY ./agent_server, so the new module ships by construction; no new dependency (stdlib + httpx, already present), and USER developer is untouched (Invariant Client/Viewer User Role (AUTH-002) #17).

Internal constants, none of them user-visible: watcher tick 1 s, NOTIFY_LIMIT = 20 / 10 s per agent server-side, _GEN_TTL_SECONDS = 120, and client-side PIPELINE_PUSH_DEBOUNCE_MS = 500 under a PUSH_MIN_GAP_MS = 1500 floor. The floor is load-bearing rather than decorative: it caps a viewer at 40 push refreshes/min against the Work read's existing 120/60 s per-viewer budget, so a burst can never reach that limiter — whose 429 would be visible. The store's debounce also gains earlier-deadline-wins, so a later 2 s agent_activity push cannot postpone a pending 500 ms stage refetch; without it the store would have quietly defeated the feature.

Fixes abilityai/trinity-enterprise#533

User-visible change: none

No .vue file is touched. PortalWorkCard.vue / PortalWork.vue render item.steps exactly as today; ent#525's three steps states and their three sentences are unchanged; no new copy, control, default, or empty/error/loading state. The only perceivable difference is when the same card updates — ~1–1.6 s after the agent writes instead of ≤12 s.

One open question for the maintainer

Should a pipeline-state file that is present but unreadable say "doesn't report steps", or "steps could not be read right now"?

When a state file exists but cannot be parsed (truncated mid-cp, bad JSON, failed download), pipeline_state._read returns none, which the card renders as the positive claim "<Agent> doesn't report steps." — while docs/memory/architecture/workspace.md defines the third state as unknown = "stopped, unreachable, unreadable, or two runs on one agent". The code and the written contract disagree about this one case.

What the branch does today: (a) — leave it. _read is byte-identical to dev (verified: 2521 bytes, 58 lines, both sides), so this lane neither causes the disagreement nor changes it. It is also not made more likely: the push-driven refetch lands ~1.5 s after the watcher first sees a new (mtime, size), i.e. after a KB-sized copy has completed — better phasing than today's arbitrary 12 s poll.

Both sentences already exist, so nothing new would be invented — but which one a person reads is a user-visible fork with two plausible answers, which is why the plan declined to decide it. Options:

• (a) leave it (what this branch does; the written contract stays wrong)
• (b) map "present but unreadable" to unknown — ~4 lines in _read plus one test, and the code conforms to the contract
• (c) leave the code and correct workspace.md instead

Verification

Gate run (WAVE-4), detached /verify-local --skip-agent — prod images, isolated sibling stack:

• unit: 15,159 passed / 31 skipped
• build + import-smoke: OK
• boot + health: OK
• integration: 70 passed / 13 skipped / 2 registry-deselected
• npm run test:unit: 2,660 passed

Verify mode note. /verify-local FULL refuses its agent stage in global mode on a host with a live dev stack (that stack owns trinity-agent-network, and agent-trinity-system on it is live infrastructure). This lane touches docker/base-image/**, so the agent half was proved manually instead — inside the freshly built base image f2f79d8f2ba6, in a throwaway container on the default bridge, with an in-container stub receiver.

Base-image proof (plan §10), verbatim. Write a state file, then rewrite it byte-identically, then advance the stage:

STUB POST /api/agents/probe/pipeline-state/changed Bearer trinity_mcp_probe {"pipeline_id":"digest","instance_id":"i1","stage":"synthesis"}
STUB POST /api/agents/probe/pipeline-state/changed Bearer trinity_mcp_probe {"pipeline_id":"digest","instance_id":"i1","stage":"synthesis"}
STUB POST /api/agents/probe/pipeline-state/changed Bearer trinity_mcp_probe {"pipeline_id":"digest","instance_id":"i1","stage":"publish"}

Two identical lines, then exactly one more. The byte-identical rewrite firing again is the contract, not a defect: the signature is (mtime_ns, size) and the watcher's job is "any change" — a writer that rewrites the same bytes has still written, and a growing file must fire again so a slow write self-heals. Coalescing is not the watcher's job; it lives at the backend's 20/10 s limiter and the client's 500 ms / 1500 ms floor.

The negative halves, which are what make the positive lines mean anything:

• 8 idle ticks → 0 POSTs, with the heartbeat's own STUB /api/agents/probe/heartbeat lines still landing every 5 s (so the stub was demonstrably alive and listening throughout)
• deleting the file → no POST (a deletion is not a stage advance; the read and the poll handle it)
• id -u → 1000 (Invariant Client/Viewer User Role (AUTH-002) #17 unchanged)
• import smoke inside the image → IMPORT OK /app/agent_server/pipeline_state_watch.py

trinity_mcp_probe is a throwaway fixture string, not a credential.

Post-rebase neighbourhood (rebased onto 682fce300):

cd tests && python3 -m pytest unit/test_ent533_pipeline_state_broadcast.py \
  unit/test_ent533_agent_pipeline_state_watch.py unit/test_918_report_broadcast.py \
  unit/test_ent467_ws_agent_scope.py unit/test_ent525_portal_work.py unit/test_agent_heartbeat.py \
  unit/test_1483_ws_setters_wired.py unit/test_1483_route_order.py unit/test_1310_auth_wiring.py \
  unit/test_293_admin_gate_rejects_agent_keys.py unit/test_models_centralized.py \
  unit/test_2338_journey_catalog.py -m "not slow" -q -p no:randomly
  → 268 passed, 1 skipped

Per file, so any later delta is attributable: test_ent533_pipeline_state_broadcast 38 · test_ent533_agent_pipeline_state_watch 29 · test_918_report_broadcast 1 · test_ent467_ws_agent_scope 37 · test_ent525_portal_work 59 · test_agent_heartbeat 12 · test_1483_ws_setters_wired 1 · test_1483_route_order 1 skipped · test_1310_auth_wiring 12 · test_293_admin_gate_rejects_agent_keys 25 · test_models_centralized 4 · test_2338_journey_catalog 50.

python3 lint_sys_modules.py          → OK: 140 violations in 49 files; baseline allows 206 — no new violations
python3 lint_root_test_placement.py  → OK: no self-contained tests in tests/ root; no unmarked async under tests/unit/
cd src/frontend && npx vitest run tests/unit/portalWork.spec.js \
  tests/unit/rawColorRatchet.spec.js tests/unit/loadingGateRatchet.spec.js
  → 3 files, 53 passed

Structural passes: /review and /cso --diff both clean, with the full auth failure matrix pinned (every non-agent scope, and a different agent's key, asserted to the same 403). A Gemini second voice ran at plan stage; its two substantive findings are in the branch — the earlier-deadline-wins rule and the open question above.

Deviations from the issue text

1. The poll is kept. The source debt entry said "drop the poll"; AC2 supersedes it — the poll is the fallback that makes a dropped notice invisible.
2. No execution id on the trigger. The agent does not know which ledger row it is serving at write time. AC3's "execution/instance ids" is met with pipeline_id + instance_id; attribution stays where it already lives, in the read.
3. Route under /api/agents/…, not /api/internal/…. Verified, not assumed — see the Summary: internal is secret-gated and agents hold no secret, and its one agent-key path additionally requires a pull-pilot agent.
4. Latency is ~1–1.6 s, not sub-second. One polling tick on the agent side is the honest floor without an inotify dependency. Stated that way in the docs rather than rounded to AC1's "~1 s".
5. backend.md's router count was already stale — it read 72 against 73 present. Corrected to the post-change truth (74) rather than perpetuated. agent-runtime.md's "runs two loops" was likewise already wrong (seven schedulers are armed), so it was rewritten without an exhaustive count instead of being bumped 2→3, which would have shipped a new false claim under cover of a minimal edit.
6. docs/memory/requirements/core-agent.md section number. /validate-pr caught that this lane's new section had taken 5.23, which the file's trailing numbering run has held since ent#523/524 — two ### 5.23 headings, with both of this lane's own §5.23 cross-references resolving ambiguously. Renumbered to 5.33 (the next globally free number) in its own commit. A pre-existing duplicate ### 5.14 on dev is left alone as out of scope.

Collisions

• Lane A, ent#532 (feat(workspace): the rail's Canvas/Files dot lights on the write, not the next refetch (ent#532) #2709) — still open and draft as of this writing, so there is nothing to resolve yet. It appends to the same src/backend/main.py setter block and adds a sibling data.type branch to utils/websocket.js's default: arm. Both overlaps are append-only: whichever merges first wins, and the loser simply re-adds its own lines. Neither lane edits services/event_bus.py.
• Doc-line overlaps with other open PRs, all append-or-adjacent and none structural: src/backend/models.py, architecture/backend.md, architecture/agent-runtime.md, architecture/api-endpoints.md, architecture/workspace.md, requirements/core-agent.md, requirements/scheduling.md, feature-flows.md.
• This rebase onto 682fce300 produced exactly one conflict — the feature-flows.md index row — resolved by keeping both sides' rows in date order.

Follow-ups — listed for the maintainer, deliberately not filed

1. SECURITY, pre-existing (from ent#525), the highest-value item here. client_portal/work/pipeline_state.py::_cache is keyed on agent_name alone, but the value it stores depends on the caller's roster (holder names are masked per viewer). Within the 10 s TTL, viewer B can therefore be served an entry computed for viewer A's roster — harmless in the more-masked direction, a disclosure in the other (the ent#467 class). This lane changes only when that cache misses, never its key, so it neither causes nor worsens it. The fix is to cache the unmasked value and mask at serve time — not to hash the roster into the key, which would multiply the entries the cache exists to bound.
2. routers/auth.py::get_redis_client() has no socket_connect_timeout and never memoises failure, so every caller pays a full connect attempt while Redis is down. Platform-wide, not specific to this lane.
3. tests/unit/test_agent_heartbeat.py installed a collection-time sys.modules shim that broke test_drain_bounded.py whenever it was collected first — latent and alphabet-dependent. Removed on this branch (it was turning six green tests red); the same pattern may exist in other agent-server test files.
4. valid_id uses re.match with a $ anchor, so a trailing newline passes. re.fullmatch is the correct spelling. Not exploitable here — the backend re-validates — but it is the wrong primitive.
5. The multi-tab margin against the Work read's per-viewer limiter narrows from about 4 tabs to 3 under a sustained ≤1.5 s writer. Still a comfortable margin; worth knowing before anyone lowers that budget.
6. CLAUDE.md:242 still says "~500 endpoints across 72 routers"; backend.md is now 74.
7. .claude/agents/test-runner.md wants catalog rows for the two new test files (private submodule, so not touchable from this PR).
8. Operator note: on the verify host, trinity-agent-base:latest now points at the freshly built image. The previous image keeps its 0.8.5 tag and was not deleted.
9. If a third file-backed read ever wants push, the honest generalisation is one "agent file changed" notice surface rather than a third bespoke route. Two is not yet a pattern.

🤖 Generated with Claude Code

[github-actions]

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

[github-actions]

⚠️ Live-instance suite skipped — merge conflict against dev.

Resolve by merging dev locally and pushing the result; the next nightly re-tests.