Turn Flue episodes into a LoopSpec design review

This tutorial adds an evidence-to-eval loop to an existing Flue GitHub channel. It logs structural runtime events, joins them into episodes, measures three different things, proposes a candidate LoopSpec change, and tests that change with a removal probe.

Start with this prompt

Open your project in any coding agent and paste this prompt:

Read the LoopSpec + Flue coding tutorial at https://loopspec.cyborg.build/flue/ and its linked implementation guide. Inspect this repository before changing it.

Add LoopSpec to this project as a reviewable design and evidence-to-eval layer.

If this project uses Flue, follow the tutorial's Flue integration: project privacy-minimised structural runtime events, join them into episodes using stable IDs, and add explicit application outcomes and human-in-the-loop capability probes.

If this project does not use Flue, adapt the same strategy to its existing framework and observability stack. Do not introduce Flue just for this integration.

In either case:
- keep .loop.yaml as the design contract, not the raw log format;
- keep LoopSpec's canonical parsing, validation, linting, semantic hashing, and diffing in the LoopSpec CLI rather than reimplementing them;
- assess performance, human capability, and authority separately;
- never infer that a real person gained or lost capability from agent traces;
- generate evidence-linked candidate specs and regression evals, but require human approval before adopting or deploying a behavioural change;
- integrate with the project's lint, typecheck, tests, CI, and existing OpenTelemetry instrumentation; and
- preserve its framework conventions, privacy constraints, and security boundaries.

Implement the smallest complete example, run its checks, and explain what changed, what remains synthetic, and what requires human review.

The agent should use the project's own runtime hooks when Flue is absent. In both paths, LoopSpec remains the canonical design language and semantic engine rather than becoming a framework-specific logging format.

What the finished loop does

Flue observations ─┐
GitHub delivery ───┼─> episode ─> 3 assessments ─> candidate .loop.yaml ─> eval
application result ┤
human-only probe ──┘

1. Define the event contract

Store a small versioned union rather than copying Flue's complete observation objects. Every event carries the episode ID and the digest of the LoopSpec that produced it.

export interface LoopEvent {
  schemaVersion: "1.0";
  id: string;
  type:
    | "episode.queued"
    | "episode.settled"
    | "channel.delivery.accepted"
    | "model.completed"
    | "action.completed"
    | "outcome.observed"
    | "capability.assessed"
    | "authority.window.measured";
  occurredAt: string;
  episodeId: string;
  source: "flue" | "channel" | "application" | "human_probe";
  specId: string;
  specDigest: string;
  trace: {
    instanceId?: string;
    conversationId?: string;
    operationId?: string;
    turnId?: string;
    toolCallId?: string;
  };
  data: Record<string, string | number | boolean | null>;
  contentPolicy: "structural-only" | "explicit-assessment";
}

Use Flue's dispatch receipt submissionId as episodeId. Do not join records by matching message text or timestamps.

2. Project Flue observations

Register observe() once at module scope. Project only the cases needed by the review loop and return null for everything else.

import { observe, type FlueObservation } from "@flue/runtime";

export function projectFlueObservation(
  event: FlueObservation,
  spec: { specId: string; specDigest: string }
): LoopEvent | null {
  if (!event.submissionId) return null;

  switch (event.type) {
    case "submission_settled":
      return structuralEvent(event, spec, "episode.settled", {
        outcome: event.outcome,
        errorType: event.error?.type ?? null
      });
    case "turn":
      return structuralEvent(event, spec, "model.completed", {
        provider: event.request.providerName,
        model: event.request.requestedModel,
        durationMs: event.durationMs,
        isError: event.isError
      });
    case "tool":
      return structuralEvent(event, spec, "action.completed", {
        action: event.toolName,
        durationMs: event.durationMs,
        isError: event.isError
      });
    default:
      return null;
  }
}

observe((observation) => {
  const event = projectFlueObservation(observation, currentSpec);
  if (event) pending.push(event);
});

Keep the callback synchronous and cheap. Hash correlation identifiers. Do not record prompts, model output, reasoning, tool arguments, tool results, error messages, stacks, sender identities, or message bodies.

3. Join the existing GitHub channel

Keep the channel's signature verification and routing unchanged. After verification, retain the dispatch receipt and add one structural ingress event.

const receipt = await dispatch(Assistant, {
  id: channel.instanceId(issueRef),
  idempotencyKey: delivery.deliveryId,
  initialData,
  message
});

appendLoopEvent(projectAcceptedGitHubDelivery(delivery, receipt, {
  specId: "support_triage",
  specDigest: loopspecDigest,
  instanceId: channel.instanceId(issueRef)
}));

Test a valid and invalid signature. The invalid request must fail before the application callback runs. If the project has no GitHub channel yet, generate Flue's current one:

flue add channel github

4. Add evidence the trace cannot provide

An agent trace can tell you that a task succeeded. It cannot tell you whether a person could still perform the critical act without assistance. Add that as an explicit probe.

export function humanCapabilityAssessed(
  context: DomainEvidenceContext,
  score: number,
  threshold = 0.7
): LoopEvent {
  return {
    ...baseEvent(context, "human-capability"),
    type: "capability.assessed",
    source: "human_probe",
    data: {
      bearer: "people.support_lead",
      capability: "notice_and_escalate_severe_case",
      condition: "unassisted",
      instrument: "delayed-removal-probe",
      score,
      threshold,
      passed: score >= threshold,
      assessor: "synthetic-tutorial"
    },
    contentPolicy: "explicit-assessment"
  };
}

In a live system, this could be a short delayed flash-card exercise: remove the agent, present a severe support case, and ask the human to notice and escalate it. Record the collection condition and assessor; do not infer the result from ordinary telemetry.

Also record the authority window: seconds between the accountable person seeing the proposed action and the last moment they can stop its consequence.

5. Assess the same episodes three ways

Keep the assessment groups separate because they answer different questions.

assisted outcome       0.88  0.91  0.90  0.93  0.92
human-only recovery    0.82  0.78  0.68  0.56  0.43
intervention window     120   110    92    58    35 seconds

Expected result: performance remains stable, while capability and authority report a transition. That disagreement is the signal used by the next step.

6. Generate a candidate, not an automatic decision

Translate detected transitions into evidence-linked design patterns. Write a new .loop.yaml; do not mutate the production specification.

const assessments = assessCausalPowers(episodes);
const proposal = proposeDesignRevision(episodes, assessments, specDigest);

// capability decline -> add an unaided recovery checkpoint
// authority decline  -> require confirmation before execution

proposal.status === "requires-human-approval";
proposal.claimCeiling ===
  "The proposed effects are synthetic hypotheses. Production adoption requires human review and prospective evidence.";

Call the LoopSpec CLI through a typed spawn() adapter using an argument array. Require protocol_version: 1 for check, expansion, and semantic diff. Do not reproduce LoopSpec semantics inside the TypeScript integration.

7. Turn the finding into a regression eval

Generate a removal-probe scenario from the proposal. Compare the baseline with the candidate under explicit synthetic assumptions, then run the normal code and spec gates.

npm run lint
npm run typecheck
npm test
npm run tutorial
loopspec check generated/support-triage.candidate.yaml --fail-on-findings
loopspec diff loops/support-triage.base.yaml generated/support-triage.candidate.yaml
npm run evals

Expected tutorial output

LoopSpec × Flue: evidence-to-eval
1. Projected 50 privacy-minimised events into 5 episodes.
2. performance stable
2. capability transition-detected
2. authority  transition-detected
3. Proposed 2 review-gated LoopSpec changes.
4. Typed CLI protocol v1 validated the candidate with zero findings.
5. Generated a removal-probe eval and 52 semantic change records.

Generated files

generated/
├── events.jsonl
├── episodes.json
├── assessment-groups.json
├── revision-proposal.json
├── support-triage.candidate.yaml
├── human-recovery.eval.json
├── canonical-ir.json
├── semantic-diff.json
└── SUMMARY.md

Production boundary

The integration may automate:

• privacy-minimised structural event collection;
• episode assembly and deterministic assessment policies;
• candidate diffs and regression-scenario generation; and
• lint, typecheck, test, semantic diff, and eval reporting.

Keep these operations human-controlled:

• claiming that a real person learned, retained, or lost a capability;
• approving the candidate LoopSpec; and
• deploying a behavioural or authority change.

For Cloudflare deployments, collect and enqueue events at the edge, then run the Python LoopSpec engine in CI, an MCP server, or a dedicated service. A Worker cannot spawn the Python CLI.