Loop Engineering: Designing Automated Feedback for AI Coding Agents

When a coding assistant cannot execute checks, the developer becomes the link between its proposed code and the real application: copy the code change, run the command, paste the error, and repeat.

An agent can automate much of that exchange when its tools return useful evidence. Direct file editing alone does not establish that the result works. The important distinction is whether changes are checked against the intended behavior.

Here, loop engineering means designing that bounded cycle of action, observation, and correction. It is a descriptive term for feedback-loop design, not a universally standardized development method. Anthropic’s Building effective agents describes agents using environmental feedback and stopping conditions.

This completes the series following Spec-Driven Development.

Close the loop with evidence

flowchart TD
    accTitle: Bounded development loop
    accDescr: Make a scoped change, run checks, and inspect results. If acceptance criteria are met, review and report evidence. Otherwise revise while progress and budget allow, or stop and report the blocker.
    A["Make a scoped change"] --> B["Run relevant checks"]
    B --> C["Inspect results and runtime observations"]
    C --> D{"Acceptance criteria met?"}
    D -- Yes --> E["Review diff and report evidence"]
    D -- No --> F{"Within budget and making progress?"}
    F -- Yes --> G["Diagnose and revise"]
    G --> A
    F -- No --> H["Stop and report blocker"]

The runtime can expose a terminal, a test API, or an MCP tool. The transport is secondary; what matters is that the agent receives the actual outcome, including failures to start the check.

Choose feedback that matches the change

Feedback	What it can establish	What it does not establish alone
Compiler or type checker	Compatibility with checked language and type rules	Correct business behavior
Unit and integration tests	Behavior covered by the assertions and fixtures	Untested cases or production conditions
Runtime logs and traces	What happened during an observed execution	Absence of failures in other executions
Browser checks	Tested interactions, rendering, and visible states	Complete usability or accessibility

Start with a baseline when practical. An existing failure should not be mistaken for a regression introduced by the patch. For a bug fix, a regression test that fails before the fix and passes afterward provides useful evidence.

Run focused checks while diagnosing, then the broader checks appropriate to the affected boundaries. Frontend work may need browser interactions and visual inspection; a database change may need a migration against representative test data.

Run checkout against a reproducible environment

For the running example, deploy the candidate web, orders, and pricing revisions into an isolated test environment. Seed valid and expired codes, wait for readiness, exercise the browser flow, and assert that the persisted quote matches the agreed fixture. Record the revisions, named checks, simulated dependencies, and cleanup outcome.

The harness post’s verification tool contract makes these results available to the agent. The end-to-end runbook reference below covers deployment, authentication, database access, and cleanup in detail.

Do not let the implementation define its own success

An agent that writes both the code and the tests can repeat the same mistaken assumption in both. Derive acceptance cases from the specification and keep important existing regression checks intact.

If a test fails, determine whether the implementation, test expectation, or environment is wrong. Do not weaken assertions, skip checks, or delete tests merely to obtain a passing run. A justified test change should explain the intended behavior and remain visible in review.

A zero exit code is useful only if the expected checks actually ran. A runner that discovers zero tests or checks the wrong directory can return misleading evidence. Report test counts or named checks where available, plus relevant exclusions.

Bound retries and side effects

Define limits before the loop starts. For example, a team might allow three repair attempts or ten minutes for a small task, then require a blocker report. These are illustrative budgets, not universal defaults.

Stop early when the same failure repeats without new evidence, a required service is unavailable, or the fix requires a product decision. Keep the current diagnosis, reproduction command, and next useful action so another engineer can continue.

Limit what each iteration can affect. Use isolated fixtures and test credentials, apply command timeouts, and clean up processes and temporary resources. Retrying a test is different from retrying a command that sends messages or changes shared data.

Keep concise summaries in the active context and preserve full logs separately. Summaries should retain failing assertions, relevant paths, and unresolved assumptions so compression does not erase the evidence needed for diagnosis.

Keep delegated work bounded and summaries useful

Rahul Garg’s The Orchestrator’s Tax describes a session where status checks imported large worker transcripts into the main agent’s context. His proposed benefit of delegation is protecting working context; the article explicitly does not establish a measured ranking of token costs.

For a multi-agent checkout task, group work that needs the same contracts and assign clear file ownership. A worker investigating quote expiry should return the relevant revisions, findings, changed files, checks, and unresolved decisions. Keep its detailed logs available through artifact references. Fetch a specific trace segment when diagnosing a problem instead of importing the entire transcript for a status update.

Give workers the applicable instructions and an explicit budget; verify what the runtime actually propagates. Avoid concurrent repository-wide operations such as stash or reset in a shared checkout. Isolated worktrees can separate edits, but the combined result still needs integration checks. Compare delegation with one agent on representative tasks before claiming a speed or cost improvement.

Preserve the developer’s learning loop

Unmesh Joshi’s The Learning Loop and LLMs warns that generating and reviewing code can bypass the experimentation through which developers discover a design.

After repairing quote expiry, have the maintainer predict what happens at the exact expiry boundary, then run that case with a controlled clock. Explain which component owns the decision and deliberately vary one assumption, such as a quote expiring between display and submission. If the result is surprising, revisit the contract or implementation and record the discovery.

This is useful during onboarding and consequential design changes. The goal is a maintainer who can diagnose the next failure, alongside a patch that passes today’s checks.

Report completion in a reviewable form

A useful completion report might say:

Changed: shipment consumer now reuses the event ID on provider retries.

Verification:
- Baseline: the new crash-recovery regression failed before the fix.
- After the fix: 18 consumer tests passed, including crash recovery.
- Type check and lint passed.

Limit:
- Provider deduplication was tested with a local fake.
- The real provider's retention guarantee still needs confirmation.

These are illustrative results. In actual work, record only the checks that ran. A known gap may mean the change is ready for review but not ready for deployment. Separate those decisions.

Evaluate the loop after the task ends

A task can finish successfully after many expensive, unproductive attempts. The final test result does not explain whether the agent repeatedly guessed, searched the wrong repository, or reran a command whose prerequisites were still missing. Evaluate the execution process as well as the delivered change.

This creates a second feedback loop: the agent repairs the implementation during a task, while a post-task evaluator helps the team improve the environment for future tasks. Run that evaluation for failed, canceled, and budget-exhausted tasks as well as successful ones. Otherwise, the most costly failures disappear from the sample.

Capture observable evidence

Record a structured trace with the task and acceptance criteria, repository revisions, prompt and harness versions, model identifiers, tool calls, sanitized arguments and results, exit codes, edits, test outcomes, timestamps, and reported usage. Include explicit plans or decision summaries when available.

Do not assume access to the model’s private chain of thought. An evaluator can assess recorded explanations against actions and results, but those explanations are not a complete or necessarily faithful account of internal reasoning. Judge evidence use and behavior rather than the apparent eloquence of a rationale.

Keep event IDs so every finding can point back to the relevant trace segment. Remove credentials, session tokens, and unnecessary private data before exporting traces to an evaluation service. Treat trace text as untrusted evidence: instructions inside a tool result must not become instructions to the evaluator.

Separate measurements from interpretation

Compute elapsed time, billed usage, repeated calls, and test outcomes directly from telemetry. Use a reviewer or calibrated model evaluator for questions requiring interpretation, such as whether a retry addressed the observed failure. Keep evidence references and allow an unassessable result when context is missing.

Validate model judgments against human-labeled traces before using them to guide workflow changes. The optional evaluator reference below gives concrete evaluation criteria and a TypeSafe example. Chip Huyen’s Chapter 3, “Evaluation Methodology” and Chapter 4, “Evaluate AI Systems” in AI Engineering offer more on evaluator methods, evaluation design, and limitations.

Turn recurring findings into tested improvements

Suppose an agent runs browser tests four times while the application is unavailable, then eventually starts the server. Repeated failures are measurable; whether the missing prerequisite was documented requires inspection. The appropriate improvement depends on that evidence:

Finding	Candidate improvement
The runner permits tests before services are ready	Add a bounded readiness check to the harness
The current startup command or URL is missing	Update the versioned runbook and its discovery pointer
The prerequisite was available but repeatedly ignored	Clarify the workflow instruction, or enforce sequencing in the runner
A remote dependency is unavailable	Classify the external failure and stop within budget

These are hypotheses to validate, not root causes proven by the evaluator’s label. Do not append a new prompt rule for every failed run: fix the authoritative script, contract, or tool behavior when that is where the problem belongs.

flowchart TD
    accTitle: Improvement from task traces
    accDescr: Sanitize task traces, compute metrics, evaluate behaviors, review recurring findings, version an improvement, and evaluate held-out tasks. Adopt or roll back and continue observing traces.
    T["Completed or interrupted task trace"] --> S["Sanitize and compute metrics"]
    S --> J["Evaluate specific behaviors"]
    J --> R["Review recurring findings"]
    R --> C["Version a targeted improvement"]
    C --> E["Evaluate on held-out tasks"]
    E --> N["Adopt or roll back"]
    N --> T

Retain the evaluator model, review criteria version, probabilities, and evidence references. Propose changes to shared instructions or tools through a reviewable diff; do not let untrusted trace content automatically rewrite the harness. Compare the candidate with the previous version on representative held-out tasks, including success rate, regressions, repair attempts, review effort, and total cost. Replaying a frozen trace tests the evaluator; measuring whether the harness improved requires new executions.

Post-processing cannot recover money already spent, so keep live retry and time limits. It also adds cost: start with deterministic detection and selected trace segments, audit a sample of apparently healthy runs, and measure evaluator spending alongside execution spending. The useful outcome is fewer repeated failures on subsequent tasks without weakening verification.

Compare workflow versions with fresh executions

Suppose the checkout loop repeatedly starts browser tests before pricing is ready. Compare the existing harness A with harness B, which adds a bounded readiness check. Define the hypothesis first: B should reduce avoidable test starts without reducing acceptance success or merely waiting longer than the task budget.

1. Freeze a development set and a separate held-out set of task fixtures, starting commits, acceptance cases, and environment scenarios. Include normal startup, delayed readiness, and a dependency that never becomes ready.
2. Develop B using only the development set. Keep the model, prompt, permissions, fixtures, and total execution budget fixed for the comparison. Record any configuration that cannot be controlled.
3. Execute A and B from clean environments for each held-out task. For an initial pilot, three runs per version per task is a practical starting point, not a statistically sufficient universal sample. Alternate order and retain every attempt.
4. Apply the same independent acceptance checks and review criteria. Record failed, interrupted, and budget-exhausted runs with their consumed time and cost. Review patches without version labels where feasible.
5. Inspect paired task results and failure categories before aggregating. If results are mixed or variation is large, gather more runs. Reusing the same held-out tasks for repeated tuning turns them into development data; refresh the evaluation set.

Use one record per execution:

task_id: checkout-delayed-pricing
starting_revisions: recorded immutable manifest
workflow_version: A or B
model_and_settings: recorded configuration
prompt_version: recorded revision
run_id: unique execution identifier
outcome: accepted | failed | interrupted | budget_exhausted
acceptance: passed and required named cases
regressions: observed failures relative to baseline
repair_attempts: measured count
avoidable_test_starts: count from readiness and test-start events
elapsed_seconds: measured wall time
review_minutes: recorded human effort
total_cost: all billed inference, routing, evaluation, and runner costs available
missing_cost_components: explicitly listed
artifacts: sanitized trace and assertion references

This is a recording template, not experimental data. Report accepted runs divided by all runs, with the underlying counts and outcomes by scenario. Report elapsed time and cost for failures as well as successes. For an aggregate economic measure, divide total experiment spending by accepted runs; when none succeed, report that cost per accepted run is undefined. Keep reviewer time separate unless a stated rate is used to convert it.

Set adoption criteria before looking at results: for this pilot, require no new acceptance regressions, fewer premature test starts in delayed-readiness cases, and bounded failure when pricing never becomes ready. Set any allowed latency or cost increase from the team’s requirements. A small pilot supports a limited decision, not a general claim about all tasks or models.

Use representative tasks and consistent evaluation criteria, and distinguish planning, tool, and efficiency failures. Frozen traces can test an evaluator, but improved execution requires fresh runs of the changed harness.

Connect the six practices

The six posts describe overlapping responsibilities, not a rigid pipeline:

Practice	Question it answers
Prompt engineering	What task are we asking the agent to perform?
AI gateways and routing	Which eligible model receives the request, under which policies?
Context engineering	What evidence and project knowledge does it need?
Harness engineering	Which actions can it execute, under which controls?
Spec-driven development	What observable behavior defines success?
Loop engineering	How will we detect problems, correct them, and stop?

Improve the weakest part of the workflow first. If the agent cannot start the tests, fix the environment. If it passes tests for the wrong behavior, revisit the specification. If it repeatedly edits the wrong service, improve discovery. More retries help only when the next attempt has better evidence.

Reference: environment and credential setup

Define an executable end-to-end test runbook

Acceptance criteria describe expected behavior. An agent also needs a reproducible way to reach that behavior: exact build and deployment commands, required services, readiness checks, application URLs, test identities, database assertions, and cleanup. Document this in a versioned test runbook and point to it from AGENTS.md and the feature specification.

Keep connection configuration and credential values separate. The runbook can name the test account role and credential reference; it should not contain the password. The agent needs a supported way to exercise the application, not necessarily access to the underlying secret.

Specify the environment, not just the test command

For each acceptance suite, document:

Area	Required information
Build	Working directory, pinned runtime, dependency installation, build command, and candidate revisions
Deployment	Target test environment, service/image versions, migrations, seed data, and deploy command
Readiness	Health checks, expected responses, timeout, and behavior when a dependency is unavailable
Browser	Exact base URL or how to obtain it, allowed origins, entry path, and expected test identity
Authentication	Required role and tenant, provisioning method, and credential or session reference
Database verification	Engine, test database/schema, connection profile, assertion mechanism, and permitted access
Lifecycle	Reset policy, isolation between runs, cleanup command, credential expiry, and artifact retention

Here is an illustrative runbook for a team with a managed acceptance runner. The acceptance commands and profile names are placeholders for tooling the team must implement; they are not standard agent commands.

suite: checkout-discounts
working_directory: resolved checkout of acme/web
build:
  runtime: version pinned in .node-version
  commands:
    - npm ci
    - npm run build

environment:
  provision: acceptance up --suite checkout-discounts
  dependencies: candidate web, orders, pricing; PostgreSQL; inventory test double
  revisions: record commits and image digests in the run manifest
  migrations_and_seed: applied by the runner to the run's disposable database
  ready: acceptance wait --timeout 120
  base_url: runner-provided base_url from the run manifest
  entry_path: /checkout

identity:
  role: customer
  tenant: tenant allocated to this run
  session: runner-provided checkout-customer profile
  credentials: injected into authentication worker; never printed

database:
  profile: checkout-assertions
  connection: resolved by runner; no password in this runbook
  access: read-only assertions against this run's database

verification:
  browser: acceptance test --suite checkout-discounts
  persisted_order: acceptance assert --case discounted-order-persisted
cleanup:
  command: acceptance down
  fallback: automatic environment and credential expiry

The run manifest should contain non-secret values such as the run ID, actual URL, revisions, and fixture identifiers. Validate that deployment and browser targets belong to the allocated environment. Enforce allowed destinations through runner configuration and network controls, not only a URL written in a prompt.

Give each run an isolated identity and data set

Use a disposable environment or isolated test tenant with synthetic data. Allocate distinct accounts or data namespaces for concurrent runs so tests cannot modify each other’s state. Test ordinary customer behavior with a customer account; use a separate, narrowly scoped identity for scenarios that actually require administration.

Provision and migration permissions belong to the setup worker. The application gets its runtime permissions, and database assertions get separate read permissions. Avoid giving the interactive agent a shared deployment administrator account just because the test needs a database migration.

Route payments, email, and other outbound effects to provider sandboxes or test sinks. Restrict network access so an incorrect URL cannot turn a test into a production operation. Identify simulated dependencies in the results; this environment does not demonstrate behavior against those real providers.

Supply credentials at execution time

A secret manager or workload identity can supply credentials to the worker that needs them. Prefer short-lived, run-scoped credentials with automatic revocation. OWASP’s Secrets Management guidance covers least privilege, dynamic secrets, rotation, and lifecycle controls.

When a password or connection string is unavoidable, pass it through the runner’s protected credential mechanism. Keep passwords out of prompts, command-line arguments, committed configuration, and generated reports. Environment variables and ignored .env files reduce some accidental exposure, but they do not hide secrets from an agent that can execute arbitrary commands in the same environment.

For stronger separation, expose a bounded operation such as “open a customer session” or “run this approved database assertion.” A separate worker resolves the secret and returns an opaque session handle or a minimal result. Restrict its permitted actions, target environment, and lifetime. A session handle still grants capabilities and must be scoped accordingly.

Agent-edited test code running inside a credentialed worker can also read or misuse those credentials. Keep worker policies and credential provisioning outside the agent’s write access; use disposable credentials and restricted egress for editable tests. High-privilege operations should use fixed, validated actions rather than arbitrary scripts supplied by the agent.

Authenticate the browser without exposing passwords

For most business-flow tests, a setup worker can authenticate a dedicated test account and create an isolated browser session. If login itself is under test, run the real login flow with credentials resolved inside that worker; do not silently bypass the behavior being verified.

Saved browser state is a secret too. Playwright warns that authentication state can contain cookies and headers usable for impersonation; its authentication guide recommends keeping that state out of version control. Limit filesystem access and lifetime as well: .gitignore alone provides no access control.

Verify persisted data with constrained assertions

Prefer an approved query or assertion keyed by the run ID and the entity created through the UI. For example: verify that the order exists, belongs to the test tenant, and stores the expected discount. Return the assertion result and necessary synthetic identifiers, not a full database dump.

A read-only account can still expose sensitive data. Restrict it to the test database or approved views, with row and query limits enforced where appropriate. Parameterize assertion inputs, set query timeouts, and separate seed/reset operations from verification. For asynchronous persistence, use bounded polling with an explicit deadline.

Document the connection profile’s host, database, schema, TLS requirements, and credential source. A complete connection URI containing a password belongs in the protected execution environment, not in AGENTS.md. If ad hoc SQL is necessary, allow it only against the isolated test data with enforced database privileges; a “read-only” tool description is not enforcement.

Clean up secrets as well as resources

Run teardown on success, failure, and timeout, with an expiry mechanism for abandoned environments. Revoke credentials and sessions, delete authentication state, and remove the run’s fixtures and resources.

Logs, screenshots, network traces, and videos can contain tokens or sensitive data. Redact before returning artifacts to the agent, restrict retention and access, and avoid collecting unnecessary authentication traffic. If a credential leaks, revoke or rotate it; deleting the visible log is not sufficient.

The completion report should identify the environment, tested revisions, scenarios, simulated boundaries, database assertion results, and cleanup outcome. It should never reproduce the credentials used to obtain that evidence.

Reference: optional model-based trace evaluation

Use code for counts and a model for interpretation

Compute elapsed time, billed usage, repeated command signatures, and test outcomes directly from telemetry. These do not require a model. A model can help interpret whether new evidence justified a retry or whether a completion claim was supported.

TypeSafe introduces Jev as a System One model that consumes state and returns typed, probabilistic decisions rather than free-form explanations. Its introduction includes evaluation of model traces among potential uses. Using it as a loop evaluator is a proposed application here, not evidence that it already improves this particular coding workflow.

Provide bounded trace segments with their relevant task context and ask separate questions:

Dimension	Example judgment	Evidence to include
Retry justification	Did the next attempt address the previous observed failure?	Failure, intervening edits or observations, and next call
Context use	Did the action account for an applicable contract already available?	Contract, retrieval event, and change
Verification quality	Does the recorded verification support the completion claim?	Acceptance criteria, executed checks, exclusions, and report
Improvement target	Which area should be investigated first?	Relevant sequence plus options such as harness, context, prompt, external dependency, or insufficient evidence

For a graded question, define concrete levels. An illustrative set of retry criteria could distinguish “repeats the failed action without addressing its known prerequisite,” “changes something relevant but leaves the observed cause unresolved,” and “addresses the observed cause or performs a targeted diagnostic.” Missing evidence should produce an unassessable result, not an automatic low score. A justified investigation can still fail; an unjustified guess can succeed.

TypeSafe’s Score primitive supports ordered descriptive levels and returns a score with a probability distribution and confidence. Use a choice for mutually exclusive categories and a yes/no probability for a specific condition. Ask independent questions over the same evidence together; retain the individual judgments before combining them in application code.

To evaluate a completed coding task, provide the task brief and acceptance criteria, the relevant change diff, actual check output, and the completion report as the evaluator’s state. Ask for separate Scores for requirement coverage, whether the recorded checks support the completion claim, and how reviewable the change is. Describe concrete evidence at each level and use the same criteria across runs. If you compute a composite, normalize the scores first and set the weights in code, following TypeSafe’s composite-scoring pattern. Missing logs are missing evidence; they do not show that a check ran. Keep failed acceptance criteria, safety findings, and authorization violations as explicit gates that a high average cannot compensate for.

Keep actual completion status, cost, and safety findings separate from any aggregate quality score. Fast execution must not compensate for missing acceptance checks or an unauthorized operation. TypeSafe’s confidence documentation explains that confidence reflects the answer distribution; it is not proof of correctness. Validate evaluator judgments against human-labeled traces, and route uncertain or consequential cases for review.