Creating System Design Documents with AI: A Practical Guide

Ask an AI tool to “design a payment system,” and it can produce a convincing document in minutes. The harder work is establishing whether the proposed components exist, the requirements are real, and the failure handling matches the API contract.

System design is the planning of how a system’s parts work together. A useful design document helps a team make and implement those decisions. It explains the problem, boundaries, alternatives, behavior under failure, and evidence still needed. AI can accelerate research, drafting, and consistency checks, while engineers remain responsible for the requirements and architectural decisions.

This guide turns the practices from Context Engineering, Harness Engineering, and Loop Engineering into a documentation workflow. The running example is a hypothetical order-export feature. Its numbers and constraints are illustrative, not production measurements.

1. Start with a brief that exposes unknowns

In this example, a job is an export that runs in the background, and a worker is the process that runs it. Before asking for an architecture, write a short summary in docs/design/order-export/brief.md:

# Order export

## Goal
Let an authenticated merchant export their own orders as a CSV file.

## Requirements
- R1: Only users with export permission can request or download an export.
- R2: Every query and download must respect merchant isolation.
- R3: Retrying a submission must not create a second job within 24 hours.
- R4: Jobs survive worker restarts and reach a visible terminal state.
- R5: Download access expires 24 hours after completion.

## Proposed targets — require stakeholder confirmation
- Up to 100,000 orders per export.
- Up to 20 new jobs per minute at peak.
- 95% of accepted jobs complete within 5 minutes at that load.

## Existing constraints
- Reuse the application authentication and deployment platform.
- Prefer existing infrastructure when it meets the requirements.

## Out of scope
Scheduled exports, new file formats, and cross-merchant reporting.

## Open questions
- Which fields may be exported, and which must be redacted?
- Must the export reflect a consistent point-in-time snapshot?
- How long should job metadata and stored files be retained?

Give requirements stable identifiers so the document can connect them to decisions and checks. Keep targets separate from observed measurements. A capacity estimate based on guessed row sizes should show those assumptions and explain how to measure them.

Resolve questions that change the design before treating it as implementation-ready. For example, snapshot consistency may change the query strategy, transaction duration, and database load. The AI should not silently choose a business policy.

2. Choose the tool mode and provide evidence

A chat interface can work well for a bounded document: attach the brief, sanitized source excerpts, existing contracts, and a template. Run the generated checks locally and return their actual output. A repository-connected assistant can inspect files, edit the document, and run commands when its environment permits those actions.

Use the tool your team can operate with approved data access. Check what it can actually read and execute; the model cannot verify a repository it has never seen. Avoid uploading credentials or unnecessary customer records as design context.

Start with an evidence-gathering prompt:

Read docs/design/order-export/brief.md and inspect the relevant code,
API contracts, migrations, deployment configuration, and operational docs.
Do not draft the proposed architecture yet.

Create evidence.md with:
- Confirmed facts, each linked to a source path and symbol or section.
- The repository revision inspected.
- Conflicting or stale sources.
- Assumptions and unanswered questions, explicitly labeled.
- Existing components we could reuse and evidence of their capabilities.

Distinguish current implementation from intended behavior.
Do not infer a queue, service, or provider guarantee from its name.
Prioritize questions whose answers would change the design.

For an existing system, prefer code and deployment evidence over a plausible description. For a new system, record stakeholder decisions and external dependency contracts. If sources conflict, preserve the conflict until someone resolves it.

3. Give the agent concise repository guidance

The AGENTS.md format provides a place for project instructions, including commands and conventions, and supports nested files for scoped guidance. Check how your particular tool discovers and combines those instructions. Do not assume that llms.txt, a hidden directory, or a custom environment variable participates in a universal loading hierarchy.

Keep stable guidance in AGENTS.md; keep the current feature’s requirements in its brief. For a design workspace, a small example is:

# Design documentation workflow

- Read the relevant brief before changing a design.
- Cite repository evidence for claims about current behavior.
- Label proposals, assumptions, and unresolved decisions explicitly.
- Preserve requirement IDs across the design and review checklist.
- Reuse existing infrastructure unless a requirement justifies a change.
- Explain failure handling, data ownership, and alternatives.
- Run the documented checks and report their actual results and limits.
- Do not label a design accepted based on lint results.

## References
- Domain vocabulary: docs/domain.md
- Security requirements: docs/security.md
- Design template: docs/design-template.md
- Validation commands: tools/README.md

Create or correct these reference paths for your repository. A short index is useful only when it points to maintained information. No magic line count guarantees good reasoning; remove duplication and put detailed procedures where they can be loaded when needed.

Instructions guide behavior. Filesystem permissions, tool restrictions, and CI controls enforce boundaries. A sentence in Markdown does not create a sandbox.

4. Draft decisions before expanding the document

Start with a small set of files:

AGENTS.md
.spectral.yaml
package.json
package-lock.json
docs/design/order-export/
  brief.md
  evidence.md
  design.md
  api.yaml
  diagrams/
    containers.mmd
    retry-sequence.mmd
  adr/
    0001-background-processing.md
  review.md

Add artifacts only when they help answer a design question. A small internal change may need one document and one diagram. A public API change may need a precise contract and compatibility plan.

Use this prompt to compare approaches first:

Using brief.md and evidence.md, compare synchronous export with a
background-job design that reuses confirmed existing infrastructure.

For each option, explain latency, database load, failure recovery,
operational effort, and implementation cost. Show any capacity arithmetic
with units and assumptions. Identify what needs a prototype or measurement.

Recommend an option conditionally on the stated requirements.
Record the recommendation as a proposed ADR with context, alternatives,
consequences, and conditions that would cause us to revisit it.
Do not mark unresolved stakeholder choices as approved.

For deeper guidance on architectural decisions and risk analysis, see Mark Richards and Neal Ford’s Chapter 21, “Architectural Decisions” and Chapter 22, “Analyzing Architecture Risk” in Fundamentals of Software Architecture.

Turn quality goals into review scenarios

Treat quality attributes and trade-offs as design drivers. Replace a generic request for a “scalable, reliable system” with scenarios that name a condition, an observable response, and a way to check it:

Characteristic	Example design question	Proposed evidence
Performance	At the agreed peak job mix, do 95% of accepted exports finish within five minutes, including queue time?	Representative load test with completion percentiles and queue age
Reliability	After a worker dies during file upload, does the job reach a terminal state or recover without exposing a partial file?	Fault-injection test and state-transition trace
Security	Can a user from another merchant infer or download an export using its job ID?	Cross-merchant authorization tests for status and download
Operability	Can on-call staff tell whether a backlog comes from database reads, workers, or file storage?	Dashboard and alert review using a staged failure

These are proposed acceptance scenarios; the team must confirm targets and run checks against an implementation or prototype. Ask the AI to rank the characteristics that matter most, describe what each candidate design improves or worsens, and record the trade-off in the ADR. A background worker may avoid a long-lived HTTP request while adding queue delay, durable state, and recovery work. “More scalable” is too vague to justify that cost.

Map domain terms and data ownership

Establish a shared domain vocabulary and explicit boundaries. For this example, ask a domain expert what an order means, whether cancelled orders belong in exports, and whether “completion” means a file was created or that a merchant can download it. Put agreed definitions in the brief and use them consistently in the API, job states, diagrams, and tests.

Draw the boundary of the existing orders domain before creating an “export service.” The orders system may own order facts while export processing owns job state and temporary files. Those are candidate boundaries, not a mandate to split deployments or databases. If the export worker reads an orders table directly, record whether that access respects the orders domain’s rules and what schema changes could break it. An owned query API or a purpose-built read model may be appropriate when direct access creates unacceptable coupling, but each adds latency and operational work.

Make data ownership and distributed transactions explicit trade-offs. For the order-export proposal, write down the authoritative owner of order data, job state, and file metadata; then trace what happens when the job record commits but queue publication fails. Prefer a recoverable path that the existing platform can support. If the proposal introduces a broker, outbox, or separate service, explain why a simpler in-process worker or database-backed job runner does not meet the requirements. Record the extra moving parts and the failure they solve in the ADR.

Review the domain language and boundaries in brief.md and design.md.
List terms whose meanings differ across product, API, and storage.
For each data item, name its authoritative owner and allowed readers.
Trace failure between every durable write and downstream side effect.
Compare the simplest design using existing infrastructure with the
proposed split, including coupling, consistency, recovery, and operations.
Flag unresolved ownership and business-policy questions for human review.

Then expand the selected proposal into a document with these sections:

Section	What the reviewer should learn
Status, owner, and scope	Who owns the proposal and what it changes
Requirements and evidence	Which facts, targets, and unknowns drive it
Context and boundaries	Who uses the system and what it depends on
Proposed design	Responsibilities, data ownership, and interactions
Contracts and data model	Requests, responses, states, constraints, and evolution
Failure behavior	Timeouts, retries, duplicates, partial success, and recovery
Quality attributes	Capacity, latency, security, reliability, and cost assumptions
Operations and delivery	Metrics, alerts, rollout, rollback, and migration
Alternatives and decisions	Why this choice fits and what it sacrifices
Verification and open questions	Evidence needed before implementation or release

Work through a capacity estimate

Use rough estimates to expose assumptions before a detailed design, and ask the AI to show units at every step. Suppose, only for this exercise, that all 20 peak jobs each contain 100,000 orders and that each serialized CSV row averages 2 KB:

Calculation	Illustrative result	What it tells us
20 jobs/min × 100,000 rows/job	2 million rows/min, or about 33,000 rows/s	Peak arrival workload if every job is maximal
100,000 rows/job × 2 KB/row	About 200 MB/job	Approximate uncompressed output size
20 jobs/min × 200 MB/job	About 4 GB/min, or 67 MB/s	Sustained output throughput needed just to keep up at this hypothetical peak
33,000 rows/s × 2 KB/row	About 67 MB/s	A cross-check on the output calculation

These are planning figures, not measured database or storage capacity. CSV escaping, compression, headers, read amplification, temporary files, concurrent queries, and retries can change the actual load. The five-minute 95th percentile target also cannot be proven from an average throughput calculation. Measure a representative export, a maximal export, and concurrent exports against the existing database and file store. Capture row-size distribution, query time, serialization rate, queue wait, and end-to-end completion time before selecting worker concurrency.

Try a sensitivity question: if only one in ten jobs reaches 100,000 rows, how much does the expected output rate change, and can the system still survive several maximal jobs arriving together? This separates typical capacity from the burst the design must absorb.

5. Generate contracts and diagrams from the same decisions

Use stable names across prose, API schemas, and diagrams. The C4 model offers context, container, component, and code views; use the levels that help your audience. A container view describes applications and data stores, not necessarily Docker containers. For more diagramming guidance, see Chapter 23, “Diagramming Architecture” in Fundamentals of Software Architecture.

For order export, first establish the actors, API, job store, worker, and file store that the proposal needs. Mark proposed components as proposed. A sequence view can then expose a particularly important failure:

The document should pair the diagram with a responsibility and failure table. It gives reviewers a quick way to spot an unowned recovery path:

Proposed component	Responsibility and owned state	Failure behavior to decide and test
Export API	Authorize requests; create jobs and scoped idempotency records in the job store	If the store write fails, return an error without claiming a job exists; if the response is lost after commit, a retry finds the same job
Job store	Hold durable job state, merchant ownership, lease or claim data, and result metadata	Detect abandoned work and make it eligible for a bounded retry; define how conflicting claims are resolved
Worker	Read authorized merchant data and produce a file for a claimed job	On a crash, release or expire the claim; prevent an incomplete file from becoming downloadable; bound repeated failures
File store	Hold completed output until its retention deadline	A failed upload leaves the job incomplete; a failed deletion is visible for cleanup and alerting

These rows describe candidate responsibilities, not evidence that the repository already has these components. A design that adds a queue must also say whether the job store or queue is authoritative for recovery. Ask the AI to trace crashes before and after each durable write, then inspect whether retrying can create duplicate files or expose partial output. Brendan Burns’s Chapter 11, “Work Queue Systems” in Designing Distributed Systems is a useful companion for queue and worker patterns.

sequenceDiagram
    accTitle: Retrying an export submission after a lost response
    accDescr: The API commits a job and its idempotency record, but the client misses the response. A retry with the same merchant, key, and payload returns the existing job.
    participant U as Merchant client
    participant A as Export API
    participant D as Job store
    U->>A: POST export with idempotency key
    A->>A: Authenticate and authorize merchant
    A->>D: Atomically persist job and scoped key
    D-->>A: Job committed
    Note over U,A: Response lost before client receives it
    U->>A: Retry same key and payload
    A->>A: Authenticate and authorize merchant
    A->>D: Find merchant-scoped key and payload hash
    D-->>A: Existing job
    A-->>U: Return existing job identifier

This is a proposed interaction, not proof of a working implementation. The design must also address concurrent submissions, transaction failure, key expiry, and the same key with a different payload. If a broker is involved, explain how a committed job becomes discoverable after a crash between database commit and message publication.

Generate the API contract with those decisions in view:

Draft api.yaml and update design.md together.
Define submission, status lookup, and authorized download behavior.
Specify authentication, merchant scoping, idempotency-key scope and expiry,
same-key/different-payload behavior, job states, and error responses.

Use the same identifiers and state names in all artifacts.
Document how durable jobs are discovered and retried after worker failure.
Flag missing decisions instead of inventing provider guarantees.
Keep the document a proposal until the review criteria are met.

For R5, “the signed URL expires” is insufficient by itself. The design must prevent issuing a fresh URL after the job’s access deadline and prevent an earlier URL from outliving that deadline. File deletion and metadata retention are separate policies to resolve.

Review the API as a consumer

An OpenAPI file can describe valid requests while leaving clients unsure what to do next. The contract should evolve without surprising consumers, and the design should identify those consumers, the API owner, and how changes are managed. Apply these checks to the export API before implementation:

Client situation	Contract question the design must answer
Submission succeeds, but the response is lost	Can the client safely retry with the same merchant-scoped key and payload, and does it receive the original job identifier?
The key is reused with different parameters	Which stable error response tells the client that the request conflicts with the original submission?
The job is queued, running, complete, or permanently failed	Which states, timestamps, result links, and error details can the client observe? Which states are terminal?
The client polls too often or the service is overloaded	What rate limit or retry guidance is returned, and how should the client respond?
The file is ready or access has expired	How does the client distinguish a pending result, an expired download, and a job it is forbidden to view?

Specify these cases in api.yaml with representative request and response examples. Confirm status codes, headers, and error shapes with existing repository conventions instead of letting the AI choose them from memory. The client should be able to implement a complete submission-to-download flow from the contract and examples, including failure handling.

Name the consumers and the team that owns the API. Before changing a published contract, compare the candidate specification with the previous version and review semantic changes: removing a field or changing the meaning of a job state can break a client even when both files pass lint. Adding an optional response field is usually easier to absorb, but actual consumer behavior still matters. Record how affected clients will be notified, migrated, and eventually moved off an old version when a breaking change is necessary.

Once an implementation exists, test the producer against the interactions real consumers depend on. Consumer contract tests or equivalent request-and-response fixtures can check retry, status, download, and error cases in CI. Keep a small end-to-end test for the full export journey as well. Spectral checks the configured description rules; neither it nor a passing consumer contract test proves the data isolation and recovery properties in the design.

Act as a merchant client integrating with this export API.
Read api.yaml and the design without inventing undocumented behavior.
Walk through submission, a lost response and retry, polling, a failed job,
successful download, and expired access. For each step, show the request,
expected response, and the next client action. List any ambiguity with its
contract location. Then compare the proposed contract with the last
published version and flag changes that could break known consumers.

6. Automate structural checks

Use repository-local, locked tooling so validation is reproducible. In the design workspace, install the chosen packages once, review the resolved versions, and commit the manifest and lockfile:

npm install --save-dev --save-exact @stoplight/spectral-cli @mermaid-js/mermaid-cli

Record a compatible Node runtime and the browser dependencies required by the Mermaid CLI. Subsequent runs should use npm ci. These commands belong in the repository containing the design artifacts; they are not prerequisites for writing Markdown in a chat.

Spectral requires a ruleset. Create .spectral.yaml:

extends: ["spectral:oas"]

Run explicit files through the local tools:

./node_modules/.bin/spectral lint docs/design/order-export/api.yaml \
  --ruleset .spectral.yaml --fail-severity=warn

mkdir -p build/design
./node_modules/.bin/mmdc \
  -i docs/design/order-export/diagrams/containers.mmd \
  -o build/design/containers.svg
./node_modules/.bin/mmdc \
  -i docs/design/order-export/diagrams/retry-sequence.mmd \
  -o build/design/retry-sequence.svg

The Mermaid CLI renders diagram definitions to files such as SVG. Keep the rendered output for visual inspection: labels can overlap or become unreadable even when rendering succeeds. Check that every required diagram was actually processed.

Use a bounded repair prompt after a failure:

Fix the attached validation failure in the named artifact.
Preserve the agreed requirements and explain any contract change.
Do not disable rules merely to make the check pass.
Run the same check again and report its exit code and remaining findings.
Stop after three unsuccessful repair attempts and report the blocker.

Three attempts is an example budget. Choose one appropriate to the task. Preserve the original tool output, including the file, rule, and location; do not manufacture a line number or diagnosis when the validator does not provide one.

7. Review semantics separately from syntax

A lint result establishes only what the configured rules checked. A render establishes that a diagram could be generated. Neither proves isolation, recovery, capacity, or business correctness.

Maintain a traceability table in review.md:

Requirement	Design evidence	Verification still needed
R1: Export permission	Authorization described on all three operations	Denied-role submission and download tests
R2: Merchant isolation	Ownership checks on jobs, queries, and downloads	Cross-merchant identifier and download tests
R3: Submission deduplication	Scoped unique key, payload comparison, atomic creation	Concurrent retries and lost-response tests
R4: Durable processing	Job states, worker lease, retry limit, recovery path	Crash injection before and after file creation
R5: Access deadline	Download authorization and URL lifetime bounds	Boundary-time and previously issued URL tests

Before implementation, these are planned tests. Label them as executed only when there is an implementation or prototype and recorded results.

Ask the AI for a separate critique pass:

Review the brief, evidence, design, API, ADR, and diagrams together.
Find contradictions, unsupported claims, and missing failure paths.
For each finding, cite the artifact section and affected requirement.
Explain a concrete scenario in which the proposed design fails.
Do not rewrite the proposal or approve it. Return prioritized findings
and the evidence or decision needed to resolve each one.

A fresh review context may reveal inconsistencies, but another model can share the author’s mistakes. Have the responsible engineers review the decisions, and involve security, operations, or product owners where their requirements are affected. Record accepted trade-offs explicitly.

For a repeated design-document workflow, a typed scorer such as TypeSafe Jev’s Score primitive can add a consistent semantic-quality signal. Include the approved brief, requirements, traceability table, design, API, ADR, diagram source or text descriptions, and results from checks that have actually run. Score dimensions such as clarity, requirement coverage, evidence traceability, and operational readiness separately, using concrete levels such as 0–4. Keep launch gates and owner approval outside any weighted score; a polished document cannot compensate for an unresolved requirement or missing verification. See Loop Engineering’s model-based evaluation pattern for safeguards when evaluating task results.

Make operational readiness reviewable

The System Design document template calls for named operational signals, owners, launch gates, and rollout milestones. For this example, add a compact checklist to review.md and replace the placeholders with team decisions:

Review item	Evidence to request before launch	Owner or decision needed
Completion target	Load test showing job completion percentiles at the agreed arrival rate and row-size mix	Product confirms target; service team measures it
Queue health	Dashboard for oldest queued job, jobs by state, retry count, and terminal failures; alert thresholds tied to user impact	Service and on-call teams agree thresholds
Isolation and access	Tests for cross-merchant job IDs, downloads, expired access, and log redaction	Security and service owners review
Recovery	Crash tests around job commit, claim, file upload, and completion; documented replay procedure	Service team demonstrates recovery
Rollout and rollback	Limited-audience rollout, stop condition, rollback steps, and cleanup of jobs/files created during a failed rollout	Release owner approves sequence

An alert name alone is not operational evidence. Record its threshold, measurement window, owner, and runbook action. Generate failure scenarios to inspect, especially retries and partial failures, and check each proposed mechanism against this system’s actual contracts.

8. Keep evidence and approval distinct

For occasional documents, a checklist and CI log are enough. If this becomes a repeated workflow, add a small validation runner with fixed check identifiers, command arguments, a working directory, timeouts, and retained output. Avoid executing arbitrary shell strings taken from generated task files.

Track structural verification and design review separately. A record might include the candidate revision, artifact hashes, tool versions, check results, outstanding questions, and reviewer decision. Any artifact change should invalidate the corresponding old evidence.

A runner that writes passing into an agent-writable JSON file provides bookkeeping, not an enforced approval boundary. If a gate matters, enforce it through controlled CI and review permissions. Run candidate validation code with restricted privileges, and review changes to the rules as carefully as changes to the document.

Use a final handoff prompt:

Summarize the changed artifacts and proposed decisions.
List commands actually run, their outcomes, and evidence locations.
Separate structural checks from behavioral tests and human review.
List unresolved assumptions with owners and next actions.
State whether the document is a draft or ready for design review.
Do not claim approval, implementation readiness, or deployment without evidence.

When a requirement changes, update its contract, diagrams, decision record, and verification plan in the same review. The document earns its place when a future engineer can trace a decision to its reason, identify what remains uncertain, and verify that the implementation still follows it.