Cheaper Models and a Strong Harness: Building Distributed Systems in Rust with AI Agents

In the early days of autonomous coding agents, industry conventional wisdom seemed straightforward: achieving high-standard software demanded the largest, most expensive frontier model available. Engineers loaded entire repositories into massive context windows, hoping raw LLM intelligence would magically resolve distributed edge cases, tricky concurrency bugs, and infrastructure manifests.

The reality, however, fell short. Unconstrained frontier models often hallucinated non-existent library APIs, introduced subtle race conditions, drifted away from team architectural standards, and ran up astronomical API bills during manual debugging loops.

Over the past year, a quiet transformation has taken place across two fronts:

1. The Rise of Cost-Effective Models: Smaller, ultra-fast, and economical models (such as Flash-tier and compact instruction-tuned models) have become exceptionally capable at tool calling, structured outputs, and targeted reasoning.
2. The Maturation of Strong Harnesses: We realized that the quality of AI-generated software is not primarily a function of LLM parameter count. Rather, it is an emergent property of the environment, feedback loops, and deterministic verifiers that constrain and guide the agent.

By combining fast, inexpensive models with a rigorous, automated engineering harness, you can achieve outcomes identical to or better than unconstrained frontier models—at a fraction of the token cost and latency.

To test this hypothesis to its practical limit, I adopted an AI-first approach to build five complex distributed systems inspired by Byte Byte Go:

• URL Shortener
• Distributed Web Crawler
• Multi-Channel Notification System
• OpenTube (Video Streaming Platform)
• Search Autocomplete System

Each system was built on a standardized foundation using our Microservices Template in Rust, running on Kubernetes, featuring user interfaces designed via OpenDesign, strictly validated with SonarQube, and quality-scored on agent runs using Jev (TypeSafe AI’s System One model).

[!NOTE] A note on speed and pragmatism: These five reference architectures were built in record time as rapid proofs of concept. Due to this high development velocity, they undoubtedly contain edge-case bugs and areas for refinement. Nonetheless, they stand as undeniable proof that AI can genuinely assist and accelerate real-world software engineering.

Below, I share the principles, architecture patterns, and practical lessons learned along the way.

---

The Core Thesis: The Harness Outperforms Raw Intelligence

In our previous guides on Harness Engineering and Loop Engineering, we defined an agent’s harness as its boundary layer: the runtime environment, tools, execution sandboxes, and verification sensors through which the model interacts with source code.

When you rely solely on model intelligence without a well-designed harness, you demand probabilistic perfection. When you wrap an agent in a robust harness, the model only needs to propose locally plausible solutions. The harness assumes responsibility for deterministically catching failures, exposing compiler errors, enforcing safety boundaries, and guiding autonomous self-correction.

flowchart TD
    accTitle: Verification loop in a strong harness
    accDescr: An AI agent using a cheap model proposes changes to a microservice template. The code undergoes Rust compiler checks, SonarQube static analysis, Kubernetes smoke tests, and Jev semantic scoring before reaching human review.
    A["Requirements & Constraints Specification"] --> B["Fast / Cheaper Model (Proposal Generation)"]
    B --> C["Standardized Scaffolding (microservices-template)"]
    C --> D["Rust Compiler & Clippy (Type & Memory Oracle)"]
    D -- "Compilation Errors" --> B
    D -- "Build Succeeded" --> E["SonarQube Quality Gate (Security & Code Smells)"]
    E -- "Quality Gate Failed" --> B
    E -- "Quality Gate Passed" --> F["Kubernetes Smoke Tests & Health Probes (/healthz)"]
    F -- "Runtime Failure" --> B
    F -- "Probes Healthy" --> G["Jev (TypeSafe AI) Semantic Quality Scoring"]
    G -- "Score Below Threshold" --> B
    G -- "Score Approved" --> H["Verified PR Ready for Human Review"]

This architecture shifts the burden of correctness from neural network weights to an external, deterministic pipeline. A model costing $0.10 per million tokens can quickly iterate over compiler diagnostics until producing mathematically sound code, outperforming $15.00 frontier models that fail silently in production.

---

Pillar 1: Why Rust Is the Ideal Language for AI Agents

Many teams pick dynamic or permissive languages (like Python or JavaScript) for coding assistants, assuming they are “easier” for models. In practice, the exact opposite is true. Dynamic languages conceal bugs until runtime, forcing the agent to decipher ambiguous stack traces or navigate complex debugging setups.

Rust is a statically typed systems language featuring ownership semantics, memory safety without a garbage collector, and an expressive algebraic type system. For an AI agent operating inside a harness, Rust is the ideal partner:

1. The Compiler as a Deterministic Oracle: If code compiles in Rust, entire classes of distributed bugs—such as null pointer dereferences, data races across threads, use-after-free conditions, and unhandled enum variants—are mathematically eliminated.
2. Actionable Diagnostics: rustc and cargo clippy do not merely flag errors; they explain why a borrow failed and suggest the exact fix (help: consider borrowing here: '&'). Cheaper models ingest these structured diagnostics and correct the code within one or two rapid iterations.
3. Safe Concurrency by Construction: Implementing lock-free data structures, Tokio communication channels, and asynchronous I/O in distributed systems demands strict discipline. Rust enforces compile-time thread safety via Send and Sync traits.

// Example: Strict domain modeling in the URL Shortener
// Compile-time invariants prevent invalid states from ever reaching storage
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ShortUrl {
    id: UrlId,
    original_url: Url,
    created_at: DateTime<Utc>,
    expires_at: Option<DateTime<Utc>>,
}

impl ShortUrl {
    pub fn new(raw_url: &str, ttl: Option<Duration>) -> Result<Self, DomainError> {
        let original_url = Url::parse(raw_url).map_err(|_| DomainError::InvalidUrl)?;
        let created_at = Utc::now();
        let expires_at = ttl.map(|d| created_at + chrono::Duration::from_std(d).unwrap());

        Ok(Self {
            id: UrlId::generate(),
            original_url,
            created_at,
            expires_at,
        })
    }
}

When the agent proposes a business logic change, the type system prevents invisible regressions.

---

Pillar 2: Standardized Foundations with microservices-template

An agent placed into an empty repository spends tens of thousands of tokens debating folder structures, logging libraries, and environment variable loaders.

To ensure immediate productivity, every service was bootstrapped from the microservices-template. This template establishes a clean hexagonal architecture (ports and adapters):

• Domain Core: Pure domain entities and business rules with zero external dependencies.
• Ports: Trait definitions for repositories, message brokers, HTTP clients, and caching layers.
• Adapters: Concrete implementations (PostgreSQL via SQLx, Redis via redis-rs, Kafka via rdkafka, gRPC via Tonic, and HTTP via Axum).
• Observability & Metrics: Structured JSON telemetry with tracing, trace propagation via OpenTelemetry, and Prometheus metrics exposed at /metrics.
• Kubernetes Manifests: Multi-stage distroless Dockerfile builds and declarative manifests with Readiness and Liveness probes (/readyz, /healthz), ConfigMaps, and resource limits.

Because the foundation is predictable, prompts sent to cost-effective models do not waste context explaining how to configure logs or database pools. The model’s reasoning capacity remains 100% focused on distributed business logic.

---

Pillar 3: Cloud-Native Verification with Kubernetes

A microservice responding cleanly on localhost:8080 often breaks under internal pod DNS resolution, secret injection, autoscaling events, or network segmentation policies.

In our harness, the agent validates solutions directly against local Kubernetes clusters (using kind):

• Declarative Manifests: The agent generates and validates Deployments, Services, ConfigMaps, and Horizontal Pod Autoscalers (HPAs).
• Health & Readiness Probes: The harness queries /healthz and /readyz endpoints to verify startup order and graceful degradation when dependencies are unavailable.
• Inter-Service Communication: Services communicate over CoreDNS internal domains (such as http://storage-service.default.svc.cluster.local:8080), compelling the agent to configure network routing accurately.

---

Pillar 4: Strict Static Auditing with SonarQube

Code compiling in Rust does not guarantee immunity from excessive duplication, high cyclomatic complexity, or future maintainability issues.

Our harness integrates SonarQube as a mandatory quality gate. Once code passes cargo clippy and unit tests, automated static analysis triggers. The agent can only conclude its task when meeting the configured Quality Gate:

• Zero Security Hotspots and zero security vulnerabilities.
• Zero Bugs across static execution paths.
• Duplicated Lines Density strictly below 3%.
• Maintainability Rating A, ensuring small, modular functions.

If SonarQube flags an overly complex function or redundant adapter logic, the harness feeds the exact rule violation back to the agent for targeted refactoring.

---

Pillar 5: Rapid UI Generation with OpenDesign

Distributed backends require clear administrative and user-facing dashboards. Rather than letting the agent generate unstyled CSS or disconnected components from scratch, we integrate OpenDesign (via Model Context Protocol).

The agent gathers screen requirements, calls OpenDesign tools to instantiate cohesive design system components, and delivers modern, responsive, accessible frontends fully wired to the Rust APIs. This eliminates discrepancies between backend OpenAPI contracts and user-facing interfaces.

---

Pillar 6: Objective Evaluation with Jev (TypeSafe AI)

A major hurdle in AI agent engineering is objectively verifying whether an output is truly production-ready or merely masking defects. Asking another generative LLM for a code review frequently yields wordy, flattering, non-committal feedback (“The code looks great and is well-formatted!”).

To overcome this, we integrate Jev, the flagship System One model from TypeSafe AI. Unlike generative models producing freeform prose, Jev acts as a typed software primitive: it consumes natural language context, Git diffs, and test outputs, returning structured judgments and calibrated success probabilities.

// Conceptual harness integration with TypeSafe JEV
// Returns typed scores and probabilities instead of verbose opinions
let evaluation = jev_client.evaluate_agent_run(AgentRunContext {
    specification_id: "SPEC-003",
    git_diff: &diff_summary,
    test_results: &test_run_output,
    sonarqube_metrics: &sonar_report,
}).await?;

if evaluation.adherence_to_spec.score < 0.85 {
    return Err(HarnessError::SpecDivergence(evaluation.adherence_to_spec.reason));
}

if evaluation.production_readiness.probability < 0.90 {
    return Err(HarnessError::ReadinessRisk(evaluation.production_readiness.factors));
}

By scoring critical dimensions—such as functional specification adherence, error handling completeness, and API ergonomics—the harness programmatically decides whether a pull request can proceed or must be routed back to the agent with targeted remediation instructions.

---

The Testbed: Five Distributed Systems from Byte Byte Go

To stress-test this methodology across diverse real-world workloads, we implemented five classic distributed system designs inspired by Byte Byte Go:

Project	Key Architectural Challenges	Harness Validations Performed
URL Shortener	High-throughput Base62 encoding, distributed Snowflake ID generation, Redis caching, and Cassandra/PostgreSQL persistence.	Rate limiter load tests, sub-2ms redirect benchmarks, and cache stampede protection.
Web Crawler	Distributed crawl frontier queue, concurrent Tokio workers, robots.txt politeness delays, and SimHash content deduplication.	Per-domain rate limit enforcement, partitioning tests, and exponential backoff on HTTP 429/503.
Notification System	Multi-channel fan-out (Email, SMS, Push Notifications), prioritized queues, per-user rate limiting, and idempotent delivery guarantees.	Idempotency key deduplication, broker failure recovery, and external gateway simulation.
OpenTube	Async HLS video transcoding pipeline, presigned object storage URLs, simulated CDN edge caching, and a reactive player.	Video segment (.ts) checksum validation, multipart upload resumption, and OpenDesign UI integration.
Search Autocomplete	In-memory Trie with frequency-ranked terms, top-K caching, dynamic rank updates, and n-gram indexing.	Sub-millisecond prefix query benchmarks, memory profiling, and write-lock concurrency stress tests.

Practical Highlight: The Search Autocomplete Trie in Rust

In the search autocomplete system, the agent needed to implement an in-memory Trie supporting high-concurrency reads while an async background worker continuously updated query frequency rankings.

On its initial pass, a cheaper model struggled with Rust’s Arc<RwLock<TrieNode>> semantics, triggering borrow checker errors when attempting to hold read guards across async suspension points (await).

Because the harness fed back the exact compiler error:

error[E0277]: `RwLockReadGuard<'_, TrieNode>` cannot be sent between threads safely
   --> src/trie/engine.rs:42:13
    |
42  |     tokio::spawn(async move {
    |     ^^^^^^^^^^^^ `RwLockReadGuard` cannot be sent across await boundaries

The model immediately recognized that locking a std::sync::RwLockReadGuard across an await point would block Tokio runtime threads. In a single follow-up iteration, the agent refactored the logic to snapshot top-K results prior to the I/O call, resolving the compiler error and preventing async deadlocks.

Without a strict compiler acting as an automated sensor, an unguided frontier model would have generated seemingly clean code that failed silently under concurrent production load.

---

Economic and Efficiency Comparison

Comparing traditional frontier-heavy development against a strong-harness approach demonstrates compelling advantages:

Dimension	Frontier Model (No Harness)	Cheap Model (Weak Harness)	Cheap Model (Strong Harness)
Input Cost (per 1M tokens)	$3.00 – $15.00+	$0.05 – $0.20	$0.05 – $0.20
Output Cost (per 1M tokens)	$15.00 – $75.00+	$0.20 – $0.80	$0.20 – $0.80
Debugging Strategy	Verbose conversational reasoning	Blind trial-and-error	Deterministic compiler & test feedback
Quality Validation	Model self-assessment	Hope / Manual human review	Rust Compiler + SonarQube + Jev
Architectural Drift	High (reinventing patterns per task)	High (unstandardized code)	Low (anchored by microservices-template)
Delivery Reliability	Variable (confident hallucinations)	Very low (runtime crashes)	High (mathematically verified and tested)

By using faster, cost-effective models, teams can run 5x to 10x more iterations within self-correcting loops while keeping overall API expenses over 80% lower. Rapid iteration combined with unyielding automated verification produces more reliable software than one-shot frontier model runs.

---

How to Implement an AI-First Harness in Your Team

To adopt this methodology across your organization:

1. Pick a Rigorous Compiler: Favor languages with strict type systems (Rust, Go with advanced linters, or strict TypeScript). Let the compiler shoulder primary responsibility for catching mistakes early.
2. Build Reusable Foundation Templates: Maintain consistent templates (like our microservices-template) with built-in metrics, structured logging, error handling, and container configurations.
3. Automate External Quality Gates: Integrate SonarQube for static analysis and code smell prevention, local Kubernetes clusters for runtime verification, and visual tools for UI review.
4. Use System One Models for Evaluation: Replace vague text reviews with calibrated, typed judgments using Jev. Measure specification adherence with objective metrics.
5. Invest in Fast, Inexpensive Iteration: Channel your budget into rapid, automated feedback cycles. Allow agents to self-correct until all deterministic checks pass.

---

Conclusion

The future of software engineering in the AI era is not about waiting for a flawless frontier model that never makes mistakes. The real competitive advantage lies in building resilient engineering harnesses that elevate fast, accessible, imperfect models into exceptional software builders.

While these projects were built rapidly and naturally harbor bugs and edge cases, they offer tangible proof that AI is already a powerful, practical assistant for modern software engineering.

By combining the architectural structure of microservices-template, the discipline of Rust, Kubernetes container orchestration, SonarQube static auditing, OpenDesign interface generation, and Jev’s objective evaluation, you can build production-grade distributed systems today—reliably and cost-effectively.

Explore the repositories, review the Kubernetes manifests, and clone the projects on GitHub:

• microservices-template
• url-shortener
• web-crawler
• notification-system
• opentube
• search-autocomplete-system