TL;DR: Nolan Lawson benchmarked AI coding agents building the same Pokémon app in 12 JS frameworks/meta-frameworks, measuring cost, time, tokens, Lighthouse performance/accessibility, JS bundle size, TBT, LCP, and lines of code.
- Cheapest/fastest: Hono+htmx, Astro, Nuxt, Vanilla, and Next.js were generally lowest cost/time. SvelteKit, Gracile+Lit, and Angular were most expensive; Angular also failed to produce a working app twice.
- Lighthouse scores were mostly saturated: Performance/accessibility/CLS barely differed. Meaningful differences showed up in JS bytes: Astro ~0 KiB, Vanilla ~7, Gracile+Lit ~8, Hono+htmx ~18; React Router ~115, TanStack Start ~118, Next.js ~142.
- MPAs generally beat SPAs on TBT and LCP, as expected because Lighthouse/Core Web Vitals penalize client-side JS. Next.js, Angular, and Nuxt had worse TBT/LCP; Astro, htmx, Lit, and SvelteKit were strong.
- Lines of code: Hono+htmx was smallest (~1,370), Angular largest (~2,009). This matters in the agentic era because code size affects future token/cost of maintenance.
- Big caveat: Results are stochastic, based on one small content-heavy app, and Lighthouse is SPA-biased. The point isn’t “pick framework X” but make framework choices with evidence rather than vibes; benchmark is on GitHub.
How to Measure the AI Cost of Rust Frontend Frameworks: A Reproducible Benchmark Protocol for Hermes Agent
The choice of a frontend framework is no longer just a technical decision—it is a recurring line item in your AI development budget. Nolan Lawson’s 2026 benchmark demonstrated that building the same application with different JavaScript frameworks can cost anywhere from $2.67 to $4.97 per run, a spread of nearly 86% driven entirely by the framework’s interaction with the coding agent. For Rust frontends—Leptos, Dioxus, Yew, and others—no equivalent data exists. This article provides a complete, paste-ready protocol for Hermes Agent to generate that missing table. It specifies the task, the controls, the metrics, and the output format, so you can measure the true cost per task for every Rust framework in the ecosystem.
1. The Core Principle: Cost per Successful Task, Not Token Price
A common mistake is to treat token price as the cost. The Prompt-Induced Waste benchmark (2026) makes clear that end-to-end cost and task success depend jointly on prompt semantics, inference effort, harness policy, model, task difficulty, and context management. The same task can produce wildly different token counts depending on how the agent reasons, retries, and stops. Therefore, the metric you must collect is cost per successful task (or median cost across successful runs), not raw token spend. You must also record success rate; a cheap framework that fails half the time is not cheap.
2. Fix the Variables: What to Control
Before Hermes Agent runs a single prompt, lock down the following. Any variation here will contaminate your cost comparison.
| Variable | Recommended Setting | Rationale |
|---|---|---|
| Model | Claude Opus 5.5 (high effort) | Matches Lawson’s benchmark, enabling direct comparison with JS data. |
| Harness | Claude Code (or the harness with the lowest fixed overhead for your setup) | Harness design alone can change cost by 4.8× for an identical question. Keep it constant. |
| Effort | “High” or “max” | Lower effort may reduce cost but risks task failure, which invalidates the cost-per-success metric. |
| Runs per framework | 5 (median reported); minimum 3 if failures occur | Lawson used 5 runs; Angular required only 3 due to repeated failures. |
| Framework version | Latest stable release | Outdated versions may have poor training-data coverage, artificially inflating cost. |
| Official AI tooling | Enable MCP servers, LLMs.txt, or official skills only if the framework provides them |
This reflects real-world usage. Do not manually inject documentation; let the agent discover it. |
3. The Task Specification (Copy This to Hermes Agent)
Use a task that is small but non-trivial, exercises data fetching, state, and rendering, and has a deterministic conformance suite for validation. The following spec is adapted from Lawson’s Pokémon benchmark.
Task: Build a Pokémon Browser
Using the PokéAPI, build a web application with the following features:
- List view: Display a paginated list of Pokémon (name, sprite, types).
- Search: A text input that filters Pokémon by name (client-side or server-side).
- Type filter: A dropdown or set of buttons to filter Pokémon by type (e.g., Fire, Water, Grass).
- Detail view: Clicking a Pokémon opens a detail panel or page showing its stats and moves.
Constraints:
- Do not use Tailwind, Bootstrap, or any utility-first/component CSS library. Use plain CSS or the framework’s recommended styling approach.
- Do not pre-generate static HTML. The application must fetch live data from the API.
- Use the framework’s official scaffolding tool if one exists (e.g.,
cargo leptos new,dx new).- You will not be told when or how the application will be evaluated.
Success criteria (for your own validation, not the agent’s):
- The application builds without errors.
- All four features are functional in a browser.
- A conformance suite (Playwright or similar) passes at least 90% of checks against the accessibility tree, not raw DOM structure.
This spec is intentionally framework-agnostic. It does not mandate SSR or CSR, allowing each framework to play to its strengths, while the conformance suite ensures the output is genuinely usable.
4. Metrics to Collect (The Table Columns)
For each run, Hermes Agent must record the following. The first five are mandatory; the rest are strongly recommended.
| Metric | How to capture | Why it matters |
|---|---|---|
| Cost (USD) | Provider-reported token counts (input + output + cache) × list price at time of run. Include all retries and diagnostic loops. | The headline number. Use the same model and provider for all frameworks. |
| Input tokens | From API response or harness logs. | Captures context-reading cost. Rust’s syntactic density means it tokenizes at ~1 token per 3–4 characters, higher than many languages. |
| Output tokens | From API response. | Captures code-generation cost. |
| Time (minutes) | Wall-clock from prompt submission to final artifact. | Agent time correlates with cost but is not identical; a slow harness may still be cheap. |
| Success (Y/N) | Did the conformance suite pass ≥90%? If N, record the failure mode. | A cheap failure is worse than an expensive success. Median cost should be calculated over successful runs only, with success rate reported separately. |
| Lines of code (LOC) | Count non-blank, non-comment lines in src/ (or equivalent). |
Proxy for maintainability. Fewer lines mean lower future token costs for edits. |
| Bundle size (kB) | Release build, Brotli-compressed WASM + JS. | Captures the framework’s runtime floor. Leptos (180–260 kB) and Dioxus (220–320 kB) are lighter than Yew (260–400 kB). |
| Build time (s) | cargo build --release or equivalent, cold cache. |
Rust compile times are a real cost in agentic loops; a framework that forces frequent rebuilds is more expensive. |
5. Frameworks to Test (Minimum Viable Set)
At a minimum, test the three established web-first frameworks. Add the others if resources permit.
| Framework | Why include it | Expected cost driver |
|---|---|---|
| Leptos | Smallest bundle floor; fine-grained reactivity; has community MCP tooling. | Fine-grained model may confuse agents trained on React; but small output reduces future edit costs. |
| Dioxus | Largest community (~38.9k stars); React-like syntax; first-party MCP and LLMs.txt. |
React familiarity may lower iteration cost; bundle is mid-range. |
| Yew | Longest history; largest ecosystem among Rust frontends; virtual DOM. | Virtual DOM boilerplate and larger bundle floor may increase token spend. |
| Sycamore (optional) | Fine-grained, no virtual DOM. | Small ecosystem → sparse training data → likely higher cost. |
| Sauron (optional) | Elm-like architecture. | Niche architecture → very sparse training data → likely highest cost. |
| Iced / egui (optional) | GUI-first libraries that target WASM. | Not web-first; WASM bundles are large (Iced ~1.3 MB) → context and build costs are high. |
6. The Output Table (Paste This Format to Hermes Agent)
Hermes Agent should produce a table with exactly these columns. Each row is one framework; each numeric cell is the median of successful runs.
| Framework | Median Cost (USD) | Success Rate | Input Tokens (M) | Output Tokens (k) | Time (min) | LOC | Bundle (kB, Brotli) | Build Time (s) | Notes |
|---|---|---|---|---|---|---|---|---|---|
| Leptos | 180–260 | ||||||||
| Dioxus | 220–320 | ||||||||
| Yew | 260–400 | ||||||||
| Sycamore | ~200–300 (est.) | ||||||||
| Sauron | ~200–300 (est.) | ||||||||
| Iced | ~1,300 | GUI-first; WASM heavy | |||||||
| egui | Large | Immediate mode; not web-first |
After the table, Hermes Agent must also report:
- The spread:
(max median cost − min median cost) / min median cost, expressed as a percentage. This is the headline finding—Lawson’s JS spread was 86%. - Failure modes: For any framework with success rate <100%, describe the failure (e.g., “build error,” “hydration mismatch,” “agent used forbidden CSS library”).
- Outlier runs: If any run’s cost deviates by more than 2× from the median, report it separately. LLMs are stochastic; outliers are data, not noise to be discarded.
7. Critical Caveats (Report These Alongside the Table)
No benchmark is universal. Hermes Agent must include these limitations in its output:
- Task specificity. A content-heavy Pokémon browser may not predict cost for a dashboard, a real-time app, or a form-heavy workflow. The framework that is cheapest here may not be cheapest elsewhere.
- Stochasticity. Five runs is a small sample. Lawson himself noted that the cheapest React Router run ($2.82) was cheaper than the most expensive Hono+htmx run ($3.19) despite Hono+htmx having a lower median. Treat medians as directional, not definitive.
- Rust’s baseline tax. Rust code is more verbose and token-dense than TypeScript or Python. A medium Rust project (8k–20k LOC) consumes 80k–180k tokens just for the static codebase, before any agent reasoning. Every Rust framework will likely cost more per task than the cheapest JS frameworks; the relevant comparison is Rust vs. Rust.
- Tooling asymmetry. Dioxus and Leptos have official AI tooling; Yew, Sycamore, and Sauron do not. This is a real cost factor, but it is also a moving target—tooling may appear after this benchmark is run.
8. What “Good” Looks Like
If the protocol is followed correctly, you should expect to see a table where:
- Leptos and Dioxus are the cheapest Rust options, likely in the $3–$6 per task range for this spec (higher than JS due to Rust’s token density and smaller training corpus).
- Yew sits in the middle, with higher token counts driven by virtual-DOM boilerplate.
- Sycamore and Sauron are the most expensive, possibly exceeding $8–$10 per task, because the agent must reason from first principles rather than pattern-matching to training data.
- Iced and egui are outliers—their WASM bundles and non-web-first architectures make them unsuitable for this comparison, but including them documents why they are not web frontend frameworks in the agentic sense.
The goal is not to crown a winner. The goal is to replace “this framework feels better for AI” with a number you can put in a budget. Run the protocol, publish the table, and let the cost per successful task speak for itself.