The real
cost of AI

Research source document · Evidence reviewed through August 20, 2026

The real cost of AI: August 2026

Status: initial research snapshot. Evidence cut: 2026-08-20. Recheck by: 2026-11-20, or after a material model release, pricing change, or hardware generation shift.

This summary draws on the inference-economics notebook (17 captured references, 5 verified claims), the GPU ownership notebook (17 references including grade-A captures and a compiled 8-source pricing file, 4 verified claims plus the TCO crossover model), and the model-selection notebook (21 references, 6 verified claims incl. router economics and index-sensitivity findings). Counts as of 2026-08-21 (evening).

The short version

The sticker price of a token is not the cost of an answer. In August 2026, the same standard API workload (100K input + 20K output tokens) costs anywhere from $0.018 on Gemini Flash-Lite to $2.00 on Claude Fable 5 — a 111x price range across comparable models. Two corrections cut the other way: measured token-efficiency variance means the same input produces 2.65x+ more output tokens on some models, and cheap-token models can cost more per successful task via verbosity and retry loops — so the posted range is an upper bound on effective dispersion, and it still understates the real spread: adjusted for quality using the Artificial Analysis Intelligence Index, the cost-per-quality- point spreads by 47x. Read that 47x as order-of-magnitude evidence of non-linear pricing, not a precise ratio: the index is an editorially-weighted nine-benchmark composite (text/English-centric; agentic-weighted since v4.1), and version overhauls move scores by ~23 points — as large as the frontier spread itself. The marginal cost of capability accelerates sharply above Intelligence Index 50.

Running your own GPUs is a crossover problem, not a preference. Specialist GPU clouds (“neoclouds” like RunPod, Lambda, Together AI) charge 50-70% less than hyperscalers for the same H100: a median of $3.99/GPU-hour vs $7.89. The TCO model — now corrected for procurement scope — shows the answer splits by who’s buying: a lean operator adding GPUs to existing infrastructure (~$30K/GPU street price) breaks even against hyperscaler on-demand at ~21% utilization and beats median neocloud pricing above ~37%; an enterprise node-loaded buyer (~$94K/GPU all-in per io.net’s measured 3-year figure) breaks even vs hyperscaler on-demand only at ~57%, essentially never beats median neocloud pricing, and never beats cheap neoclouds or spot. Reserved commitments beat node-loaded on-prem at every utilization. Utilization assumption AND procurement scope together dominate the decision.

The control premium is real, and buyers pay it knowingly. Independent cost itemization puts true self-hosting at 3–5× the pure GPU price — matching the node-loaded math. For a compliance-driven subset, the premium is not optional: sovereign cloud IaaS spending is forecast at $80B in 2026 (+35.6%), Italy’s cloud market grew 20% YoY explicitly on sovereign-data demand, and the EU’s Technological Sovereignty Package (June 2026) extends data-governance obligations to AI providers. The other evidenced motives: guaranteed capacity with no rate limits or peak contention, latency for on-network workloads, vendor independence as a hedge against the list-price and subsidy moves forecast above, and very-high sustained utilization. In practice the control premium runs roughly 2–4× effective cost vs neoclouds — cost-optimal is not decision-optimal when compliance or capacity certainty is a hard constraint.

A significant fraction of current AI spend is subsidized — through provider credits, free tiers, academic programs, and below-cost pricing funded by venture capital — though the exact fraction is opaque. The effective cost of AI is materially higher than what most teams currently pay, and the subsidy distorts the build-vs-buy decision by making API inference appear cheaper than its true cost.

What the evidence establishes

Inference economics — five verified claims

The inference-economics notebook has 17 references and 5 verified claims:

  1. C1 (verified, 3 independent grade-A sources): The same API workload costs $0.018 to $2.00 across major LLMs — a 111x price range driven by the 21x blended-price spread in the frontier basket.

  2. C2 (verified, 3 independent grade-A sources): Quality-adjusted cost diverges from token price by 47x across the frontier basket — far more than the 2x the original hypothesis predicted. DeepSeek V4-Flash at $2/index-point vs Claude Fable 5 at $94/index-point.

  3. C3 (verified, 3 independent grade-A sources): The price-per-quality frontier is non-linear: models in the 40-50 Intelligence Index range offer 10-47x better cost-per-index-point than models in the 55-60 range. The marginal cost of capability accelerates above index 50.

  4. C4 (verified, 1 grade-A source): Closed-weight flagships carry a 10% premium over open-weight equivalents in blended price ($4.47 vs $4.08/Mtok), but the premium concentrates at the frontier, not uniformly across tiers.

GPU ownership — three verified claims and a crossover model

The GPU notebook has 7 references (a compiled 8-source evidence file grade C + direct captures including A6 grade A, A7 grade A) and 3 verified claims, plus the TCO crossover model (analysis/tco-crossover-2026-08.md):

  1. C1 (verified, 3 independent sources): Specialist GPU clouds (“neoclouds” like RunPod, Lambda, Together AI) charge 50-70% less than hyperscalers for the same H100 GPU-hour: median $3.99/GPU-hr vs $7.89/GPU-hr (+98%). The ~2x hyperscaler premium is consistent across H100, A100, H200, and B200.

  2. C2 (verified, 4 independent sources): On-premises GPU clusters break even with hyperscaler on-demand at roughly 50-83% sustained utilization, but rarely win against specialist GPU clouds at any utilization once fully loaded.

What this means for decisions

If you build with AI APIs

Don’t compare token prices — compare cost-per-quality-point for your workload. A model that is 10x cheaper per token but takes 3 retries to get a correct answer is more expensive than the one it replaced. The non-linear frontier means the biggest cost leverage is not picking the cheapest model, but picking the cheapest model that clears your quality bar — and that bar is workload-specific. Two caveats now carry the same weight as the rule itself: posted-price ranges are upper bounds (token-efficiency varies 2.65x+ by model), and the quality index behind any cost-per-quality ranking is an editorially- weighted composite — use it to find the non-linear region, not to rank models to a decimal.

If you run ML infrastructure

The GPU market is split: neoclouds at $3-4/GPU-hour, hyperscalers at $7-12. If you’re paying hyperscaler on-demand for sustained inference, you are likely overpaying by 2x. Whether owning beats renting now has a two-part answer. Procurement scope first: a lean team adding GPUs to existing infrastructure (~$30K/GPU) breaks even vs hyperscaler on-demand at ~21% utilization and beats median neocloud pricing above ~37% — but an enterprise node-loaded buyer (~$94K/GPU all-in) needs ~57% just against hyperscaler on-demand, essentially never beats median neoclouds, and never beats cheap neoclouds or spot. Utilization second: above ~70% sustained, on-prem wins on raw cost in the lean regime. And if you’re buying for sovereignty, compliance, or guaranteed capacity, you’re paying a 2–4x control premium on purpose — price it as insurance, not as infrastructure.

If you budget AI spend

Instrument cost per successful task, not cost per token. The 111x price range across models means a routing decision (cheap model for easy queries, expensive model for hard ones) is the single largest cost lever available — but the routing infrastructure itself has a cost that must be accounted for, and the savings are quality-conditioned: a measured 8-week pilot realized 58% cost reduction at a 91% response-acceptance rate, so the acceptance threshold you set — and the residual quality cost it implies — lands on you, not the router.

The 24-month horizon — six dated bets

The forecast notebook has now asserted its dated binary bets (evidence cut 2026-08-20, resolution by August 2028):

Bet Claim P Confidence
FC1 Model efficiency (not hardware) drives >50% of further price decline 0.70 medium
FC2 Hyperscaler custom silicon reaches 25%+ of inference workload by mid-2028 0.45 low-medium
FC3 Jevons paradox holds — a 50% unit-cost cut raises volume more than 50% 0.65 medium
FC4 Subsidies contract 50%+, raising effective cost 1.5–3x for subsidized users 0.50 low-medium
FC5 Open-weight models reach quality parity on most workloads within 24 months 0.55 medium-low
FC6 Edge becomes cost-advantageous for a materially larger workload set 0.60 medium-low

The structural read: the 10x/year compression era is ending (fixed-quality decline is decelerating toward 1.5–5x/year and bifurcating — commodity approaching free, frontier reasoning moving up in price), so planning should treat unit cost as a shrinking but non-zero line item while total spend likely still rises (FC3). The least evidenced bets — subsidy contraction and demand elasticity — are the ones that would move budgets most.

The routing layer itself — thin fees, thin moats

The middleware that performs the routing is not where the money is, and may not be where it stays. OpenRouter — the category leader, processing over $100M in annualized inference spend and more than a quadrillion tokens/year by mid-2026 — charges a flat 5.5% on credit purchases with token prices passed through at provider list rates. Its fee is roughly one-tenth the size of the savings its own category claims to deliver. The layer is also structurally commoditizing: open-source gateways (LiteLLM) replicate the unified-API function free forever, 10+ commercial competitors ship the same feature set, and the standardized OpenAI-compatible interface makes switching cost nearly zero. Stripe’s $7.5B acquisition of OpenRouter (August 2026) reads as the category’s sustainability answer: routing survives as payments infrastructure, not as a standalone margin business.

Who absorbs the cost of model fit? Not the router. The 5.5% fee is the smallest line in the fit-cost stack; the real costs of matching models to workloads — evaluation effort to pick the threshold model, retry and quality-mismatch costs when the cheap model fails, re-integration each time a current model is deprecated — are absorbed by the developer. Routing-as-a- service removes the transport problem, not the fit problem.

If you’re evaluating fine-tuning vs retrieval

Three verified findings from the fine-tuning notebook, all pointing the same direction. PEFT (LoRA/QLoRA) makes the compute almost free — a 70B QLoRA run costs tens of dollars on a single H100 vs 8×H100 hours for full fine-tuning. Compute is nonetheless only 2–3% of a real programme: dataset curation, evaluation, and MLOps dominate, and two conditions widen that gap — fine-tuning causes unpredictable safety drift even on clean data (budget recurring safety evals per refit), and every base-model deprecation forces a refit, making compute a subscription rather than a purchase. On the build decision itself: RAG and fine-tuning are substitutes for knowledge-heavy tasks (RAG is updateable and cheaper to iterate) but complements for behavior-heavy tasks — fine-tune the behavior, retrieve the knowledge.

Evidence boundaries

Update log

Read or download the Markdown source