Local LLM vs. API Cost: When Does Self-Hosting Actually Win?

Pricing checked: August 10, 2026. OpenAI's live standard short-context rates were Sol at $5 input, $0.50 cached input, $6.25 cache write, and $30 output; Terra at $2, $0.20, $2.50, and $12; and Luna at $0.20, $0.02, $0.25, and $1.20 per million tokens. Batch and Flex were half standard rates. Fast and long-context requests use different multipliers. Those mechanics are large enough to change the infrastructure decision.

Run your workload in the API vs. Local AI Calculator. Enter a capacity you have derived from a representative benchmark; do not accept its illustrative default as evidence. This article shows how to produce that input and where the calculator stops.

A local LLM and an API are not interchangeable merely because both accept text and return tokens. Compare endpoints only after they pass the same task set and operating threshold. A cheaper local answer that creates more review, escalations, unsafe actions, malformed output, or missed deadlines is not the same unit of work.

The local example uses gpt-oss-120b because it fits the chosen device and has a disclosed benchmark. OpenAI described it against the then-current o4-mini on selected 2025 benchmarks. That does not establish parity with 2026 Sol or Terra, and this article does not claim it.

This analysis preserves that canonical structure instead of inventing a competing registered model. The calculations are transparent Software Second inference, not AEM output. Read the Software Second methodology for the broader rule: compare credible alternatives against the same outcome and include the system that operates, not only the component with a posted price.

Do not ship from this arithmetic. NVIDIA's source workload has only 128 output tokens, while this scenario assumes 400. Production TTFT and end-to-end latency include queueing, batching, networking, prefill, and decoding; per-user throughput can fall as concurrency rises even while aggregate throughput initially rises. Benchmark the exact model, quantization, server, prompt distribution, output distribution, concurrency, thermal conditions, and p95 SLO.

The national power envelope is modest: at the EIA's preliminary 13.79 cents/kWh commercial average, 730 hours at the 140W SoC TDP is about $14.09 and at the 240W PSU rating about $24.16. Those are not measured wall-power results, and cooling, UPS, space and network remain. More importantly, power is not the cost line that determines this example; engineering and reliability are.

The production crossing is 12,081 Terra requests/day, more than twice active capacity. Calling that a break-even would be false. The next step is not to ignore capacity—it is to size another active fleet, add its cost, and calculate a new step. This is why the common claim that self-hosting always crosses eventually is incomplete.

The cache row uses 720 uncached input tokens at $2/M, 1,080 cached-read tokens at $0.20/M, and 400 output tokens at $12/M. It omits the $2.50/M cache-write cost until the reuse schedule is known. Batch and Flex have different latency and availability characteristics, so the discount applies only if the work can accept that service.

The output-length rows recalculate local capacity instead of carrying forward the 400-output limit. At 50 output tokens, 1,800/1,725.47 + 50/55.37 is 1.95 serial seconds per request, yielding about 24,417 planned requests/day at the same 55% factor. At 1,000 output tokens, service time rises to 19.10 seconds and planned capacity falls to about 2,488/day. The 2,718/day price crossing is therefore not capacity-valid.

This table answers how sensitive the arithmetic is to price, not which model should win. Providers tokenize differently—Anthropic warns that its newer tokenizer can produce approximately 30% more tokens for the same text, depending on workload—and task quality may differ far more than price. Export actual billed token counts and evaluate each endpoint independently.

Lambda listed a GH200 96 GB at $2.29 per GPU-hour and an H100 PCIe 80 GB at $3.29 on August 10. At 730 continuously rented hours, compute alone is $1,671.70 and $2,401.70 per month. Per-minute billing makes rental useful for pilots and temporary demand; always-on rental needs the same operating-cost discipline as owned hardware.

Privacy is not a binary local-good/API-bad claim. OpenAI says API inputs and outputs are not used for training by default; standard abuse-monitoring logs may be retained up to 30 days, and eligible customers can apply for Zero Data Retention or Modified Abuse Monitoring subject to endpoint restrictions. Local operation can provide stronger direct control, but it also transfers security, access, audit, patching and incident responsibility to the operator.

A control requirement should be stated as a requirement, measured, and priced. It is legitimate to choose a more expensive local service for offline availability or jurisdiction. It is not legitimate to claim that the local system is cheaper by leaving the cost of achieving that control out of the ledger.

1. Export at least a representative month of billed requests, uncached input, cached reads and writes, output, retries, tool fees and fallback calls.
2. Build a task evaluation and measure acceptance, human review, escalation, safety, structured-output success and failure consequence for every candidate endpoint.
3. Benchmark prompt and output distributions under expected concurrency; set TTFT, p95 end-to-end latency, uptime and recovery targets.
4. Price setup, security, monitoring, incidents, maintenance, upgrades, migration, power, facility, storage and required redundancy with named owners.
5. Derive planned capacity from the benchmark and an explicit reserve; reject any crossing above that capacity and resize the fleet step by step.
6. Test cache, batch, output length, cheaper adequate models, traffic variance and engineering hours before approving hardware.
7. Pilot before purchase, and preserve an API fallback if the local quality or capacity margin is narrow.

If the infrastructure choice changes human capacity or downstream business value, carry the result into How to Calculate AI ROI and the AI ROI Calculator. Infrastructure savings are an input; they are not the whole business case.

These are generalized assumptions. Your traffic, quality threshold, engineering burden and control requirements will be different. Run the API vs. Local AI Calculator, review the methodology, then run the analysis against your company before committing capital or platform ownership.