Compare the same work first
A reasoning API and a small local model do not automatically provide an equivalent service. Choose representative tasks, an acceptable answer quality and a response-time target. Compare the systems that pass those requirements by cost per successful task.
Include retries, tool calls and rejected responses. A lower price per token can still mean a higher cost per completed case if the model needs more attempts. Keep privacy, availability and operating constraints in the decision as well: these can justify an architecture even when it costs more.
Reconstruct the API bill from the traffic mix
Calculate uncached input, cached input and output separately using the applicable rates, then add other billable units. Batch discounts or commitments count only when the workload meets their conditions. A blended price per million tokens is convenient for a model, but must be recalculated when the input/output mix changes.
An application that reads long documents and returns short answers has a different effective price from an agent that generates extensive output. Use a dated offer or your contract rather than treating a market average as a universal rate.
Price the full server and the hours billed
Start with the instance you would order, including its GPU count, CPU, RAM, storage, networking and options. Add operating work attributable to the service. Avoid adding power and cooling twice when they are already included in a rental. Owned hardware needs depreciation, power and maintenance accounted for separately.
The France and Europe GPU cloud guide explains why a single-GPU instance price cannot be compared directly with an eight-GPU node quote. Check the selected zone and actual stock before treating a public listing as deployable capacity.
A financial break-even can exceed GPU capacity
Use a deliberately hypothetical USD scenario: $4 per hour for the complete server and an API averaging $3 per million tokens. Neither figure is a provider quote. Continuous operation costs 4 × 720 = $2,880 per month. API cost reaches that amount at 960 million tokens per month, or 32 million per day.
Now suppose a workload test had produced 500 useful tokens per second at the required latency, and demand used half that capacity over time. Production capacity would be 500 × 0.5 × 86,400 = 21.6 million tokens per day. One server could not reach the financial break-even volume.
| Item | Assumption or result |
|---|---|
| Full server cost | $4/hour, or $2,880/month |
| Hypothetical blended API price | $3/million tokens |
| One-server financial break-even | 32 million tokens/day |
| Assumed one-server capacity | 21.6 million tokens/day |
| Two continuously rented servers | $5,760/month; assumed capacity of 43.2 million/day |
At 32 million tokens per day, two servers cost $5,760 a month while the hypothetical API costs $2,880. That does not establish an industry-wide winner. It shows why capacity must be checked alongside financial break-even. Redundancy and traffic spikes can require additional provisioned capacity.
Measure the workload that supplies the inputs
Keep the model, precision, context distribution, output lengths and quality requirement explicit. Measure successful throughput, errors, time to first token and response latency as load rises. The vLLM, llama.cpp and TensorRT-LLM comparison describes that serving test.
A throughput figure from one request mix cannot price a different one. Nor is the GPU-busy percentage a direct measure of the fraction of useful service capacity sold. If a model spans several GPUs, price the complete replica and use its measured capacity rather than dividing a single-card estimate.
For sensitivity analysis, the open inference cost calculator extends the calculation to owned hardware. Replace its assumptions with your own measured mix and dated prices before making a purchasing decision.
Check memory before choosing local hardware
Weights alone can rule out a configuration. Exactly four bits for 70 billion dense parameters takes 35 decimal GB, before metadata and runtime allocations. A 32 GB GPU cannot hold that complete raw payload. Lower precision or offloading changes the trade-offs rather than making the memory requirement disappear.
After weights, budget KV cache for the intended context and concurrency. Then include buffer space and test latency on the complete service. API, cloud and owned hardware can each be appropriate; the decision follows from these inputs rather than a fixed daily token threshold.
Sources and method
English adaptation published November 3, 2026. All dollar amounts and the 500 tokens/s figure in the worked example are explicit assumptions, not market quotes, exchange-rate conversions or measurements. Equations use a constant workload mix and a 30-day month. Public instance structure and GPU counts were checked against Scaleway’s GPU price grid and OVHcloud’s Public Cloud grid. Capacity measurement: vLLM serving benchmark documentation. Open calculator: llm-inference-cost source.
