Practical guide
GPU vs API Break-Even: Calculate the Required Throughput
Find the monthly request volume needed to beat API pricing using GPU hourly cost, measured throughput, utilization, and ops cost.
By StackCalcs · Updated · Sources checked
Use one comparable workload
Define a workload before comparing prices: for example, one summary with 1,000 output tokens at a fixed quality and latency target. Benchmark the sustained output tokens per second the proposed GPU reaches on that job with realistic batching, then price the same job on the API model you would otherwise use.
Suppose the GPU costs $2.40 per hour, sustains 400 output tokens per second, and averages 50% of that rate across a 730-hour month. One GPU then completes 400 × 3,600 × 730 × 0.5 ÷ 1,000 = 525,600 summaries a month for 2.40 × 730 = $1,752. If the API works out to $0.018 per summary, self-hosting breaks even at 1,752 ÷ 0.018 ≈ 97,334 summaries a month; adding $2,000 a month of operations work moves that to about 208,445. The GPU vs API Break-even Calculator does this sizing for you, prices the API side from the selected model’s official token rates, and shows the result at other volumes and utilization levels.
Build a fully loaded GPU rate
For rented hardware, include the instance or host required to use the accelerator, attached storage, and unavoidable networking. For owned hardware, convert purchase cost over its useful service hours and add power, hosting, and support. Use productive hours in the denominator; a GPU reserved but idle still costs money.
Benchmark end-to-end throughput with realistic batching, prompt lengths, output lengths, and concurrency. Peak tokens per second from a model card is not completed-workload throughput.
The calculator takes GPU price and throughput as your inputs because GPU offers and benchmarks are not interchangeable catalog items. Engineering and monitoring go in the monthly ops input. Redundancy, peak headroom, model quality, latency, networking, and contract terms are excluded unless you add them to those inputs.
FAQ
What does “API always cheaper” mean?
A fully used GPU would avoid less API spend per month than it costs, so no volume makes self-hosting cheaper with the entered price, throughput, and utilization.
Why can self-hosting lose again above the break-even volume?
GPUs are added in whole units. Just after traffic outgrows one GPU, the second one is mostly idle, so cost jumps before the API bill catches up. The volume table shows each step.
Official sources
- https://aws.amazon.com/ec2/instance-types/accelerated-computing/
- https://cloud.google.com/compute/gpus-pricing
- https://developers.openai.com/api/docs/pricing