2 × H100 SXM on Vast.ai. Tokens/s is the InferenceX measurement.
Services
Serving 1B input and 200M output tokens a month with Llama 3.3 70B Instruct in batch costs about $299.50 on 4 rented RTX 4090s (an estimate) or $491.19 on 2 H100s at InferenceX’s measured rate, against $164 on the DeepInfra API. The API is cheaper at this volume.
Pick a model and enter your monthly tokens below. The calculator sizes VRAM and GPU count, applies measured or scaled throughput, and compares the bill with hosted APIs.
$299.50
this month · $0.25 per 1M tokens · or $5,905 if the GPUs stay on
AA Intelligence: not scoredLMArena
Cheapest setup for this model that fits in 85% of VRAM and can reach 50 tok/s per user.
Quality is the Artificial Analysis Intelligence Index (higher is better), linked to each model's page. AA's current index version does not score older open models such as Llama 3.3 70B or gpt-oss, so those show a dash rather than a guess. LMArena is linked for a second opinion.
2 × H100 SXM on Vast.ai. Tokens/s is the InferenceX measurement.
Free, from this page
One email with this page’s inputs, the recommended model, GPU, and provider, a deployment checklist, and whether self-hosting or the API is the smaller bill.
LLM hosting plan
Llama 3.3 70B Instruct4× NVIDIA GeForce RTX 4090
$299.50 this month
DeepInfra API $164.00 for the same work. GPU rental only.
Llama 3.3 70B in FP8 is about 70.6 GB of weights, so it needs two 80 GB H100s with room for KV cache. InferenceX measured 1.5k total tokens a second per H100 at 50 tokens a second per user. At 1.2B tokens a month that is $491.19 on Vast.ai as a batch job, and $12,874 if the two GPUs stay on all month.
DeepInfra serves the same model for $0.1 / $0.32 per million input and output tokens, $164 for this workload. Hosted open models are priced near GPU cost, so self-hosting pays off mainly when the GPUs stay busy, when you need a fine-tuned or private model, or when data cannot leave your network.
Every GPU setup for Llama 3.3 70B Instruct at 50 tokens a second per user. Weights must fit in 85% of total VRAM. A100 and A10 have no FP8, so FP8 models load in BF16 there.
| GPU | Provider | Precision | Weights | GPUs | Tokens/s per GPU | $/hour | Always-on / month | This workload | Per 1M tokens |
|---|---|---|---|---|---|---|---|---|---|
| NVIDIA GeForce RTX 4090 | Vast.ai RTX 4090 24GB on-demand median | FP8 | 71 GB | 4 | 450 estimate | $1.6178 median | $1,181 | $299.50 | $0.25 |
| NVIDIA H100 80GB SXM | Vast.ai H100 SXM on-demand median | FP8 | 71 GB | 2 | 1.5k measured | $4.4089 median | $3,218 | $491.19 | $0.41 |
| NVIDIA GeForce RTX 4090 | RunPod Secure Cloud RTX 4090 | FP8 | 71 GB | 4 | 450 estimate | $3.56 on-demand | $2,599 | $659.05 | $0.55 |
| NVIDIA A100 80GB SXM | Vast.ai A100 SXM4 80GB on-demand median | BF16 | 141 GB | 4 | 455 estimate | $4.5013 median | $3,286 | $823.92 | $0.69 |
| NVIDIA H100 80GB SXM | Modal Nvidia H100 SXM5 | FP8 | 71 GB | 2 | 1.5k measured | $7.8984 preemptible | $5,766 | $879.95 | $0.73 |
| NVIDIA H100 80GB SXM | RunPod Secure Cloud H100 SXM | FP8 | 71 GB | 2 | 1.5k measured | $7.98 on-demand | $5,825 | $889.04 | $0.74 |
| NVIDIA H100 80GB PCIe | Vast.ai H100 PCIe on-demand median | FP8 | 71 GB | 2 | 893 estimate | $5.0844 median | $3,712 | $948.80 | $0.79 |
| NVIDIA H100 80GB SXM | Lambda 1× H100 SXM | FP8 | 71 GB | 2 | 1.5k measured | $8.58 on-demand | $6,263 | $955.88 | $0.80 |
| NVIDIA GeForce RTX 4090 | RunPod Serverless RTX 4090 worker | FP8 | 71 GB | 4 | 450 estimate | $5.36 serverless | $3,913 | $992.28 | $0.83 |
| NVIDIA H100 80GB PCIe | RunPod Secure Cloud H100 PCIe | FP8 | 71 GB | 2 | 893 estimate | $5.78 on-demand | $4,219 | $1,079 | $0.90 |
| NVIDIA H100 80GB SXM | RunPod Serverless H100 80GB worker | FP8 | 71 GB | 2 | 1.5k estimate | $10.78 serverless | $7,869 | $1,201 | $1.00 |
| NVIDIA H100 80GB PCIe | Lambda 1× H100 PCIe | FP8 | 71 GB | 2 | 893 estimate | $6.58 on-demand | $4,803 | $1,228 | $1.02 |
| NVIDIA A100 80GB SXM | RunPod Secure Cloud A100 SXM 80GB | BF16 | 141 GB | 4 | 455 estimate | $7.16 on-demand | $5,227 | $1,311 | $1.09 |
| NVIDIA A100 80GB PCIe | RunPod Secure Cloud A100 PCIe 80GB | BF16 | 141 GB | 4 | 432 estimate | $7.16 on-demand | $5,227 | $1,381 | $1.15 |
| NVIDIA H100 80GB SXM | AWS p5.4xlarge | FP8 | 71 GB | 2 | 1.5k measured | $13.76 on-demand | $10,045 | $1,533 | $1.28 |
| NVIDIA A100 80GB SXM | Modal Nvidia A100 80GB | BF16 | 141 GB | 4 | 455 estimate | $9.9936 preemptible | $7,295 | $1,829 | $1.52 |
| NVIDIA A100 40GB SXM | Lambda 1× A100 SXM 40GB | BF16 | 141 GB | 8 | 347 estimate | $15.92 on-demand | $11,622 | $1,910 | $1.59 |
| NVIDIA A100 80GB SXM | RunPod Serverless A100 80GB worker | BF16 | 141 GB | 4 | 455 estimate | $10.88 serverless | $7,942 | $1,991 | $1.66 |
| NVIDIA H100 80GB SXM | Google Cloud a3-highgpu-8g | FP8 | 71 GB | 2 (4 copies on 8) | 1.5k measured | $88.49 on-demand | $64,598 | $2,465 | $2.05 |
| NVIDIA A100 80GB SXM | AWS p4de.24xlarge | BF16 | 141 GB | 4 (2 copies on 8) | 455 estimate | $27.45 on-demand | $20,036 | $2,512 | $2.09 |
| NVIDIA A100 80GB SXM | Google Cloud a2-ultragpu-1g | BF16 | 141 GB | 4 | 455 estimate | $20.28 on-demand | $14,801 | $3,711 | $3.09 |
Quality is the Artificial Analysis Intelligence Index (higher is better), linked to each model's page. AA's current index version does not score older open models such as Llama 3.3 70B or gpt-oss, so those show a dash rather than a guess. LMArena is linked for a second opinion.
| Model | AA index | Params | Cheapest setup | Self-host / month | API $ per 1M in / out | API / month |
|---|---|---|---|---|---|---|
| Llama 3.3 70B Instruct Llama 3.3 Community | — | 70.55B | 4 × RTX 4090, FP8 Vast.ai | $299.50 est. | DeepInfra $0.1 / $0.32 | $164.00 |
| Llama 3.1 8B Instruct Llama 3.1 Community | — | 8.03B | 1 × RTX 4090, FP8 Vast.ai | $34.09 est. | DeepInfra $0.02 / $0.04 | $28.00 |
| gpt-oss-120b Apache-2.0 | — | 116.83B, 5.1B active | 4 × RTX 4090, MXFP4 Vast.ai | $49.93 est. | Together $0.15 / $0.6 Groq $0.15 / $0.6 Fireworks $0.15 / $0.6 | $270.00 |
| gpt-oss-20b Apache-2.0 | — | 20.91B, 3.6B active | 1 × RTX 4090, MXFP4 Vast.ai | $35.25 est. | Groq $0.075 / $0.3 | $135.00 |
| Qwen3.8 27B Apache-2.0 | 20.1 | 27.78B | 2 × RTX 4090, FP8 Vast.ai | $117.93 est. | Groq $0.8 / $4 | $1,600 |
| Gemma 3 27B Gemma terms | — | 27.43B | 2 × RTX 4090, FP8 Vast.ai | $116.44 est. | DeepInfra $0.08 / $0.16 | $112.00 |
| Mistral Small 3.2 24B Apache-2.0 | — | 24.01B | 2 × RTX 4090, FP8 Vast.ai | $101.93 est. | DeepInfra $0.075 / $0.2 | $115.00 |
| Qwen3 235B A22B 2507 Apache-2.0 | — | 235.11B, 22B active | 4 × H100 80GB SXM, FP8 Vast.ai | $706.52 est. | DeepInfra $0.09 / $0.55 Together $0.2 / $0.6 | $200.00 |
| DeepSeek V3.2 MIT | — | 685.36B, 37B active | Does not fit | — | DeepInfra $0.26 / $0.38 | $336.00 |
| Claude Sonnet 5.5 Anthropic, closed weights | 54.2 | — | API only | — | Anthropic $2 / $10 | $4,000 |
| Claude Haiku 4.5 Anthropic, closed weights | — | — | API only | — | Anthropic $1 / $5 | $2,000 |
| GPT-6 Luna OpenAI, closed weights | 27.8 | — | API only | — | OpenAI $0.1 / $0.5 | $200.00 |
| gpt-5-mini OpenAI, closed weights | — | — | API only | — | OpenAI $0.25 / $2 | $650.00 |
“Est.” means the throughput was scaled, not measured. A dash in AA index means Artificial Analysis does not score that model on its current index. LMArena ranks many of the same models by human votes.
On-demand hourly price and the always-on month (hourly × 730). Community and non-preemptible rates are shown only where that provider publishes them. Each price links to the provider’s pricing page, checked November 1, 2026. A dash means that provider does not list this GPU.
| GPU | RunPod | Lambda | Modal | Vast.ai | AWS | Google Cloud |
|---|---|---|---|---|---|---|
| L4 | — | Nvidia L4 $0.7992/hr Preemptible $583.42/mo Non-preemptible $2.3976/hr · $1,750/mo checked November 1, 2026 | ||||
| A10 / A10G | — | Nvidia A10 $1.1016/hr Preemptible $804.17/mo Non-preemptible $3.3048/hr · $2,413/mo checked November 1, 2026 | — | |||
| RTX 4090 | — | — | — | — | ||
| A100 | Nvidia A100 80GB $2.4984/hr Preemptible $1,824/mo Non-preemptible $7.4952/hr · $5,471/mo checked November 1, 2026 | |||||
| H100 | Nvidia H100 SXM5 $3.9492/hr Preemptible $2,883/mo Non-preemptible $11.85/hr · $8,649/mo checked November 1, 2026 |
Prices checked November 1, 2026. A month is 730 hours. Batch cost = (input + output tokens) ÷ (tokens/s per GPU × GPUs × 3,600) × the hourly price. Always-on cost keeps enough replicas on all month for the average load and for the peak users at the chosen speed, assuming output is one ninth of the tokens, as in the 8k-in / 1k-out benchmark. The bill is GPU rental only.
Throughput comes from SemiAnalysis InferenceX: total tokens a second per H100 while every user still gets the chosen output speed, on 8k-token prompts with 1k-token answers. Llama 3.3 70B (vLLM, FP8) and gpt-oss-120b (vLLM, MXFP4) are measured. Other dense models scale the Llama figure by 70.55B ÷ their parameter count. Other mixture-of-experts models scale the gpt-oss figure by 5.1B ÷ their active parameters, times 0.5 when their weights are FP8 instead of 4-bit. Other GPUs scale by memory bandwidth against the H100 SXM's 3.35 TB/s. A10 and A100 have no FP8, so FP8 models load in BF16 there, which doubles the memory and halves the estimate. Multi-GPU setups multiply by the GPU count, which assumes tensor parallelism scales linearly. Consumer cards (RTX 4090) and L4s talk over PCIe without NVLink, so their multi-GPU estimates are the least reliable. A setup is dropped when one user could not reach the chosen speed even in theory (bandwidth × GPUs ÷ active weight bytes).
Related tools: the self-hosted OCR cost calculator, the embedding cost calculator, and the speech-to-text cost calculator. Related reading: LLM API pricing comparison, LLM cost optimization, and AI agent development.
Usually not for a steady, modest workload. Hosted open models are priced close to the GPU cost. DeepInfra lists Llama 3.3 70B at $0.10 in and $0.32 out per million tokens. Self-hosting tends to pay off when GPUs stay busy, when you need a fine-tuned model, or when data cannot leave your network. The calculator prices both side by side.
The FP8 weights are about 70.6 GB. This tool lets weights use 85% of VRAM and keeps the rest for KV cache, so it puts Llama 3.3 70B on two 80 GB H100s, or four 24 GB cards. A100s have no FP8, so in BF16 it needs about 141 GB, which means four A100 80GB.
SemiAnalysis InferenceX measured Llama 3.3 70B with vLLM in FP8 at 1,496 total tokens a second per H100 while each user still got 50 output tokens a second, on 8k-in, 1k-out requests. gpt-oss-120b reached 8,973 per H100 at the same per-user speed. Faster per-user targets serve fewer tokens in total.
Yes. FP8 halves the weight memory compared with BF16, so a model fits on fewer GPUs, and decoding reads half as many bytes. H100, L4, and RTX 4090 run FP8. A10 and A100 do not, so this tool loads FP8 models in BF16 there. gpt-oss ships in 4-bit MXFP4, which is why gpt-oss-120b fits on one 80 GB GPU.
Artificial Analysis's current Intelligence Index version has no score for them, and their model pages read Unknown. The page shows a dash instead of an older or estimated number. LMArena is linked as another view.
Not on any single machine in this tool. The FP8 weights are about 685 GB, more than 85% of eight 80 GB GPUs, so the calculator shows it as API-only. It needs H200, B200, or multi-node serving.