Two myths to unlearn
Myth one: self-hosting is always cheaper. False. For most workloads under 100M tokens/month, API pricing plus batch discounts beats any self-hosted setup once you include ops time.
Myth two: self-hosting is always harder. Less false than it used to be. vLLM + a managed GPU host like Modal or RunPod removes 90% of the pain that made it a non-starter three years ago.
The unit economics
An H100 rented at ~$2.50/hr runs Llama 3.3 70B at 4-bit quantization at roughly 1,500-3,000 output tokens/second aggregate under load. That's about $0.30-0.60 per million output tokens at 80% utilization.
For comparison: Llama 3.3 via Together API costs ~$0.60/M output. Groq is $0.79. DeepInfra is $0.50. So at high utilization, self-hosting saves 20-40% — but only if you can keep the GPU busy.
Break-even is roughly 40-60M output tokens/month per H100. Below that, API providers win on cost and operational simplicity.
When self-hosting wins
High-throughput batch jobs where you can pack the GPU: nightly summarization, classification pipelines, backfills. Self-hosting shines here.
Data-residency requirements. If your compliance says 'no data leaves the VPC', self-hosting is often the fastest path to a compliant deployment.
Fine-tuned models. If you've LoRA'd Llama for your domain, hosting it yourself (or via a service like Fireworks) is usually cheaper than a bespoke fine-tune contract.
Latency-sensitive workloads where the API round-trip dominates. On-network self-hosting shaves 50-150ms.
The 2026 stack
vLLM for serving. It's the de facto standard: continuous batching, paged attention, tensor parallelism, all working out of the box on the current H100/H200/MI300 lineup.
For deployment: Modal or SkyPilot for on-demand; RunPod or Lambda for spot; Kubernetes + KServe if you're already committed to k8s.
For quantization: AWQ or GPTQ 4-bit reduces VRAM by 4x with typically 1-2% quality loss on real tasks. Skip 8-bit — it's an uneasy middle ground.
Operational realities
GPU spot instances get preempted. Design your serving layer to survive it — health checks, warm pools, and a fallback to an API provider for burst capacity.
Monitoring: track P50 and P99 latency separately, tokens-per-second aggregate, and GPU memory. Prometheus + Grafana is fine.
Model updates: new Llama and DeepSeek weights ship every 3-6 months. Bake a re-benchmark into your quarterly cadence — the model that made sense in Q1 may not in Q3.