1. Route cheap-first
Every request goes to the cheap model first (GPT-4o-mini, Gemini Flash-Lite, Claude Haiku) with a self-critique step. Escalate only what fails.
Typical savings: 60-80% at aggregate. Requires an eval to verify quality parity on your workload — the win is real but not free.
2. Prompt cache aggressively
Anything static — system prompts, tool schemas, retrieved context repeated in-session — should be cached. Anthropic pays 10x for cached input; OpenAI pays 2x.
Restructure prompts to put static content at the top. This alone recovers 30-50% on any tool-heavy workflow.
3. Batch anything async
OpenAI and Anthropic both offer 50% discounts on batch APIs with a 24-hour SLA. Every non-interactive job — nightly summaries, backfills, evals, offline scoring — should live here.
Halving the cost on 30% of your traffic is a 15% total bill cut. Almost free work.
4. Distill for hot paths
For high-volume, narrow tasks (classification, extraction), fine-tune a small open-weights model on outputs from your frontier model. Deploy the small model at 1/10th the cost.
Modal, Fireworks, Together, and Baseten all support this end-to-end. Break-even is usually ~5M requests/month on the task in question.
5. Trim context ruthlessly
The cheapest token is the one you don't send. Audit long prompts monthly — most system prompts accumulate cruft that adds nothing.
For RAG: rerank down to 3-5 chunks instead of stuffing 15. Answer quality typically improves alongside cost.
6. Stream + early-stop
Stream every response. For workflows that only need the first tool call or a short verdict, stop generation as soon as you have what you need.
Combined with structured outputs, this can cut output token spend by 40% on decision workflows.
7. Self-host bursty batch
For predictable batch workloads over ~50M tokens/month, a single H100 on Modal or RunPod running Llama or DeepSeek beats API prices by 3-5x.
Only worth it if you can keep the GPU fed. Under-utilized self-hosting is a common way to spend more than you started with.
8. Governance from day one
Set per-user and per-workflow spend limits. Alert on any workflow doubling in cost week-over-week. Attribute cost to teams so incentive stays aligned.
The cheapest optimization is the one that stops a runaway loop before it hits the invoice. Every serious deployment has learned this the expensive way.