Cost per useful output
Ignore per-token price. Measure cost per successfully validated output on your workflow. A cheaper model that fails validation twice as often is not cheaper.
Latency
P50 latency is a marketing number. Measure P95 on your actual prompts and payloads.
Structured output
JSON mode, function calling, and strict schema support vary. If your workflow depends on structured output, pick a model with first-class schema enforcement.
Long context
Long-context models unlock research and summarization patterns. Test recall on the last third of your context — models often forget it.
Safety and policy
Different vendors refuse different categories of prompts. If your workflow touches security research or health, this is not academic.
Self-hosted options
Llama, Qwen, and Mistral variants now cover most workflow needs at a fraction of the cost — if you have the ops to run them.