Most enterprises did not plan their AI spend. It grew. A coding assistant here, a chat licence there, a retrieval pipeline in one department and an agent pilot in another. Each line item looks reasonable on its own. Together they show up at the quarterly review, and few people can say what the money bought.
The fix is rarely "use less AI". It is to treat model choice as an architecture decision: every task goes to the smallest model that does the job well, across whichever providers you use, and every call can be traced to a use case.
Why one model for everything gets expensive
The most common pattern we see is a single frontier model behind every request. Summaries, classifications, simple extractions and hard multi-step reasoning all hit the same, most expensive endpoint. Nobody chose this on purpose. It happens because there is no shared answer to the question of which model is good enough for which job.
Frontier models are the right choice for a small share of work. For the rest, you pay for capability you do not use.
Three task tiers
A working model policy sorts tasks into three tiers. We describe them without model names on purpose: the names and prices change every few months, the tiers do not.
Typical tasks Model class Light Classification, routing, short summaries, simple extraction, FAQ answers on structured data Small, fast models from any major provider or a small open-weights model Medium Code explanation, documentation drafts, structured extraction from long documents Mid-size general models Complex Multi-step reasoning, agent planning, code for critical systems, high-stakes decision support Current frontier models
The policy is only useful if it is tested. Keep a small evaluation set per use case and rerun it when you switch a model. Then a downgrade to a cheaper tier is a measured decision, not a guess.
One gateway in front of all providers
When every team picks its own provider, you get many API keys, no shared cost view and no central control. A gateway fixes that: applications call one endpoint, and the gateway routes to the provider and model the policy says.
OpenRouter is one example of such a gateway. Its documentation describes automatic fallback: if the first model in a list returns an error, for example a rate limit or downtime, the next one takes over. Whether you use a hosted gateway or run your own, the point is the same. Routing rules, spend limits and logs live in one place, and you can swap a model without touching every application.
For agentic AI this layer matters even more. An agent may make dozens of model calls for one task. If those calls go through one governed path, you can see them, cap them and attribute them. That is the basis for what we build at Northstar: every agent action is approved and logged, whatever model sits behind it.
Self-hosting where the load is steady
Some workloads belong on your own infrastructure: data that must not leave your network, strict latency needs, or a high and stable volume on one model. vLLM is a widely used open-source engine for serving open-weights models in production. Its documentation lists the features that matter for cost: PagedAttention for efficient memory use, continuous batching of incoming requests, quantization down to INT4, tensor and pipeline parallelism, and an OpenAI-compatible API server.
Self-hosting is not free. A GPU instance costs money when no requests come in, and someone has to run it. A practical sequence:
- Start with external APIs through the gateway for variable or exploratory workloads.
- Move a workload to self-hosting once its volume and model are stable and you have measured the break-even for your own case.
- Run hybrid where it fits: own models for the baseline, external APIs for peaks and for tasks that need a frontier model.
Because the self-hosted endpoint speaks the same API, the gateway can route to it like any other provider.
Prompt caching and context hygiene
Long system prompts, tool definitions and reference documents are often sent with every call. Prompt caching lets the provider reuse a stable prefix. Anthropic states that caching can cut costs by up to 90 % and latency by up to 85 % for long prompts (source). Other major providers offer similar mechanisms. The saving depends on how stable your prompt prefix is, so structure prompts with the fixed part first.
The other half is context hygiene. Sending the full conversation history when only the last turns matter, or attaching whole documents when a retrieved passage would do, costs tokens on every call.
Licences are part of the bill
Seat licences are the quiet cost driver. The same person often holds a coding assistant, a chat licence and a second chat licence from another vendor. Seats get assigned and rarely reviewed. A monthly usage check and a clean offboarding step when people leave or switch tools cost little and remove spend that buys nothing.
Tag every call to a use case
Without attribution, every saving is a guess. Tag each model call with the use case, the team and, where possible, the agent run it belongs to. Then a monthly review can answer three questions: which use cases deliver, which cost more than expected, and which should stop. Our post on measuring AI impact describes how to set that up.
Where to start
Take an inventory first: providers, models, licences, and which use cases call what. Then define the three tiers, put a gateway in front, and add tagging. Self-hosting and fine-grained routing come after you have the data to justify them.
Sources
- https://claude.com/resources/articles/prompt-caching
- https://openrouter.ai/docs/guides/routing/model-fallbacks
- https://docs.vllm.ai/en/latest/
Related: AI Services – Agentic AI at enterprise scale.