How We Cut AI Infrastructure Costs by 40% in One Quarter

Last quarter our own AI bill was climbing faster than our usage, which is the kind of curve that gets a finance team’s attention. We were spending a little under fifteen thousand dollars a month across a handful of providers, and the line kept bending up. By the end of the quarter we had cut that spend by roughly forty percent while serving more traffic, not less. We did it with UniCraft — the same routing layer we ship — pointed at our own workloads. Here is exactly what moved the number.

Where AI spend actually goes

The first thing we learned is that most teams are wrong about where their money goes, and we were no exception. The intuition is that cost scales with how clever your prompts are. It does not. Cost scales with three things almost nobody measures: how often you call a premium model for a task a cheap one could have handled, how often you pay full price for an answer you already computed, and how much you overpay because you are locked to one provider’s list price.

When we broke down our invoices, the split was sobering. A large share of our calls were trivial — classification, extraction, short rewrites, routing decisions — and a meaningful fraction of those were going to a frontier model out of pure habit. We were buying a sports car to drive to the mailbox, thousands of times a day.

Intelligent model routing

The biggest single lever was routing. UniCraft exposes one OpenAI-compatible endpoint and decides, per request, which model should actually serve it. Easy calls — the classification and extraction work — route to small, fast, cheap models. Hard calls — multi-step reasoning, long-context synthesis, anything where a wrong answer is expensive — route to premium models. The application code does not change; it asks for an answer and the router picks the cheapest model that will get it right.

The discipline that makes this work is measuring quality on the routed tier, not assuming it. We ran shadow evals so that when we downgraded a class of requests, we could prove the answers were still good before we trusted them in production. The result: the majority of our volume moved to models that cost a fraction of what we had been paying, with no measurable drop in task success.

The cheapest token is the one you never spend because a small model already nailed the answer.

Semantic caching

The second lever was caching, but not the naive kind. Exact-match caching catches almost nothing in a real workload because no two user requests are byte-identical. Semantic caching catches the requests that mean the same thing — “what’s your refund window?” and “how long do I have to return this?” hit the same cached answer. Across our traffic, semantic caching deduplicated about thirty-one percent of calls. Those requests returned in milliseconds and cost us nothing, which improved latency and the bill in the same stroke.

Provider arbitrage and failover

The third lever was treating providers as a market instead of a marriage. The same open-weight model is priced differently across the dozen-plus providers UniCraft routes to, and prices move. We let the router arbitrage — send each request to whichever provider offered the best price and latency for the model we wanted at that moment. The same mechanism gives us automatic failover: when a provider degrades or rate-limits us, traffic shifts to a healthy one without a page going out at 3 a.m. Resilience and savings turned out to be the same feature.

Measure cost per resolved task, not per token

None of this is legible if you watch the wrong metric. Cost per token can fall while your real costs rise, because a cheaper model that fails sends the user back for a retry, and now you have paid twice plus a frustrated customer. We switched to cost per resolved task — the all-in cost of getting a user a correct, final answer, retries included. That number is the only one that maps to the business, and optimizing it is what kept us honest about when a downgrade was actually a savings versus a hidden tax.

The breakdown

Here is roughly where the ~40% came from, against our starting ~$15k/month:

• Model routing: ~22 points. Moving trivial calls off frontier models was the dominant win.
• Semantic caching: ~11 points. About 31% of calls deduplicated to a cached answer.
• Provider arbitrage: ~7 points. Same models, better prices, shifted continuously.

The points overlap a little in practice, but the order of magnitude held: a workload that was costing us close to $15k/month settled under $9k/month within the quarter, serving more requests than when we started. The work was not exotic. It was measuring honestly, routing deliberately, caching by meaning, and refusing to be loyal to a single price list.