Running AI in production across FastRead.io, Myautobio, Clintwin/Medsim AI, and Formulai meant the model bill was the first cost line that scaled embarrassingly faster than revenue. The 38% reduction we eventually landed didn't come from one clever trick. It came from killing three assumptions the team had been making by default.
Assumption one: the frontier model is the safe default
Defaulting every call to the most capable model feels like the risk-free choice. Nobody gets blamed for "we used the best one." It's also the most expensive possible default, applied uniformly to tasks that range from "summarize this in one sentence" to "reason through a multi-step medical scenario." We benchmarked each real task in each product against smaller and mid-tier models specifically, not against a generic leaderboard, and found a large share of calls (classification, extraction, short rewrites) held quality within a threshold we'd pre-agreed on with a much cheaper model.
Assumption two: routing should be static
Once we knew which task types tolerated a cheaper model, the next mistake would have been hardcoding that mapping forever. Instead we built routing that considers task type and a confidence signal, falling back to a stronger model when the cheaper one's output looks uncertain, rather than a flat rule that would quietly degrade quality the day user behavior shifted even slightly from what we'd benchmarked.
Cheaper isn't the goal. Cheaper-without-the-user-noticing is the goal, and the gap between those two is entirely in how carefully you route, not in which model you pick.
Assumption three: every call needs a fresh answer
A meaningful share of requests across the four products were structurally repetitive (the same classification on near-identical inputs, the same lookup phrased slightly differently). Caching at the right layer, not just literal string matches but semantically similar requests, removed a chunk of calls from the bill entirely without touching quality, because nothing was ever regenerated differently the second time. It just wasn't generated at all.
Running costs dropped 38% across the portfolio, and output quality (measured against the same evaluation set we used before the change) held steady or improved, because the routing fallback caught the cases a cheaper model would have gotten wrong.
What I'd check first on any new AI product
- Which tasks are you defaulting to your best model for, out of caution rather than evidence?
- Does your routing have a real fallback signal, or a fixed rule that will quietly rot as usage shifts?
- How much of your traffic is structurally repetitive, and is anything actually catching that before it hits a paid call?
None of this required a model upgrade or a vendor switch. It required measuring what each task actually needed, instead of defaulting every task to what the hardest task needed.