“Confidence Scores” — The Irony
My coding agent loves appending every LLM call response with a ‘confidence score’ from the LLM. Greater than 0.7 and it’s the truth, to be accepted; less than 0.3 and you reject; somewhere in the middle is for human review. But it’s not a confidence score — there is no sampling, no frequency distribution, no hidden bootstrapping, maybe not even a methodology on which to select a score. LLM calls are independent and uncalibrated unless trained. It is pure comfort blanket! Ten minutes of critical thinking makes it quite apparent the scores are largely meaningless. And before you know it you have 70% of LLM calls needing human review.
Fortunately there is a better, hidden way. A fair bit of research shows LLMs are more likely to give a correct decision when forced into a binary accept/reject choice coupled with a justification. Ask an LLM to rate a loan application 1–10 and you get a vague "7" that still needs a human. Force it to"approve or decline, with reasons," and the act of justifying makes it weigh the actual evidence. The additional reasoning required for the justification results in the correct response being given more frequently than under the low/medium/high or sliding-score approach. And a bonus side effect — much less human review.
The explanations themselves will also give you clues for patterns in the decision making that can push logic back down the ladder. For example, when determining that Chemical Brothers Inc is the same company as Chemical Bros Inc, if the LLM reported its decision was based on them sharing the same CEO in press releases published at the same time, you could codify this deterministically and get the decision for free next time.
Consult another expert
I had a routine that would geocode anything vaguely geographic, from cities, streets and sea names to parks. It had the usual ladder system from cheap to expensive solutions. But when the output is visible on a map there is a special kind of ‘sore thumb’ about putting a city in an ocean, or confusing a beach in Spain with Brazil, especially if you are drawing routes. In this scenario you can use different methods to calculate the same coordinates and check the distance between them is within tolerances.
A more expensive version, which is useful for high-value long-form analysis of complex texts — regulatory documents, for example — is to use a technique I call LLM Tennis. You task your first frontier model with producing an analysis of a text based on some instructions. You then pass this analysis, along with the original text and instructions, to a second frontier model and ask for a performance assessment of the first model — i.e. extract the assertions made and verify whether they are correct, incorrect or missing, and return a structured report. This report is then sent back to Frontier Model 1, which independently verifies each finding and accepts or rejects them accordingly. The ball goes back and forth, each pass narrowing the disagreement, until the dissonance drops below a threshold and the two converge near an acceptable truth. You should now have an acceptable analysis. You have used structured disagreement to grind off the errors. It is slow and it costs, so it is reserved for the small number of high value cases that warrant it.
Freeze decisions with a hash
Many LLM calls can be made, and budgets burned, by re-deciding things that have not changed.
The typical pattern when saving outputs or inputs is to cache with a time-to-live (expiry). But a TTL is a guess: too short and you re-pay constantly; too long and you serve a stale decision after the underlying truth has moved. You are choosing between waste and wrongness.
A separate approach is to key the decision to its inputs, not to a clock. Hash the content a decision was made from — the normalised text of a source page, say — and store the decision against that hash. When you come back, re-hash the current content. If the hash is unchanged, the decision still stands: skip the call entirely. If it changed, the source actually moved, and now — and only now — you re-evaluate. It is cheaper, faster and less wasteful than a cache-plus-TTL, and it carries no risk of staleness, because it is the change itself that triggers the work. It pairs naturally with provenance: store the hash next to the source, and a re-crawl that comes back identical costs nothing downstream.
Warm the cache, then fan out
This was quite a learning curve. If you are not caching LLM instruction prompts then you are wasting a lot of money — up to 90% or more, depending on your use case. Prompt caching is the cheapest speed-up available, and the easiest to waste.
For example, if you are processing text more than once, then that text should be a big shared block of context that every call carries. The first call writes that cache, and every call after it reads from the cache for a fraction of the price of the uncached version.
That’s the theory. But reality is different. If you look at the prices you are paying per call you will see, especially with cheaper models, that the cache takes time to warm and there is no guarantee you will hit it on your second pass. It’s as if it is distributed and not universally available. I had most success when fanning out LLM calls pyramidically, in batches — so 1, 2, 4, 6 and so on. By the time you hit the 6-at-a-time you have a 95% cache rate on all of them, versus 30% on the 2-at-a-time. This is far more economic than sending out in batches of 10 and hoping the cache works, because it won’t for the first few batches.
Everyone can be a DIY expert
I see the latest advice from Anthropic is to ‘be more ambitious’. If you think it, you can do it!
I was spending around $500 per month on Tavily for agentic search. It was fast, it returned page content with its own scoring, and sometimes I reranked its results. As well as the cost, it had weaknesses I didn’t like — it paraphrased through an LLM, which meant I couldn’t be sure what I was getting was the ‘truth’, and it could not filter reliably by date, which I needed. It was fast, but I did not need super-fast. I needed fast enough, accurate, with date capabilities, and as cheap as possible.
So I built my own. Amazingly it only took me three hours, after sweating about it for two months. I created a shared SearchProvider interface, and behind it three agentic search routines all returning the same ScoredSearchResult — title, URL, snippet, score. A synthetic scoring system that boosted what I was actually looking for, derived from the semantic disambiguation I already had, rather than what Tavily guessed I wanted. Real date filtering, because I had access to the dates. It ran three times slower but it was more accurate — no paraphrasing, so more reliable — and it cost 90% less than Tavily.
The reason it only took three hours is that the parts I needed I already had: the reranker, the source-hierarchy scorer and the fetch ladder were all written, thin and reusable. Lesson — things compound when you build them DRY and thin. Define what actually matters and the agent will build the rest for you.
Conclusion
You can do it, with thought. Cheap, fast and honest is possible. None of it is particularly clever. A bit of investigation and ambition is the difference between a system that scales affordably and one that hits your credit card daily.