Penny For Your Thoughts - the Cost of Reasoning in Your Token Budget

In The Big Token Test I showed that every non-English token carries a language tax: the same content costs 46-72% more tokens in Danish than in English, across every frontier model tested. Working through that data, something else stood out. The bulk of token consumption wasn't in the answers at all. It sat in reasoning: the hidden thinking phase between the prompt and the reply. 

Most AI models being used today are the reasoning kind. What this means in practice is that there is a part that is visible and tangible, the input and the output. But in-between the model does a lot of thinking to ensure a proper answer. I was surprised to find that this part can be 60-80% of your total token spend and sometimes more. It is billed as output tokens, the expensive kind. You never see it, but you trust that this is necessary to ensure you the proper quality. After all, thinking is valuable, right?

Realizing this made me think of the passage from the classic book The Little Prince by Antoine de Saint-Exupéry:

Reasoning is the elephant inside the Boa that we are too busy to see or take interest in until somebody calls it out. That observation became this project: an investigation of what reasoning consists of, how it works, and what good and bad reasoning look like when you read the models over the shoulder. So, in this essay I will act as your little prince and when we are done you should be able to make out the size, shape and anatomy of the specific elephant, that reasoning is in your own token spend.

The research questions I had in the beginning were many.

• What actually is reasoning?
• Is it possible to assess the quality and efficiency of this part of the LLM process?
• What are the significant differences between how foundation models reason across the different model traits: open-closed, large-small, Chinese-US-European?
• Does the language tax apply to reasoning too?
• Knowing that models today use tools, how does that affect token spend? Will a model who can use python to calculate math do less thinking and will that reduce token spend? 

Reasoning is the most heavily researched area in AI right now. Capability scoreboards like Artificial Analysis' answer which model can. I am looking for answers what it costs to let them. With that said, let me stress here, that I am under no illusion that my tiny contribution will cover all the bases here. My aim is merely to show you the elephant and use my data to call out some key insights. If you are curious about the research, I encourage you to look at the end section where I will share a selection of key references.

I will share three conclusions here that surprised me up front. 

1. Large models are expensive not because they think a lot; on the contrary, they think remarkably efficiently. You pay a premium per token for a kind of insurance: near-certainty of a correct answer, even when the task is far beneath the model. 
2. The newest frontier capability turned out to be knowing when not to think, rather than thinking more or better. 
3. The reasoning effort within the same model varies wildly and can be anywhere between 2 and 10 times as expensive for the same prompt. This is a common trait across all models rather than a quirk of any specific one, which makes predicting token spend an almost impossible task.

Companion Tools

The essay is accompanied by 4 companion tools to help readers explore in-depth and build an intuitive understanding of reasoning. 

The Nowable reasoning guide will give you a good overall introduction to what reasoning is and how it works inside a model

The field guide Elephants in Snakes, where I present key insights and data in a visual narrative

The datasite will allow you to explore all the data from the research and pit the models against each other interactively or see how they stack up on key metrics in the leaderboard. It is really fun to play around with so if you are curious about your favourite model or want to study the reasoning traces I encourage you to give it a spin.

Finally, I have released the harness used for the research on GitHub if you want to run the tests on your own models.

What I measured

I lined up twelve frontier models, seven open-weights and five closed, and ran the same sixteen tasks: ten light Danish tasks and three heavy tasks (code, two financial-document calculations), the heavy ones in two scenarios, a) with tools and b) without tools. I gave each model five passes on each heavy task cell to ensure a spread, and to map the variance and median performance of each. 

* Claude Opus 5 was added as a 13. model after first panel run. See the method note.

Throughout the test I monitored four things, kept strictly apart: 

All numbers below are computed from raw run data with declared conventions. Also, the scope matters: everything here is single-turn work without agentic loops. This is deliberate but see end note on limitations.

The headline: everyone is right, at wildly different prices

Although I did have quite a spread in model sizes from 31B to 2.8T parameters, correctness did not differentiate the models in my project. All twelve solved essentially everything.  

My task set was deliberately chosen to be common use cases, not PhD-level hard, but not easy either. For example, quite a lot of Danish context is required for the models to get most of the light questions right, and all of them did. 

Bottomline: The moment capability stops separating models, the interesting question is no longer can they, but what it costs to let them. And there the differences are enormous.

Insight 1: Most of your bill is processes you never see

This is where the elephant enters the room, or snake, so to speak. On most models this invisible thinking phase is the majority of what you pay for. On the open models, reasoning is 60-95% of billed tokens on heavy tasks. DeepSeek V4 tops the panel at 95%, and the typical open model spends two-thirds to four-fifths of your money on text you never see. On light everyday tasks the pattern holds: 60-80% for the open field.

There is one important exception: the closed Anthropic models run the opposite economy. Their reasoning share is only 20-24% on heavy tasks. Their bill is dominated by the answer, not the thinking. 

This is the really interesting takeaway here: the open and closed frontiers are not two points on one scale. Tokens may be the unit on the invoice but apart from that, they are two quite different economies.

Insight 2: Big models are expensive, but not because of how much they think

The intuition says big models think big. The data says the reverse. The newest closed frontier models think least of anyone: Claude Fable 5 spends a median of 32 reasoning tokens on the heavy tasks, Opus 4.8 spends 34, GPT-5.6 Sol spends 103. In the open field this range is roughly 400-1,200. Only one closed model thinks like an open one, OpenAI’s GPT-5.5, the top reasoning token consumer in the closed category with 409. But notably, 5.5 is already the previous generation of OpenAI models.

In other words, the premium price you pay for flagship models is a per-token price for capability rather than a bill for heavy thinking: a large model costs more for every token it produces, however few it needs. A plausible reading, and it is a hypothesis because the closed labs hide their reasoning traces, is that more of the path to the answer already lives in the weights. Regardless of these reservations, the pattern holds: a small model must reason its way to a result in many written steps, a large one sees it almost immediately. My research data is consistent with that: the smaller open models solve the same hard tasks but need between 3-30 times the thinking tokens to get there. The small model compensates for its size by thinking in steps; the large one has internalized the steps.

Note that 'the newest generation thinks least' is a trend, not a law. While OpenAI's version-jump reduced thinking by a factor of four, Anthropic's newest, Opus 5, thinks at a rate of 1.5 over Opus 4.8. 

The takeaway: overall big & closed beats open in terms of thinking less, and even Opus 5's raised spend, at 51 tokens, is roughly 10-20 times lower than the open field's.

Insight 3: Closed models hide their thinking, and their efficiency is a black box

All seven open models in the panel deliver complete raw reasoning traces on every run. The closed labs do not. Closed model providers are notoriously worried that Chinese labs will distill their models and steal the intellectual property. As a consequence the reasoning, which can be regarded as a blueprint of the intelligence, is increasingly tucked away. 

Not one of the five closed models returned a single genuinely raw chain of thought in the entire experiment: Anthropic's API returns summaries of the thinking or nothing; OpenAI's returns a token count without text. 

Everything this project can read in a trace, the language of thought, the waste, the false premises behind correct answers, is legible only on open models. Whatever makes the closed frontier models so token-efficient, we cannot inspect it. Which is kind of the point. My hypothesis that their weights simply contain more of the path (Insight 2) remains exactly that: a hypothesis, unverifiable from outside.

The key takeaway: We don’t know how closed models think. For any buyer for whom auditability matters, and in law, administration, and medicine, the justification is the decision, this is a procurement criterion in its own right. Closed models must be bought on output trust alone.

Insight 4: Reasoning efficiency varies enormously but we can measure the waste

Thinking is good. But is more thinking better? 

It turns out that not all thinking is progress. Mistral Medium 3.5 is the panel's clearest case: on a trivial task with a 120-word limit, it rewrote its answer repeatedly and counted the words by hand each time, burning over 3,000 thinking tokens on a task that should have been the cheapest in the set. I verified that this is the model's behavior, not an artifact of budget limits: with a fourfold-larger thinking budget it still counted words sixteen times. The judges agree. Europe’s only horse in the race, Mistral, has the highest redundancy scores in the entire panel, by some lengths.

But long thinking does not always equal waste or overthinking. China’s Fable challenger, Kimi K3, thinks heavily on hard tasks (median is 741 tokens) yet scores the lowest possible redundancy. Its thinking is thorough, careful and selective: it verifies the task's claims before building on them - where Mistral's seems impulsive, voluminous and circular.

The crucial takeaway here: volume and waste are independent axes. You cannot read quality off the token count in either direction. To identify if a model is overthinking or thinking overly redundant you have to look at the trace.

Insight 5: Token spend is wildly unpredictable, even when the answer isn't

The variance in token spend and efficiency isn’t just a cross-model finding. It exists within models too. Models take wildly different paths to the same destination. Across the panel, the typical gap between the cheapest and the most expensive run on the same task was a factor of two. For one run it was more than a factor of four. Same input, same model, same correct answer: a bill that doubles or quadruples purely by chance. 

Under controlled conditions, Kimi K3 used up to nine times as many tokens on the same input from one run to the next. Sometimes the model finds a short path; sometimes it takes the long way around, and nothing in the prompt decides which.

At high temperature the variance turns chaotic: in one controlled probe, GLM produced 4 tokens on one call and over 20,000 on another at temperature 1.5, on the same input.[1].

The practical takeaway: if you are trying to forecast token spend you can forecast using a model's median consumption, but any individual call can cost several times that, and budget planning has to price in the tail, not the typical case.

Insight 6: Shooting sparrows with cannons costs you between 40x and 222x in reassurance tax

What if you used a smaller model and it couldn’t answer your particular query, or even worse answered it wrongly? The experience from the early days of hallucinating models is still present in our memory. We are simply willing to pay a lot to get insurance against getting a wrong answer.

In Danish we have a saying, “shooting sparrows with cannons”, which essentially means you should align your effort with your task, otherwise you will overdo it. This is true for reasoning models too.

Every model in the panel solves the everyday tasks, some more elegantly than others. What separates them is the invoice. On a Danish legal-deadline task, all models returned the same correct answer at prices spanning from $0.00016 (Gemma) to $0.0355 (GPT-5.5). That is a factor of 222 for the identical result. 

Across the set of light tasks, the efficiency spread is 25-40x at near-identical correctness. On the heavy tasks, computed properly as correct answers per dollar actually spent, the span is 38x. 

To paint a picture: Gemma 4 delivers 1,286 correct answers per dollar when Claude Fable 5 delivers 33. The open models use more tokens, but their per-token price is low enough to more than compensate. 

This is the empirical anatomy of what I have coined the reassurance tax: the appeal to always use the big model is largely a purchase of a feeling of certainty, not of measurable output quality on tasks like these. Better safe than sorry, one might say. But that safety comes at a premium cost. 

I was not able to measure quality above the correctness ceiling. All models delivered correct results, so a flagship's surplus capability no doubt lives in tasks this set does not contain. 

When you work on large volumes of data and complex coding tasks in agentic loops, this will of course be different. But for the work most organizations and users actually run through these models daily, the data says the premium buys the same answer. 

The takeaway is trivial but it doesn’t make it any less true: for the majority of medium and light requests most reasoning models today are good enough. You don’t need a PhD-level doctor to diagnose a flu. Cast the right model for the right tasks if you want to reduce spend. 

A model like Gemma 4, which is open, can be self-hosted at roughly $2,000 per month and serve 50-150 users, enough for a small organisation. Or run it from OpenRouter directly. Either way, the prospect of a 40x cost reduction on trivial reasoning tasks, even if it is just on a subset of tasks, is likely to make someone pay attention.

Insight 7: Tools raise the bill, and don't change the result

Modern AI models are able to use tools to search and do things like calculations and coding as a kind of scratch pad. The idea is that this can help with either precision or efficiency of the answer produced. Thus, handing a model a tool sounds like it should reduce thinking, by offloading the reasoning the model should otherwise do on its own. 

In general, tool usage is known to increase input tokens, the cheap kind, and the idea is that we will then save on the more expensive side: reasoning, which is billed as output tokens. However, the opposite happens. For the models that actually take the tool, total token consumption rises, the median is 1.31x, with the heaviest tool users (the Anthropic trio) paying 3 times more in total tokens and 2.4 times more in cost. And on this task set the tools bought no correctness: results were the same with and without, on simple and complex tasks alike.

One model in twelve breaks the pattern: GLM 5.2 is the only model in the panel for which tools lower the total price (0.75x), because it genuinely exchanges heavy thinking for cheap tool rounds. 

The key takeaway: Tools spend more tokens, not less. The general rule if you want token efficiency is to consider tool use and design harnesses to only use them when they add value. Tools can pay, but as a model property, not a rule of thumb.

Insight 8: Tool use is a training signature, not a capability

Whether a model reaches for an offered tool is not as predictable as you may think. Despite getting the same invitation, the models handle this very differently: Gemma, GPT-5.5, and GPT-5.6 Sol never grabbed a tool (0%); Mistral almost never (7%); Claude Sonnet, Fable, and Inkling always did (100%). 

The signature persists across generations within one lab: Anthropic's three generations grab 93-100% and OpenAI's models both grab 0%, which points to training culture rather than architecture. There are also examples of how this changes over generations: Moonshot took Kimi from 87% in K2.7 to 40% in K3 in one version jump.

The key takeaway: Grab propensity is a property you should be aware of per model. It might increase the quality of an answer, or it might not. But model behaviour directly drives your bill (Insight 7).

Insight 9: Benchmarks measure answers. The differences live in the process.

On the industry's most-cited model scoreboard, Artificial Analysis' Intelligence Index (AAII), Kimi K3 and Claude Fable 5 sit almost side by side: 57.1 against 59.9, the kind of narrow gap that makes them seem roughly equal and one could assume that price is the only real differentiator. But the four things that separate them operationally never appear in that score: Kimi K3 thinks twenty-three times more than Fable. Fable answers in about five seconds where K3 takes over half a minute. Fable grabs the offered tool every single time, K3 fewer than half the time and K3's bill swings roughly three times more from run to run. Same capability band, opposite operating characteristics.

The key takeaway: A capability index like AAII is a statement about the destination. Everything in this report is about the journey, and the journey is what you pay for.

Insight 10: You don't control what language the model thinks in, and your language barely taxes the thinking

This project builds on findings from my previous Big Token Test, where I documented a language tax for most non-English languages. Overall, though, as reasoning makes up the majority of your bill the interesting question here is really: what language does a model think in? 

Fortunately, in reasoning tokens the tax turns out to be the small part, around 10% for Danish, once you separate genuine thinking from tokenization.

The bigger finding is about control. We posed the same six culture-neutral tasks in Danish, English, and Chinese to the five open models and measured the language of the trace.

·      An English prompt produced English thinking 100% of the time. 

·      A Chinese prompt produced Chinese thinking only 53% of the time 

·      A Danish prompt produced Danish thinking just 33% of the time. 

The pattern is clear: English is the gravitational center the models fall back to. The per-model behavior spans four types: Gemma thinks in English no matter what you ask in; DeepSeek and Mistral mirror the prompt language reliably; GLM loses Danish to English but holds Chinese; and Kimi 2.7 falls back to English even from a Chinese prompt, five times out of six, the most counterintuitive result in the experiment.

Mistral reasoned harder on the English versions of the same tasks (about 1.5x), GLM genuinely harder on Danish (1.36x), DeepSeek flat across all three. And the Chinese figures should be read as indicative only: nearly half the Chinese calls fell back to English thinking, and Chinese tokenization makes its token counts incomparable by construction.

The key takeaway: If it matters that the model reasons in your language, say because auditing the thinking in Danish is the point, that is a property to test per model, not something a Danish prompt buys you. Only two of five delivered it reliably.

The rule of thumb

Across twelve models, heavy and light tasks, this is the consumption model the data supports:

tokens = model × prompt × ε(model) × tool use

where ε(model) is the specific model variance. I’ve decomposed the spend on the heavy runs: which model you choose explains 54% of the variation in the bill, what you ask it explains 27%, and 13% is pure variance: the same model on the same task simply costing more one run than the next. Your choice of model matters twice as much as your choice of task, and a sixth of the bill is randomness nobody controls. Tool use then multiplies on top (Insightht 7).

Routing is the obvious answer

I guess most of you by now have arrived at the same conclusion. There is a lot to gain by delegating your tasks to small models. The obvious fix is a router, i.e. a layer that picks the model for you. Many companies are looking into this, and the topic is too extensive to cover in depth here but too important not to mention. 

For the purpose of this section, I tested two router versions. There are plenty of other third party routers out there and to get an overview you can check RouterArena's leaderboard.

In the first run I used OpenRouter's original Auto router built on market-leading NotDiamond (since deprecated by OpenRouter itself). Auto collapsed all available models to two preferred generalist models. It was 28x more expensive than the cheapest correct panel answer on every single task and delivered slightly lower correctness, while the routing layer itself added roughly 3x tokens and 2x latency on heavy tool tasks.

Then I ran its successor, Auto Beta, which is a bit different. Where most routers have learned their routing capability like a model learns to answer based on data, Auto Beta routes on evidence: it classifies the task and follows in real time what the developer community actually spends on for that task type, with a cheap-favouring default. The difference is dramatic. Same sixteen tasks, four times cheaper than its predecessor. The spread is five models instead of two, perfect heavy-task correctness, and 406 correct answers per dollar, which would rank it third in our entire panel, behind only the two cheapest open models themselves. Auto Beta supports the main findings of my research: route cheap, because correctness is equal, and let evidence decide which model.

Auto Beta's choices were 100% stable over the 5 runs I made. The same prompt was sent to the same model every time, so its implicit task taxonomy is fully legible in the assignments:

Three things stand out. The router distinguishes writing code from debugging code  and reserves its single premium choice, the only closed model it touched, for the debugging task, which matches where the community's spend evidently points. 

Its legal pick, DeepSeek V4 Pro happens to be the one panel model that derives Good Friday from first principles, so the routed answers carried the correct premise. And its one failure is a miscast, not a policy error: the logic puzzle went to a cheap model that worked hard (848 thinking tokens per run) and simply couldn't solve it. Like everything else here, it is a snapshot of a rolling evidence window.

A few considerations worth mentioning here.

First, in theory a router can never beat knowing your own task set. If you know your workload, one well-chosen cheap model remains unbeatable. If you have enough examples and your data is homogenous enough building your own router or harness might yield even better results.

However, many custom routers fail because they are too deterministic and not sufficiently granular in their routing or simply because the world is moving so fast that what they learned gets outdated. 

OpenRouter’s Auto Beta has remarkably high performance and manages the balance between price and quality impressively well. 

OpenRouter states that it does not store prompts or responses by default, but its rankings, and therefore Auto Beta's routing, are powered in part by what it calls anonymous input categorization: sampled user prompts, read and classified by task type, joined with the spend metadata it retains anyway. 

This way OpenRouter has quietly built a moat in this market. They are one of the only companies with that much data on model-task preferences. Their routing quality is a function of traffic visibility. Few players sit on enough cross-model traffic to build it:  aggregators like OpenRouter and Azure AI Foundry and product owners like Cursor. 

You should be aware of the tradeoff: using a router like Auto Beta means that your data is likely to be used for router training. This is not a new tradeoff: most AI services today harvest your data for training unless you have an enterprise deal, but it might be a showstopper for some organisations.

A router's real product value is its insurance against not knowing your workload, and adaptability on the traffic you haven't seen yet. In a sense using a capable router can efficiently reduce the reassurance tax. My test shows that what this tax buys, the felt certainty that someone competent is handling it, can be purchased from a routing layer at a tenth of the price, and with better grounds than the flagship habit ever had, because the router's cast rests on evidence of what thousands of others actually use for the task type: crowd-sourced assurance that moves with the market. 

However, while you no longer overpay per token, you pay in delegated trust you cannot audit. My five test runs showed one task type miscast, failed on every run, invisible to anyone who trusted, precisely because the checking was what they had outsourced. 

Wrap up

And so we have reached the end of the journey. At the start I promised to show you the elephant inside the boa and asked you to keep three surprises in mind. You have now seen the data behind all three. The flagships are expensive because of what each token costs, while they think less than anyone. The newest frontier skill is knowing when thinking is unnecessary. And the same model, given the same prompt, can bill you double or quadruple purely by chance.

The grown-ups look at the invoice and see a hat. If you have read this and the reasoning guide, you should now see what is actually in there: a hidden thinking phase that makes up most of what you pay for. A reasoning model is an instrument with measurable properties: the share of invisible reasoning, the waste, the variance, the grab rate, the latency, the transparency of its trace. Study them before you put a model into production, because otherwise you pay for thinking you never see, accept waste you could have measured, and trust justifications you never checked.

The path forward is a middle one: neither the most expensive model for everything nor the cheapest for everything, but knowing what you are paying for, when it is worth it, and building your systems accordingly.

If you want to manage token spend, every scenario carries its own trade-off:

1. Mono-model flagship (all-in on Claude or OpenAI) costs money. You overpay 65-89% on ordinary work in exchange for simplicity and the felt certainty that quality is handled. Certainty that you mostly don't need at this task level, as my tests have shown.

2. A data-driven third-party router (like Auto Beta) costs control. You save most of the flagship premium, but you pay in delegated trust you cannot audit, the occasional invisible miscast, and data exposure to the routing layer.

3. A homebuilt router costs work, ongoing. Built on your own measurements, this can seem like the best of all worlds. Your casting rules, your answer key, near-optimal cost. But you also own the maintenance, and this is no small feat: eight of thirteen of my own models that I pinned for this study had changed within a few weeks. Built without your own measurements, an LLM guessing which model fits, it is the worst corner of the map: the build cost and the blind spots, with nobody's evidence behind the choices.

4. Manual casting in custom harnesses costs work, per task. Humans or fixed pipelines assign each job to a chosen model. Maximum control, quality and economics, and it scales exactly as far as your attention does.

Ending remarks: Some key reservations

Every conclusion above holds for single-turn work without agentic loops. Even amid the current agentic software boom, single-turn calls remain a large and real share of actual real-world usage (not fancy benchmarks), so the findings stand. But three signals in our own data suggest an agentic design could invert parts of the picture. First, tool-grab propensity flips from cost to prerequisite when the task genuinely requires tools. Second, calibration, i.e. spending thinking where thinking is needed, matters. Fable is the panel's most disciplined spender: its thinking rises and falls with the difficulty of the task. That discipline is invisible when every model answers correctly anyway but over a fifty-step agent run, spending wisely at every step is exactly what you would be paying for.  The third is speed. On a single call, the difference between five seconds and half a minute may matter less. But an agent makes its calls in sequence, each step waiting for the last, so the same difference, repeated across hundreds of chained calls, turns into hours. The provider controls how fast tokens are served. The model controls how many it needs. Thinking is tokens, and tokens are time. A model like K3, which is seven times slower per call, is seven times slower per working day because it simply has more voluminous thinking. 

Whether the flagship premium justifies itself in agentic workloads is a question this project has not measured, and assuming it does would be the reassurance tax with the sign reversed. The agentic benchmark is the natural sequel.

Finally, what have I added and could a public leaderboard have answered this? I would say no because by design this is different. Capability indexes need tasks hard enough that models fail differently, so they live above the correctness ceiling. This study interntionally lives below it, in the everyday where most users spend their time, where every model is right and only the bill differs. This is the reality benchmark. Everything here is about the journey on ordinary work: what the same correct answer costs (a 222x spread), how consumption swings on identical input, whether the model grabs tools, what language it thinks in, and what its reasoning actually looks like. None of that appears in a score, which I why my hope is that this will resonate with the everyday user than how models perform on the frontier benchmarks.

[1] Note that Cross-model comparisons of variance should be read with one reservation: sampling parameters were not pinned across providers in the main runs, so part of any between-model difference may reflect provider defaults rather than the models themselves