Books have the fun property that sometimes they just burn some random facts into your mind. I still recall that je parle français comme une vache espagnole and I also still recall that Pi seconds are a nanocentury. The latter comes from an excellent chapter in the Programming Pearls stating that napkin math and ballpark estimates are often an engineer’s best friend. Particularly, doing such estimates strategically and repeatedly helps to alleviate a common AI pitfall: You can reject AI use cases that are currently infeasible, but you should not let that rejection become permanent.

Why? Because a common theme in AI is that you look at a problem or a proposed AI driven solution and you see that this just can’t be done with the current models’ capabilities, costs, hallucination rate or some other restriction. You turn away for 6 months, and suddenly some competitor has implemented the very thing that you thought impossible with the newest model iteration. This is a pity, because now you are on the back foot.

So, what to do instead? Whenever an idea is deemed too expensive, too unreliable or beyond current model capabilities try defining a cheap ballpark metric that tells you when the idea is worth revisiting.

For instance, when I was working in scientific NLP, processing a huge document base with a large model was something that would have been useful for a variety of use cases from better metadata extraction to deep information extraction. However, when I first thought about this in 2023, processing even a smaller database like PubMed Central was impossibly expensive as we can see by a quick ballpark estimate:

• PMC has roughly 12 million articles
• articles have on average ~10 pages
• assume ~400 words per page (don't bother with tokenization for now)
• Rough ballpark for token count: $12e6 \cdot 10 \cdot 400 = 48e9$ tokens

AT GPT-4's API pricing of $30 per 1M input token, this already yields $1.44 million for the input data alone; a rather expensive run. However, now that we have defined our rough estimate on how much this costs us, we can start to run this iteratively to verify our assumptions. Suppose we had run this on every major model release; we would have seen a clear trend of model capabilities becoming cheaper and cheaper. With GPT-5, this became $60,000 input token costs, and with GPT-5.6 Luna, only $10,000 input token costs (apples-to-oranges, but let’s assume Luna may compete quality-wise with GPT-4 on our task).

If you keep this ballpark estimate around, a new model release can trigger a second look as soon as the order of magnitude changes. At that point, it becomes worth refining the estimate with additional information: perhaps a self-hosted model is sufficient, or batch APIs reduce the cost further. That is, the ballpark estimate doesn’t tell us if the idea is feasible right now, but it tells us it is worth reconsidering it and it does so with little expense and at the right time.

Note that, we do two important things here. First, measure the concrete problem or use case that was formerly deemed unreachable. You could also watch general capability benchmarks. However, if so, you need to have a way to correlate their performance to your concrete problem, which often does not exist. Second, design the metric as a rough indicator: quick to compute and only accurate enough to tell you when a closer look is warranted. This is not a full evaluation setup after all.

The above example is a simple one, but the concept is quite generally useful and helps to not be surprised by model capability changes. Whenever you have a case of: “This would be nice but doesn’t work yet” it makes sense to think about an indicator that you can check once every month or model update to see if the formerly impossible is now within reach. Concrete indicators could for instance be:

• Cost to process a specific task. Take a fixed workload, multiply it by today's model prices, and see whether the order of magnitude has changed.
• Previously impossible benchmark cases solved. Keep ten examples that the model currently fails badly on. On every major release, rerun them and count how many now work. 0/10 → 2/10 → 7/10 is already a useful strategic signal
• Amount of agent engineering required. Pick one task and note what it currently takes to make it work: custom prompts, hard-coded routing, examples, retries, workflow logic. Try to do that with a leaner or no setup for new models.

In all of these we do not care about the exact number. The point is a cheap tripwire. If the number moves enough, then it is likely time to revisit your prior decisions by building a proper evaluation scenario to determine if the use case has become feasible.