How much does an LLM infrastructure cost a business?

Mine runs at about $2.40 a week for ~4,800 calls, averaged over the last 30 days. A free local model on my Mac takes 92% of the volume; paid models only get what it has measurably failed at.

When does a task move from the local model to the cloud?

After a measurement. Both models run side by side on the same task: if the quality matches, the work stays local. That's how triage, classification and summaries moved to the Mac while text-quality scoring stayed in the cloud.

How do you keep LLM costs under control?

Three rules: one gateway for every call, static prompts longer than the caching threshold, and costs logged in your own database. Once a week I read the slice: what got pricier and what's ready to go local.

← to the journal

2026-07-11#llm#экономика#архитектура

Token economy: an LLM infrastructure for under three dollars a week

Sergei Pak · designs AI operating systems · русская версия

My system makes around 4,800 LLM calls a week and pays $2.40 for them. That's the average over the last thirty days; in one week the bill hit four dollars, when the system chewed through an archive of correspondence in one go. Six months ago it was 855 calls and 50 cents. Volume grew more than fivefold, the bill fourfold. No discounts, no credits; costs grow slower than the work, which is the main sign the architecture is put together right.

Local model by default

A 9B model on my Mac takes 92% of the calls. It's free, it's fast, and the data never leaves the machine. The cloud gets a task only after a measurement shows the small model can't handle it.

Here's what one of those measurements looked like. For a month I ran a cloud model and the local one side by side on text-quality scoring. The local one gave every text the same 75 out of 100, zero variance. The cloud model spread them from 32 to 62. I cancelled the migration and left quality scoring in the cloud. Triage, classification and summaries, though, went local long ago: no quality gap there.

This economy isn't down to a lucky hardware setup. Stanford HAI's AI Index 2025 measured inference at GPT-3.5 level dropping more than 280-fold between November 2022 and October 2024, while the gap between open-weight and closed models narrowed from 8% to 1.7% on some benchmarks in a single year. Work that needed expensive cloud the day before yesterday now fits on a working laptop. The savings go to whoever noticed and rebuilt their routing.

Three rules

One gateway for every call. Every LLM request goes through a single module. It tracks the cost, picks a model for the task, and caches prompts. A direct API call from the code gets caught at commit time.
Prompts that cache. The system prompt is static and longer than the caching threshold, so a repeat call pays pennies for the part the model has already seen.
Costs live in a database. Who spent it, which model, how many tokens. Once a week I read the slice: what to move local, what got pricier and why. I wouldn't trust a line in a provider's dashboard with this.

Why a business should care

The gap between three dollars a week and three hundred isn't the real point. This is: cheap infrastructure can afford to think all the time – triaging every email, checking every contract, recalculating the numbers daily instead of once a quarter before the report. The moment each call costs real money, you start saving on how often the system pays attention, and it goes blind.

How the whole layer works, from context to autonomy, is in what is an AIOS. To size this economy against your own volume, book an online consultation.

FAQ

How much does an LLM infrastructure cost a business?: Mine runs at about $2.40 a week for ~4,800 calls, averaged over the last 30 days. A free local model on my Mac takes 92% of the volume; paid models only get what it has measurably failed at.
When does a task move from the local model to the cloud?: After a measurement. Both models run side by side on the same task: if the quality matches, the work stays local. That's how triage, classification and summaries moved to the Mac while text-quality scoring stayed in the cloud.
How do you keep LLM costs under control?: Three rules: one gateway for every call, static prompts longer than the caching threshold, and costs logged in your own database. Once a week I read the slice: what got pricier and what's ready to go local.