The 8K Constraint That Made Everything Better
What a token limit taught me about wasted engineering.
I had a pipeline that worked.
Not “worked” in the demo sense — worked in production. Real records, real classifications, real accuracy numbers I could defend. Built on a commercial LLM with a generous context window, the kind of runway that lets you be lazy and still get results.
Then I decided to rebuild the whole thing on a local open-source model running on my laptop.
The context window dropped from tens of thousands of tokens to roughly eight thousand usable tokens. Not because the model couldn’t technically accept more, but because once you’re running a 32-billion parameter model on consumer hardware with a sharded memory footprint, the practical ceiling is much lower than the spec sheet promises.
Eight thousand tokens. That’s the constraint.
And it made everything better.
The Lazy Architecture You Don't Notice
Here’s what a generous context window does to your engineering: it makes you stop thinking about what goes in.
When you have room, you pack everything. The full taxonomy. The verbose prompt. The few-shot examples. The system instructions that grew over weeks of debugging, each paragraph a scar from a specific failure you patched by adding more words.
It works. You ship it. You move on.
But “it works” isn’t the same as “it’s well-engineered.” It’s the equivalent of solving a storage problem by buying a bigger hard drive. The mess is still there — you just can’t feel it anymore.
When I hit the 8K wall, I felt it immediately.
My taxonomy alone — the reference data the model needs to classify records — was over 130,000 rows. Even compressed to unique pairs, it was still roughly 2,400 entries. There was no world in which I could fit that into 8K tokens alongside a prompt and the records themselves.
The first instinct was to call it a limitation. The model is too small. The hardware is too constrained. This approach won’t work.
The second instinct — the one that actually matters — was to ask a different question: What if the problem isn’t the constraint? What if the problem is everything I was sending that I didn’t need to?
Compression as Engineering Discipline
What followed was the most productive engineering week of the entire project.
I built a retrieval layer that indexed the full taxonomy into a vector database, then at inference time, pulled only the entries relevant to the specific records being classified. Instead of sending 2,400 taxonomy pairs, I sent roughly 500. The accuracy barely moved. The token footprint dropped by 94%.
Ninety-four percent.
That means 94% of what I was previously sending to the commercial LLM was noise. Context window filler. Tokens the model was processing, that I was paying for, that contributed nothing to the output.
I didn’t discover this because I was being rigorous. I discovered it because I had no choice.
The retrieval layer itself required real engineering. A naive vector search — “find the taxonomy entries most similar to this record” — sounds straightforward, but it retrieves at the wrong granularity. MRO records describe parts using scattered attributes. The taxonomy is organized by noun classes. Matching a sentence to a category isn’t the same as matching a sentence to a sentence.
So I built a noun-first reranking strategy. Extract the likely noun from the record first, retrieve taxonomy candidates that share that noun class, then rank within that filtered set. The candidate coverage jumped to 75%, which meant three out of four records were seeing the right taxonomy neighborhood before the model even started thinking.
None of this existed in the cloud version. It didn’t need to. The cloud version had enough room to brute-force the problem with volume.
The constrained version had to be precise.
The Universal Pattern
This isn’t an AI story. This is an engineering story that happens to involve AI.
Every experienced engineer has lived a version of this. The project that had to ship on hardware half as powerful as planned. The API that had to work within a rate limit nobody anticipated. The team that lost three engineers mid-sprint and somehow built something tighter and more coherent than the original plan.
Constraints don’t degrade engineering. They reveal what was wasteful.
The generous version of any system carries accumulated slack — decisions made when resources were abundant, patterns adopted because they were easy rather than correct, complexity that grew because nobody had a reason to challenge it.
Constraints are the reason.
I’ve seen this pattern across thirty years of building software. The database that performed better after we halved the hardware budget because we were finally forced to fix the query plan. The monolith that became a cleaner architecture not because microservices were better in the abstract, but because the deployment constraint forced us to think about boundaries we’d been ignoring. The startup that built a better product than the enterprise competitor because they couldn’t afford to build a mediocre one.
There’s a phrase in architecture — “the constraints are the design.” It means the building isn’t shaped despite its limitations. It’s shaped by them. The lot size, the setback requirements, the soil conditions — these aren’t obstacles to the architecture. They are the architecture.
Software is the same. We just forget it because compute is cheap enough to let us forget.
What the Constraint Actually Revealed
Let me be specific about what the 8K limit forced me to confront:
The prompt was bloated. Instructions that had grown over weeks of iteration contained redundancies, edge-case handling that applied to 2% of records, and verbose explanations that the model didn’t need. Trimming the prompt wasn’t just a token exercise — it was a clarity exercise. A tighter prompt produced more consistent output because the model had fewer competing instructions to weigh.
The retrieval was lazy. Sending the full taxonomy was the equivalent of giving someone an entire encyclopedia when they asked a specific question. The vector-indexed, noun-first retrieval didn’t just save tokens — it gave the model a more focused, higher-signal context. The model wasn’t better. The context was better.
The batch size was arbitrary. The cloud version processed records in larger batches because it could. The constrained version forced me to find the batch size that actually balanced throughput with accuracy. Smaller batches with better context outperformed larger batches with diluted context.
Every one of these improvements applied backwards. When I eventually compared the constrained pipeline’s accuracy against the original cloud version, the numbers were within a few percentage points — not because the local model was as capable as the commercial one, but because better engineering around the model closed the gap that raw capability had been masking.
The expensive model wasn’t doing better work. It was compensating for worse engineering.
The Uncomfortable Implication
Here’s what most teams don’t want to hear: if your AI pipeline requires a large context window to function, you probably have an engineering problem, not a model problem.
The context window is not a feature to maximize. It’s a resource to minimize. Every token you send should earn its place — not because tokens cost money (though they do), but because extraneous context degrades output quality in ways that are hard to measure and easy to ignore.
This applies far beyond local inference. Teams running production workloads on the largest commercial models are paying for context they don’t need, getting results that are good enough to mask the waste, and building architectures that become more expensive and more fragile as they scale.
The 8K constraint didn’t limit what I could build. It revealed what I should have built from the start.
The Lesson I Keep Relearning
Thirty years in, and I keep arriving at the same place through different doors:
The best engineering happens when you can’t afford the obvious solution.
Not because scarcity is virtuous. Not because constraints are fun. Because the obvious solution — more compute, more tokens, more budget, more time — is almost always the one that carries the most hidden waste. It works well enough. It ships. And it accumulates cost and complexity that compounds silently until something breaks.
The constraint forces you to do what discipline should have demanded all along: understand the problem well enough to solve it precisely.
Eight thousand tokens wasn’t a limitation.
It was an education.
About the Author
Raghu Vishwanath is Managing Partner at Bluemind Solutions, a product engineering firm specializing in MRO master data governance. He writes about software engineering, AI, and building platforms that last.