THE AI CONTEXT WINDOW ARMS RACE IS MISSING THE POINT

Look at the size of mine! But is bigger necessarily better?

In some contexts, a 100,000-token window beats a 32,000-token one, and a million-token window beats both. Subquadratic, a recently-launched startup, is claiming 12 million tokens and a thousandfold efficiency gain.

But researchers are split on whether their claims are right.

Headline numbers are doing the work they always do – the selling. But they mask the right way to think about context.

Firstly, because greater size doesn’t necessarily mean more context. It can just mean more noise for a model to sift through.

And secondly because advertised context is what gets put on the spec sheet. But effective context is what the model can actually find, connect and defend inside that window. The gap between the two is the 'lost-in-the-middle' problem. Named by Liu et al. in 2023, it keeps showing up in long-context benchmarks.

Which means a model with a vast advertised window can swallow a whole claims file and still miss the decisive paragraph. Hand it a technical dossier and it'll skip the contradiction in the appendix. Drop in a policy pack and it'll cite the wrong clause.

Advertised size guarantees nothing.

Nevertheless, Subquadratic’s methods are well worth watching. The aim of their ‘sparse-attention’ architecture is to reduce the cost of long context. Whether it improves its quality as well is a question the field hasn't settled yet.

But if both cost and quality gains turn out to be real, headline numbers start mattering even less.

So what’s today’s in-the-end-at-the-end?

Bigger context windows are a bit like megapixels. The number describes a technical capability, not a better outcome. Effective context is what we all crave.