If this is as good as it gets

Every time a new AI model comes out, the internet splits into two camps.

The first camp is blown away. The demos look incredible, the motion work is slick, and it will build you a polished website in an afternoon. The second camp is unimpressed. They know how transformers work, they’ve read the papers, and they’ll tell you it’s the same machine with the knobs turned a bit further. Tuned to score well, better at some things and quietly worse at others. A bit of a show.

I saw exactly this play out recently. Someone with real technical depth posted that they weren’t impressed, and the replies told them, not always politely, that they were wrong because the thing is amazing.

I think both camps are right. I also think it matters a lot less than either of them believes.

The sceptics have a fair point. The labs are in a race, and the scoreboards are how the race gets reported. There’s an old idea from economics called Goodhart’s law: once a measure becomes a target, it stops being a good measure. Researchers have started calling the AI version “benchmaxxing”, training towards the tests rather than towards what people need. A study this year of sixty widely used benchmarks found that nearly half have simply run out of headroom, with the top models bunched too closely to tell apart. People have even bet more than $225 million on which model will top the leaderboard. So when someone who lives in the technical detail says recent progress looks more like polish than leaps, I don’t think they’re being a killjoy.

It’s also just how the market works. Every phone needs more megapixels than last year’s. Bigger numbers sell, and nobody is ever allowed to settle for what they’ve got.

But the people who are wowed aren’t imagining it either. I use AI to make videos, games and little personal helper tools, and what it can now do with motion and design genuinely impresses me. Nobody who has watched it work is being fooled.

Here’s the question I care about more. If progress stopped tomorrow and this was as good as it gets, would it matter?

I don’t think it would, not for years.

Most people I talk to use AI as a cleverer search box. They ask it questions and get answers back, which is useful, but it isn’t where the power is. The power shows up when it’s editing your files, organising your processes and building the tool you were missing. Very few people use it that way yet.

There’s a name for this gap: the capability overhang, the distance between what AI can already do and what people actually use it for. The term has been around AI circles for a while, and OpenAI pushed it into the mainstream in January. (Worth saying they sell the stuff, so of course they’d like you to use more of it.) Their figures suggested that, on the same paid plan, the typical power user leans on roughly seven times more of its deeper, step-by-step reasoning than the typical user. Other surveys show plenty of people now have access to AI at all. Access isn’t the same as depth, though, and depth is where the gap sits.

Fig. 1 · The overhang

The gap isn't in the models. Even if what AI can do stopped rising today, what most of us do with it would have years of climbing before it caught up.

Part of the reason is simple: it takes time. You only really learn what it can do by living with it for a while. And part of it is that plenty of people tried something two years ago, it didn’t work, and they quite sensibly gave up. With every other piece of software, that’s the right call. If it couldn’t do something last year, it probably can’t now.

AI doesn’t work like that. It changes month on month, and sometimes the same model will do something it refused to do before just because you asked it differently. My one rule is never to assume what it can’t do. I regularly go back to things it failed at and try again.

So if this really were the ceiling, the job would be to make what we already have cheaper, smaller and able to run on our own machines. That’s not a trivial wish. The build-out has a cost that lands on everyone: memory chips are being swallowed by data centres, and the price of RAM has gone through the roof. That doesn’t just hit PC builders. RAM is in your phone, your laptop and the box under your telly.

I don’t want progress to stop. I’m curious where it goes next, and I’ll be first in the queue to try it. But whichever camp you’re in, the thing that matters is the same. What we already have is enough to change most people’s lives, and most people haven’t really opened the box yet.

The race will carry on, because that’s what races do. Meanwhile, for most people, the biggest upgrade available isn’t the next model. It’s using this one properly.

If this is as good as it gets

Further reading