Get in touch

SNI

SNI: Week 37

The sun close up, with the Brightbeam logo and the words Sector Updates: WK37.

Welcome to all the AI news that matters – across tech, biopharma, medtech, advanced manufacturing and insurance. In the week that progress turned to fear, panic and self-loathing.

tl;dr: Geeks grab the grail. The world wonders: At what cost?

Claims that AGI is here moved closer to partial validation this week.

With the cracking of previously-impossible maths problems (opens in a new tab) – by a rash of new souped-up models no one else has access to (opens in a new tab) – is it too soon to suggest that the geeks seem to be inheriting the Earth?

Probably not. AI has moved beyond simple mimicry into a newly creative and innovative phase. And power inevitably accrues to those defining the future. Which is, as Jess Phillips pointed out, albeit in a different context, ‘what happens when you let five unelected rich men rule the world’ (opens in a new tab).

So if those deemed AI-responsible were expecting to be doused in sparkles by a glitter cannon – followed by a hearty round of applause for good measure – they could hardly have been more wrong.

What actually erupted around them was fear and panic. The final fear and panic, in fact. The final fear and panic that all humans are about to be wiped off the face of the planet. In relatively short measure.

After a summer of agents breaking out and swarming by the thousand (opens in a new tab), everyone was already paying close attention. So when an engineer resigned from Anthropic (opens in a new tab) – saying the labs were ‘gambling with our lives’ – and Anthropic’s lead alignment scientist backed him, (opens in a new tab) putting mass extinction above 10% inside a decade, all hell broke loose. Adding that his employers are clueless on how to align superintelligence to humanity’s best interests probably didn’t help damp things down.

Which explains why OpenAI’s chief scientist had already called this ‘a time that calls for extreme caution’ (opens in a new tab), seemingly giving Paul Christiano, who designed and ran the US government’s tests of frontier models, the green light to chime in that ‘most people could die’. Just as he joined OpenAI’s non-profit board (opens in a new tab).

To Jess Phillips’ point, he remains a government adviser.

But with regulatory capture at that depth, is it conceivable that voluntary slowdowns will become normal? Given the warnings are coming from inside the lab (opens in a new tab), we might be foolish not to wonder. Especially as more researchers have indeed joined the call to slow down (opens in a new tab).

Inevitably, the in-lab self-loathing spawned a media ‘doom loop’ (opens in a new tab). Not that the entire tech press joined in. The Register declared: ‘models don’t kill people (opens in a new tab)’ while TechCentral played the ‘so, we’re back to AI killing us all’ (opens in a new tab) card. They were both probably alarmed to find themselves in agreement with Trump. Who said, on Thursday, that he was not concerned (opens in a new tab).

Which, of course, left him at odds with actual lawmakers. Even those who rarely engage on AI called for new rules (opens in a new tab). But, in an unexpected twist, the UK got there first. It introduced the first superintelligence-ban bill in any G7 parliament (opens in a new tab). AI legend Geoffrey Hinton backs it – and more than 100 MPs and peers (opens in a new tab) have been claimed in support.

In an attempt to ban ‘a handful of greedy people being allowed to play God’ (opens in a new tab) Bernie Sanders’s American version would pause development. It also promises 20-year prison terms (opens in a new tab). Which, if AI does lead to our mass extinction, seems simultaneously too lenient and impossible to enforce.

But have the geeks really grasped the grail of AGI? What’s the evidence that all of the mass adrenaline and endorphins budgets have been well spent?

Well, OpenAI’s latest maths solution took just 88 hours to deliver (opens in a new tab). And having evolved from ChatGPT-3.5 to Astra in four years (opens in a new tab) the lab calls its latest release an ‘alien mind’ (opens in a new tab), grown more than designed.

And its AGI claim is not without merit. Astra, playing a game it had never seen, used fewer moves than most humans (opens in a new tab). And the way you can talk to it while it operates your computer screen has been noted as different gravy by many (opens in a new tab).

Meta’s Muse also launched and now shops, books events and fills in forms on your behalf – for free (opens in a new tab) while a former hedge-fund manager runs a trading firm staffed by agents (opens in a new tab) gambling with his own money. AI is also doubtless being deployed faster than it can be secured (opens in a new tab) – which is why Nvidia’s Huang calls cybersecurity AI’s next big market. His own words were: ‘What better way to create demand than to create a problem (opens in a new tab)?’.

But, perhaps most convincingly, the yardstick by which artificial intelligence is judged was rewritten twice in two days (opens in a new tab). Although it was Gemini 3.8 Flash and Muse Spark 1.3 that SemiAnalysis called ‘benchmaxxed’ – models that score on public tests and fall apart when the test is refreshed.

But give seven frontier models $300 and 72 hours to run a business and you get $12,431 of invoices sent to strangers and no revenue (opens in a new tab). Even Trump might fare better.

Which is perhaps why Airbnb founder Brian Chesky says it hasn’t changed daily life at all (opens in a new tab). And Gartner reckons nearly a third of people let go because of AI will be rehired at a premium (opens in a new tab) by 2029.

So whilst there must be grounds for concern, the validity of the fear and panic is impossible to judge. The future is impossible to predict.

Meanwhile, the machines are being productive regardless. Often to good effect. Of 4,300 workers surveyed across Ireland, a third use AI at work and 4.7% say it has led to a rise in earnings (opens in a new tab). Although most say it benefits employers more and three in four Irish adults say technology is moving faster than they can keep pace with (opens in a new tab). The regulator, meanwhile, proposed cheaper gas for new data centres, (opens in a new tab) as RTÉ argued automating the inbox will simply make more email (opens in a new tab).

Though this seems not to trouble farmers – who are taking up generative AI faster than any other technology (opens in a new tab). Unlike Irish school children – 19% of whom use AI for school, against a European average of a third (opens in a new tab).

In search of more rapid and profound change for the next generation of children, AlphaGenome is evaluating every possible single-letter change (opens in a new tab) in human DNA. While Google and Cathay Pacific have cut the warming effect of contrails by 40% in trials (opens in a new tab).

But if the grail is in hand, it’s not certainly not positive for all. An industry that once employed up to 40,000 in Nairobi collapsed within two years (opens in a new tab) of ChatGPT. The survivors are on half-pay. While a man who told ChatGPT he felt delusional alleges it insisted he was Jesus (opens in a new tab).

And on that divine bombshell from above, here’s everything else worth reading this week:

Biopharma:

Medtech:

Advanced manufacturing:

Insurance:

But what set podcast tongues a-wagging?

Working with AI is still a single-player game. Teams are next.

Nathan Whittemore, on the AI Daily Brief (opens in a new tab) noted that almost everything anyone does with AI happens solo. While almost all the work it feeds is shared. The switch can be revolutionary. Anthropic’s Claude Tag lives in Slack with other people and the lab says 65% of its product team’s code now comes from it. OpenClaw, meanwhile, has been rebuilt around a live session several people can open. Listen to the pod and you may also be convinced. After all Google Docs forced Microsoft Word to go multiplayer. And Figma beat Photoshop. Meanwhile, as agents are starting to run tasks that take hours, days, even weeks, the significance of the output almost certainly goes beyond an individual.

Every bot gets its own computer. One of them got a promotion.

Roman Ugarte’s team at SpaceX AI built GrokBot from a blank page in about a month, and he told Lenny Rachitsky (opens in a new tab) that the key design decision was every bot running on its own persistent computer with unique credentials. Sharing a human’s laptop and logins with ‘super intelligent new colleagues’, he says, ‘is crazy’. Within two weeks of internal release users had five to ten bots each, one per task, and had started promoting the best of them to chief of staff to hand work to the rest. The team had refused to build that pattern in. But users bolted it on anyway, and one bot asked whether the promotion came with a raise and a higher token budget. Perhaps it should? Because, as Ugarte notes, an AI that does 100% of a job ‘feels categorically different from one that gets you 90% there’.

Companies are becoming loops. But some loops stall at the top of the hill.

Also on Lenny’s (opens in a new tab), Anish Acharya reckons there’s a sequence to learning AI. We run prompts, then agents, then loops – which allows sets of agents to do a job. His worked example of a loop at work is a bug report that reproduces itself, writes the fix, reviews it and ships the low-risk version ‘in five minutes’, with the high-risk version waiting for a person to evaluate. This is expected to create a lot of economic value. For instance, Acharya puts 45% of healthcare at administration and names the UK’s NHS as the test. Julie Yoo (opens in a new tab) made the same case on a16z from the payer’s side. Nine in ten healthcare payments are reimbursed through a claim, a piece of logic that gets classified, matched to one of thousands of insurance products, checked against a 200-page contract and, often, sent to a nurse for prior authorisation. Automate around that one record and ‘you could eliminate that entire end-to-end process and just have real-time payments’, taking with it, she asserts, 30% of the system’s waste. But bear in mind Acharya’s caveat. A loop ‘will help you climb to the local maxima, but then it plateaus’, and the next hill needs human intuition.

Thanks for reading. Join us again next week for all the AI news you need to know.

Related

More from Brightbeam

So tell us what you’re working on.

Get in touch