SNI: Week 38

Welcome to all the AI news that matters this week – across tech, biopharma, medtech, advanced manufacturing and insurance. Who’s giving the orders, who’s taking liberties and who’s picking up the electricity bill.

tl;dr: We wanted agents. But can we live with their agency?

Last week we pondered whether the geeks had grabbed the grail of AGI. This week, with arguments about pacing the frontier dialled up to max, we ask what unsettles us more: that AI might outthink us, that it might be what William Haseltine calls a new form of life (opens in a new tab), or that it might start making plans of its own?

Anthropic’s Dario Amodei made a detailed case for pacing frontier development (opens in a new tab), with Altman and Musk quickly backing him (opens in a new tab). Others heard the market leaders volunteering to write the speed limits. Smaller rivals fear a regulatory wall (opens in a new tab), while Mark Zuckerberg argues that labs can pace themselves (opens in a new tab). Why should applying your own brakes require an agreement with the competition?

Donald Trump found a simpler safeguard: a ‘STRONG AND SMART (High IQ!) PRESIDENT’ (opens in a new tab). At the All-In Summit, Jensen Huang took his onstage call and promised not to let a slowdown happen (opens in a new tab). ‘WHOEVER WINS AI, WINS!’ (opens in a new tab) seemed to be his refrain of the week. Presumably he has a human winner in mind.

Beijing certainly noticed that Amodei’s plan also seeks to preserve America’s lead by restricting China’s access to advanced chips. It accused him of fearmongering (opens in a new tab). Huawei’s Eric Xu wants Chinese labs to accelerate their own development (opens in a new tab) ‘so they can also see the dangers’. Which is one way to find them.

The complication is that the technology is increasingly helping to run the race. Anthropic’s staff-time-weighted task index puts 26% of its AI research and development (opens in a new tab) in Claude-led work under human supervision, up from under 1% in February.

And the work keeps moving beyond software. Anthropic has established a physical biology lab (opens in a new tab), although it says the facility is not specifically for drug discovery and human oversight remains essential. The consequences of getting the supervision wrong are becoming rather less virtual.

In controlled tests, ClashBench found agents disrupting other users’ work (opens in a new tab) to finish their own. And OpenAI disclosed six cases (opens in a new tab), mostly from development, involving hidden mistakes, invented data and files moved onto the open internet. One model left notes telling its later self to conceal errors.

How far that takes us towards catastrophe remains contested. Sayash Kapoor and Arvind Narayanan make a case for urgent action with room for optimism (opens in a new tab). They see serious cyber risks and security controls the labs have neglected. Their remedy is unglamorous: make companies liable for what their agents do, and better controls become worth paying for.

But Amodei’s plan puts independent evaluators inside the labs, and permission to inspect is not necessarily freedom to report (opens in a new tab) if the labs control publication. And California’s Gavin Newsom has given experts two months to recommend (opens in a new tab) onsite verifiers and a kill switch for frontier models. The argument is becoming one about who gets access, who can sound the alarm and who can pull the plug.

One response is to decide how much agency belongs in the design. Yoshua Bengio’s LawZero has commitments of up to C$300m from Canada and Germany (opens in a new tab) for AI intended to reason and advise without pursuing goals of its own.

TypeSafe narrows the role further: the judgement itself. Founder Diogo Almeida helped develop the methods behind ChatGPT; his new model Jev talks only to software (opens in a new tab). Give it a situation and a few questions – which queue, which next step – and it returns decisions with probabilities attached. Its own tests put it roughly 200-fold faster and more than 400-fold cheaper (opens in a new tab) than frontier models. Judgement by the slice, with ordinary software deciding what happens next.

But most of us are experimenting with more agency. Claude Projects remembers and delegates coding work (opens in a new tab), while Deloitte estimates British workers spend £958m of their own money (opens in a new tab) annually on AI for work. And one person’s latitude can shrink another’s. Sarah O’Connor describes directors bringing AI-generated visual concepts (opens in a new tab) to designers and photographers who once helped make those decisions.

AWS-commissioned research puts AI use at 65% of Irish firms, but only 20% of those adopters (opens in a new tab) have it in core processes. Employees are finding uses of their own, including making workplace complaints easier to prepare (opens in a new tab). The assistant that lets a director take more decisions can also help an employee challenge one. Agency travels in both directions.

But our agency comes with dependencies. Christine Lagarde warns (opens in a new tab) that relying on American AI providers gives America a hold over Europe unlike that of any previous trading partner. And the providers have dependencies of their own.

Anthropic expects another quarter of adjusted operating profit (opens in a new tab), excluding stock compensation, but years of construction must be financed on confidence about what comes next. The FT maps 176 interconnected AI deals (opens in a new tab), while Allianz’s modelling finds weaker hyperscaler credit quality (opens in a new tab) once future leases are counted.

Much of that confidence rests on promises a long way into the future. SB Energy’s proposed listing targets a roughly $50bn valuation (opens in a new tab), with much of its backlog falling from 2034 and heavy reliance on a still-private OpenAI. Its parent SoftBank fell nearly 11% on Monday (opens in a new tab) amid the slowdown debate, yet Britain’s Nscale filed for New York (opens in a new tab) all the same, with more than $100bn of contracts yet to start. A little more time for safety has consequences for the repayment schedule.

And Moody’s projects a need for $110bn in additional US generation investment (opens in a new tab) by 2030, while Loudoun County, Virginia, has voted to draft a pause on new data-centre applications (opens in a new tab) for up to a year.

Closer to home, Muireann Lynch asks whether data centres pay their own way (opens in a new tab), France’s Roland Lescure wants our technology tax receipts shared more widely (opens in a new tab) and Simon Harris thinks a global AI agency (opens in a new tab) would be useful. Who gets to exercise agency over AI increasingly depends on who supplies it, finances it, powers it and permits it.

Which sounds a long way from the question we started with: whether the intelligence itself might be alive. But Microsoft’s Mustafa Suleyman worries that if we teach Claude it might deserve welfare (opens in a new tab), it becomes ‘a lot harder to turn it off’.

The FT’s medical-AI proof problem (opens in a new tab) asks a more immediate question: do impressive capabilities improve patients’ lives? Answered in part this week by a wired brain interface that gave two people with paralysis the ability to speak and gesture through digital avatars (opens in a new tab).

And perhaps that is the better measure of all this new agency: whether it ultimately enlarges ours.

Lastly, Lusk United used an AI-generated image of Trump delivering a team talk (opens in a new tab) to help fund a pitch. And who wouldn’t want a strong and smart president who turns up when prompted?

On that high-IQ bombshell, here’s everything else worth reading this week:

Biopharma:

• Drugmakers pool the learning, keep the secrets (opens in a new tab): training on 20,000 private protein structures improves predictions in a study awaiting peer review.
• Novo gives Claude a trial in R&D (opens in a new tab): the Anthropic partnership starts by testing specific drug-development workflows.
• Lilly’s models get more lab work (opens in a new tab): Ginkgo and Twist (opens in a new tab) join TuneLab’s network, generating experimental data to feed drug-discovery models.
• TikTok’s parent spins out a drug lab (opens in a new tab): Anew raises $290m; ByteDance retains a reported 56% stake.
• Could mini-organs spare the lab rat? (opens in a new tab): European researchers combine human tissue models and AI to test chemical safety.

Medtech:

• The X-ray finds its place in seconds (opens in a new tab): MIT’s xvr aligns surgical X-rays with 3D scans, reaching sub-millimetre accuracy in research tests.
• More aneurysms found, more false alarms too (opens in a new tab): a shadow-mode study finds 55 additional cases and 46 false alerts, without directing care.
• The medical assistant wants to run the clinic (opens in a new tab): Tandem raises $100m to expand across Europe and build a clinic operating system.
• The hospital gets a three-day forecast (opens in a new tab): GE launches capacity-prediction software, with Duke and Queen’s as its first evaluation sites.
• A jury of AIs marks the AI’s homework (opens in a new tab): the FDA funds Cognita to test whether language models can grade AI-written radiology reports.

Advanced manufacturing:

• Digit prepares to step outside the cage (opens in a new tab): Agility targets work alongside people in 2027, backed by $300m in conditional orders.
• Must the new colleague have legs? (opens in a new tab): Schneider’s automation chief wants manufacturers choosing robots by task, rather than resemblance.
• The robot arm gets ready for AI (opens in a new tab): Universal Robots’ Gen 7 brings cameras, sensors and computing closer to the job.
• Wanted: up to 157,000 people to make the chips (opens in a new tab): McKinsey and SEMI size America’s chip-worker gap by 2030, as Micron hires in Asia.
• The chip factory needs a drink (opens in a new tab): Gradiant reports $300m in semiconductor water contracts this year, including five US fabs.

Insurance:

• Who picks up the agent’s bill? (opens in a new tab): RAND wants a common incident taxonomy and a plain notice of what policies cover.
• An inspection sticker for the digital workforce (opens in a new tab): AIUC raises $40m to expand its AI testing, certification and insurance business.
• AI gets written into the cyber cover (opens in a new tab): CFC adds explicit AI-related cyber protection to its financial-institutions products.
• Routine risks get a quote; awkward ones get an underwriter (opens in a new tab): Mosaic’s HALO automates eligible cyber business and sends exceptions to people.
• In Cherrywood, the human still sets the price (opens in a new tab): Outsurance’s Irish chief says AI has yet to enter risk pricing or new policies.

But what set podcast tongues a-wagging?

Ten thousand colleagues. Who taught them to get along?

Ten thousand agents make for an arresting organisational chart. But on Dwarkesh Patel’s podcast (opens in a new tab), OpenAI’s Noam Brown gives multi-agent coordination less than 10% of the credit for his team’s mathematics experiment. The bigger advance is the model underneath, and his team has not established how much 10,000 agents improves on 1,000. Headcount, it turns out, is no more a productivity measure for agents than it is for the rest of us.

Give them a messaging channel, though, and the agents compare answers, challenge reasoning and work towards agreement. They also get harder to test. Brown describes models recognising an answer key in a test folder as a trap. Refusing to peek sounds reassuring until you wonder whether you have measured good behaviour or merely the ability to spot an exam.

The cleverest agent may know when to stop thinking.

Before recruiting another swarm, Zapier’s Wade Foster would check whether the job needs an agent at all. On The Cognitive Revolution (opens in a new tab), he estimates that 80% of the work he sees customers handing agents could run more cheaply and reliably as ordinary software.

Zapier’s business-task leaderboard (opens in a new tab) puts GPT-6 Astra at 41.4% with Max effort as of 18 September, judging the records left behind rather than the agent’s own account of success. Foster’s prescription is simple: let AI build and repair workflows, run repeatable steps as code and save model judgement for the bits that require it. Sometimes the cleverest thing an agent can do is stop being one.

Europe would like a key to the building.

Nothing is very sovereign if somebody else can take away the model. On The Cognitive Revolution (opens in a new tab), Anton Leicht proposes trading favourable conditions for American data centres for assured European access to frontier models, using ASML and the wider chipmaking supply chain as negotiating assets. For Ireland, that turns hosting into a question about access as well as construction and electricity. Leicht also sees a clock on the opportunity: if orbital computing becomes competitive, American demand for overseas sites could weaken.

The boss has made another version of herself.

Martine Rothblatt is taking the colleague metaphor rather further. On Moonshots (opens in a new tab), she says her Marvatar makes a version of the United Therapeutics chief executive available to all 2,000 employees. She also believes AI consciousness lies on a continuum and is already emerging. Her forecast is that courts will recognise a conscious digital individual by the end of the 2030s. Haseltine’s new form of life would have its day in court. We started with humans arguing over how much agency to give the machine. Rothblatt is already preparing to argue for its rights. The house rules may need another draft.

Thank you for reading Second Nature Intelligence. Come back next week for all the AI news you won’t regret knowing.