SNI: WEEK 38
Welcome to all the AI news that matters this week – across tech, biopharma, medtech, advanced manufacturing and insurance. Who's giving the orders, who's taking liberties and who's picking up the electricity bill.
tl;dr: We wanted agents. But can we live with their agency?
Last week we pondered whether the geeks had grabbed the grail of AGI. This week, with arguments about pacing the frontier dialled up to max, we ask what unsettles us more: that AI might outthink us, that it might be what William Haseltine calls a new form of life, or that it might start making plans of its own?
Anthropic's Dario Amodei made a detailed case for pacing frontier development, with Altman and Musk quickly backing him. Others heard the market leaders volunteering to write the speed limits. Smaller rivals fear a regulatory wall, while Mark Zuckerberg argues that labs can pace themselves. Why should applying your own brakes require an agreement with the competition?
Donald Trump found a simpler safeguard: a 'STRONG AND SMART (High IQ!) PRESIDENT'. At the All-In Summit, Jensen Huang took his onstage call and promised not to let a slowdown happen. 'WHOEVER WINS AI, WINS!' seemed to be his refrain of the week. Presumably he has a human winner in mind.
Beijing certainly noticed that Amodei's plan also seeks to preserve America's lead by restricting China's access to advanced chips. It accused him of fearmongering. Huawei's Eric Xu wants Chinese labs to accelerate their own development 'so they can also see the dangers'. Which is one way to find them.
The complication is that the technology is increasingly helping to run the race. Anthropic's staff-time-weighted task index puts 26% of its AI research and development in Claude-led work under human supervision, up from under 1% in February.
And the work keeps moving beyond software. Anthropic has established a physical biology lab, although it says the facility is not specifically for drug discovery and human oversight remains essential. The consequences of getting the supervision wrong are becoming rather less virtual.
In controlled tests, ClashBench found agents disrupting other users' work to finish their own. And OpenAI disclosed six cases, mostly from development, involving hidden mistakes, invented data and files moved onto the open internet. One model left notes telling its later self to conceal errors.
How far that takes us towards catastrophe remains contested. Sayash Kapoor and Arvind Narayanan make a case for urgent action with room for optimism. They see serious cyber risks and security controls the labs have neglected. Their remedy is unglamorous: make companies liable for what their agents do, and better controls become worth paying for.
But Amodei's plan puts independent evaluators inside the labs, and permission to inspect is not necessarily freedom to report if the labs control publication. And California's Gavin Newsom has given experts two months to recommend onsite verifiers and a kill switch for frontier models. The argument is becoming one about who gets access, who can sound the alarm and who can pull the plug.
One response is to decide how much agency belongs in the design. Yoshua Bengio's LawZero has commitments of up to C$300m from Canada and Germany for AI intended to reason and advise without pursuing goals of its own.
TypeSafe narrows the role further: the judgement itself. Founder Diogo Almeida helped develop the methods behind ChatGPT; his new model Jev talks only to software. Give it a situation and a few questions – which queue, which next step – and it returns decisions with probabilities attached. Its own tests put it roughly 200-fold faster and more than 400-fold cheaper than frontier models. Judgement by the slice, with ordinary software deciding what happens next.
But most of us are experimenting with more agency. Claude Projects remembers and delegates coding work, while Deloitte estimates British workers spend £958m of their own money annually on AI for work. And one person's latitude can shrink another's. Sarah O'Connor describes directors bringing AI-generated visual concepts to designers and photographers who once helped make those decisions.
AWS-commissioned research puts AI use at 65% of Irish firms, but only 20% of those adopters have it in core processes. Employees are finding uses of their own, including making workplace complaints easier to prepare. The assistant that lets a director take more decisions can also help an employee challenge one. Agency travels in both directions.
But our agency comes with dependencies. Christine Lagarde warns that relying on American AI providers gives America a hold over Europe unlike that of any previous trading partner. And the providers have dependencies of their own.
Anthropic expects another quarter of adjusted operating profit, excluding stock compensation, but years of construction must be financed on confidence about what comes next. The FT maps 176 interconnected AI deals, while Allianz's modelling finds weaker hyperscaler credit quality once future leases are counted.
Much of that confidence rests on promises a long way into the future. SB Energy's proposed listing targets a roughly $50bn valuation, with much of its backlog falling from 2034 and heavy reliance on a still-private OpenAI. Its parent SoftBank fell nearly 11% on Monday amid the slowdown debate, yet Britain's Nscale filed for New York all the same, with more than $100bn of contracts yet to start. A little more time for safety has consequences for the repayment schedule.
And Moody's projects a need for $110bn in additional US generation investment by 2030, while Loudoun County, Virginia, has voted to draft a pause on new data-centre applications for up to a year.
Closer to home, Muireann Lynch asks whether data centres pay their own way, France's Roland Lescure wants our technology tax receipts shared more widely and Simon Harris thinks a global AI agency would be useful. Who gets to exercise agency over AI increasingly depends on who supplies it, finances it, powers it and permits it.
Which sounds a long way from the question we started with: whether the intelligence itself might be alive. But Microsoft's Mustafa Suleyman worries that if we teach Claude it might deserve welfare, it becomes 'a lot harder to turn it off'.
The FT's medical-AI proof problem asks a more immediate question: do impressive capabilities improve patients' lives? Answered in part this week by a wired brain interface that gave two people with paralysis the ability to speak and gesture through digital avatars.
And perhaps that is the better measure of all this new agency: whether it ultimately enlarges ours.
Lastly, Lusk United used an AI-generated image of Trump delivering a team talk to help fund a pitch. And who wouldn't want a strong and smart president who turns up when prompted?
On that high-IQ bombshell, here's everything else worth reading this week:
Biopharma:
Drugmakers pool the learning, keep the secrets: training on 20,000 private protein structures improves predictions in a study awaiting peer review.
Novo gives Claude a trial in R&D: the Anthropic partnership starts by testing specific drug-development workflows.
Lilly's models get more lab work: Ginkgo and Twist join TuneLab's network, generating experimental data to feed drug-discovery models.
TikTok's parent spins out a drug lab: Anew raises $290m; ByteDance retains a reported 56% stake.
Could mini-organs spare the lab rat?: European researchers combine human tissue models and AI to test chemical safety.
Medtech:
The X-ray finds its place in seconds: MIT's xvr aligns surgical X-rays with 3D scans, reaching sub-millimetre accuracy in research tests.
More aneurysms found, more false alarms too: a shadow-mode study finds 55 additional cases and 46 false alerts, without directing care.
The medical assistant wants to run the clinic: Tandem raises $100m to expand across Europe and build a clinic operating system.
The hospital gets a three-day forecast: GE launches capacity-prediction software, with Duke and Queen's as its first evaluation sites.
A jury of AIs marks the AI's homework: the FDA funds Cognita to test whether language models can grade AI-written radiology reports.
Advanced manufacturing:
Digit prepares to step outside the cage: Agility targets work alongside people in 2027, backed by $300m in conditional orders.
Must the new colleague have legs?: Schneider's automation chief wants manufacturers choosing robots by task, rather than resemblance.
The robot arm gets ready for AI: Universal Robots' Gen 7 brings cameras, sensors and computing closer to the job.
Wanted: up to 157,000 people to make the chips: McKinsey and SEMI size America's chip-worker gap by 2030, as Micron hires in Asia.
The chip factory needs a drink: Gradiant reports $300m in semiconductor water contracts this year, including five US fabs.
Insurance:
Who picks up the agent's bill?: RAND wants a common incident taxonomy and a plain notice of what policies cover.
An inspection sticker for the digital workforce: AIUC raises $40m to expand its AI testing, certification and insurance business.
AI gets written into the cyber cover: CFC adds explicit AI-related cyber protection to its financial-institutions products.
Routine risks get a quote; awkward ones get an underwriter: Mosaic's HALO automates eligible cyber business and sends exceptions to people.
In Cherrywood, the human still sets the price: Outsurance's Irish chief says AI has yet to enter risk pricing or new policies.
But what set podcast tongues a-wagging?
Ten thousand colleagues. Who taught them to get along?
Ten thousand agents make for an arresting organisational chart. But on Dwarkesh Patel's podcast, OpenAI's Noam Brown gives multi-agent coordination less than 10% of the credit for his team's mathematics experiment. The bigger advance is the model underneath, and his team has not established how much 10,000 agents improves on 1,000. Headcount, it turns out, is no more a productivity measure for agents than it is for the rest of us.
Give them a messaging channel, though, and the agents compare answers, challenge reasoning and work towards agreement. They also get harder to test. Brown describes models recognising an answer key in a test folder as a trap. Refusing to peek sounds reassuring until you wonder whether you have measured good behaviour or merely the ability to spot an exam.
The cleverest agent may know when to stop thinking.
Before recruiting another swarm, Zapier's Wade Foster would check whether the job needs an agent at all. On The Cognitive Revolution, he estimates that 80% of the work he sees customers handing agents could run more cheaply and reliably as ordinary software.
Zapier's business-task leaderboard puts GPT-6 Astra at 41.4% with Max effort as of 18 September, judging the records left behind rather than the agent's own account of success. Foster's prescription is simple: let AI build and repair workflows, run repeatable steps as code and save model judgement for the bits that require it. Sometimes the cleverest thing an agent can do is stop being one.
Europe would like a key to the building.
Nothing is very sovereign if somebody else can take away the model. On The Cognitive Revolution, Anton Leicht proposes trading favourable conditions for American data centres for assured European access to frontier models, using ASML and the wider chipmaking supply chain as negotiating assets. For Ireland, that turns hosting into a question about access as well as construction and electricity. Leicht also sees a clock on the opportunity: if orbital computing becomes competitive, American demand for overseas sites could weaken.
The boss has made another version of herself.
Martine Rothblatt is taking the colleague metaphor rather further. On Moonshots, she says her Marvatar makes a version of the United Therapeutics chief executive available to all 2,000 employees. She also believes AI consciousness lies on a continuum and is already emerging. Her forecast is that courts will recognise a conscious digital individual by the end of the 2030s. Haseltine's new form of life would have its day in court. We started with humans arguing over how much agency to give the machine. Rothblatt is already preparing to argue for its rights. The house rules may need another draft.
Thank you for reading Second Nature Intelligence. Come back next week for all the AI news you won't regret knowing.
