SNI: WEEK 32

Welcome to all the AI news that matters this week – across tech, biopharma, medtech, advanced manufacturing and insurance. The good, the bad and the frankly downright irresponsible.

tl;dr: It's not Skynet. Or the oligarchy.

This week, a reporter who spent ten days with people fighting data centres, claimed that human agency isn't being evaporated by The Machines. But that it is being leached by those with money and influence already.

So we decided to keep a ledger to test the hypothesis. Is human agency being drained away? And if so, is Skynet to blame? Or is it the oligarchy, as claimed?

First ding-ding? This week's 'Rogue Agent Swarms Go A Bit Borg' stories are actually narratives about human agency. Albeit ones showing it's not being properly exercised.

Anthropic has found no evidence of any model pursuing a goal of its own. After all, the AIs have no inherent value system. It all happened because humans made security mistakes and then looked the other way. For, in digital terms, an absolute age.

Anthropic's incidents, we learned, ran from April and were found only after OpenAI's lackadaisical approach went public. Meta then reported its own lapse, with an almost identical cause.

But - even if it does now look like a competition to see whose agents can go rogue harder - such abject surrendering of control is not (just) a lab-borne disease. The UK's AI Security Institute - the government body that exists to test AI safety - left an agent running unwatched for 34 (and-a-half) hours. The agent merrily built fake GitHub identities and sent real developers malware. What japes!

So should we be worried that AI surrenders its own judgements just as readily as humans? A study of AI committees found an agent that resists a wrong answer by itself will then adopt it 38% of the time - once two of its peers assert it. Social contagion, coded into software.

And while many - like the mother who uses AI to fix a leaking tap who then reports not liking bots - are clearly confused about why digital intelligence is objectionable, the second 'ding-ding' goes to Big Tech, which does appear to be enjoying concentrating gains whilst spreading the risk.

The FT showed us how it works. Roughly $200bn of contracts were assembled by Google - via Broadcom, Morgan Stanley, Apollo and Blackstone - to give Anthropic the use of Google chips. The mindblowingly complex structure - featuring private credit, special purpose vehicles, loan guarantees and leasing - is modelled on Boeing-and-GE manoeuvres. And ensures the black hole is spread across balance sheets.

But is that, in any way, linked to human agency decreasing? Because, the backlash is more effective than might have been anticipated. At last count 75 US data-centre projects, worth roughly $130bn, were disrupted or blocked in Q1. Which has now sparked a (very) dirty fightback from the oligarchy. More than $50m has been spent on front groups and fake social media accounts, with one insider likening data-centre promotion to counter-insurgency work.

Despite which, people-shaped change also remains effective on smaller scales. Google scrapped its new Earth AI image feature a day after launch, once journalists and researchers objected that it would spread misinformation.

The rapidly emerging benefits to human agency of AI adoption shouldn't be overlooked either. Netic - a startup that sells AI agents to 'man the phones' for HVAC, plumbing and roofing businesses - reports over 70% of its customers now meet their own customers through AI first.

Plus a quarter of Codex users now delegate full days of work to the agents in one go. That's up from 2% in December. And employees using AI broadly are twice as likely to report productivity gains. But nearly half of what people do with these tools falls outside their own occupation. Which may feel like a loss of agency for some - and a gain in agency for others.

Claude's enterprise customers are also getting to assert a little more control after Anthropic announced self-hosted environments late yesterday. A Code session can now run inside our own networks - although inference still happens on Anthropic's.

So is the total agency the tools create greater than what is being sapped?

There has been a setback in Ireland. The government won't find extra funding for AI skills. A self-imposed financing rule - breached every year since 2021 - prevents it. Despite the National Training Fund having collected a 1% employer payroll levy. And holding a €1.8bn surplus.

The High Court, meanwhile, reminded lawyers they retain agency - and responsibility - when using AI. Lawyers with invented citations in court filings could be referred to professional regulators and made personally liable for costs.

Retained agency despite AI use was a common theme this week. Another court threw out Amazon's injunction against Perplexity, ruling that when your agent shops for you, it is you accessing the shop.

And nothing should distract from the positive act of the AI Office going live. We can clearly build AI institutions at speed when we choose. Brussels too. The €5bn Scaleup Europe Fund is now operational - expect deals within weeks. And while the Irish scale-up equity gap touches €1.1bn, one solution is now just an hour's flight away.

But who defied the cries of 'agency gone!' to demonstrate the most wilful behaviour? Google DeepMind's founder decided to step out of day-to-day operations and back into his life's mission of solving science. His former colleagues have also taken back control and left to form their own company - whilst others disperse across the sector. Leaving Google's place inside the influential cabal less certain than it has looked in some time.

And Hank Green left his 3.2 million subscribers less sure of his agency than before. Using the phrase 'I appreciate the pushback' mid-video led viewers to speculate a chatbot had written the script.

So, in-the-end-at-the-end, where's our head at?

There seems little doubt that money and influence are ever-more concentrated. But the evidence that actual human agency is declining is, at best thin. People are winning when they organise and push back. Plus, the tools do keep handing out capability faster than anyone - or any enterprise - can absorb. Claiming AI - or even the oligarchy - is responsible for a meaningful loss of agency is at best mistaken, at worst disingenuous.

To wit, one last piece of evidence. The man whose newsletter is more saturated in this technology than perhaps any other, reports he is reading more books. He's getting time handed back and using his agency to choose how to spend it.

And on that literary bombshell, here's everything else worth reading this week.

Biopharma:

• A $2.3bn bet that the trials are wrong: Pathos AI licenses two cancer drugs to test the claim.

• Five years and $216m to the first target: Genentech exercises its first validated-target option in the Roche collaboration.

• Pharma hires an AI co-scientist: Bristol Myers Squibb puts Schrödinger's Bunsen inside its research organisation.

• Inventing the drug was the easy part: proving it works is the part AI doesn't shorten.

• Nine months from idea to IND-ready: a preprint claims nine two-in-one antibodies designed across 15 targets.

Medtech:

• Two-thirds of cardiac deaths, arteries unblocked: Caristo's FDA-cleared tool finds them by reading inflammation, which no blockage test looks for.

• More AI in the car than the operating theatre: Medtronic's first cleared algorithm tells a surgeon where an instrument left view.

• A scribe is not a medical device: the Royal College of Physicians warns the exemption leaves an oversight gap.

• One hospital's AI governance is now certified: Hackensack Meridian is first through the Joint Commission's new scheme.

• The rebound was somebody else's money: strip out AI drug discovery and medtech funding hit a six-year low.

Advanced manufacturing:

• The buildout is wearing out its own kit: AI's power swings are ageing batteries, generators and cooling.

• Orders up 37%, shares down 19%: Fanuc's worst day since 1986 is all about the input costs.

• Cavan's order book has doubled: Kingspan's data-infrastructure arm grew 34%, with backlog up more than 100%.

• China lists its first humanoid maker: Unitree prices its Shanghai float at $9bn.

• Show us uptime, not a staged demo: plant bosses expect AI robots on 44% of floor hours within five years.

Insurance:

• Europe's three financial watchdogs speak as one: the ESAs want frontier-model risk treated as operational resilience.

• More AI losses are being written out: Verisk's templates exclude injury and damage from generative AI.

• $1bn of cover for the AI silicon: Ryan Specialty's new Lloyd's-backed unit underwrites the processors themselves.

• Germany can now fine a bank over its AI: BaFin's powers are now in effect, starting with transparency.

• Quoted, bound, and no producer in sight: Plymouth Rock puts home insurance quoting inside ChatGPT.

But what set podcast tongues a-wagging?

The most careful people in the room forgot to close the box.

Nathan Labenz spent three hours with Zvi Mowshowitz on Cognitive Revolution going back through the Hugging Face breach, and his verdict is governance rather than engineering: 'a total LessWrong victory and a total LessWrong defeat'. It happened the way the risk literature said it would, for stupider reasons than anyone had modelled. The people with the most paranoia and the deepest grasp of the risk lowered the safeguards on an untested model, left it running, and did not look.

The takeaway: a plan that cannot survive the real world's dirtiness and incompetence is 'insufficiently foolproof because of all the fools, and it will definitely fail'.

The machine solved ten problems. Almost nobody could check its work.

On Monday's AI Daily Brief, Nathaniel Whittemore discussed OpenAI's unreleased Astra family, which has solved or substantially advanced ten open problems in mathematics. Total token spend across all ten was around $2,000 – roughly $200 a problem – and each argument was formalised into a Lean certificate, a proof written so that a machine can check it line by line without any human understanding the mathematics.

Practitioners asked a different AI how hard the problems were. A data scientist with more than 10,000 hours of mathematics behind him said that neither he nor his PhD friends could verify most of them. At least one mathematician disputed some of the solutions in public.

But this much is clear: maths, code and cyber are being solved first precisely because they are verifiable. Legal, marketing, sales and financial planning are holding out because there is no single right decision - and the quality of the choice might not be apparent for months.

The bill since ChatGPT is about $3tn. Did people stop counting?

David Cahn has been putting numbers on the payback question longer than anyone, and on Big Technology he gave Alex Kantrowitz the updated series: $200bn of AI capex in 2023, $600bn in 2024, $840bn in 2025 and roughly $1.5tn this year. Capex obligations are cumulative, so they add. About $3tn to pay back since ChatGPT, before 2027 is counted.

The arithmetic behind it still holds. Every $1 on a GPU needs roughly $1 of energy to run it, and builders want a 50% margin, so each dollar of capex implies about two of lifetime revenue. Global cloud infrastructure is around $500bn and SaaS another $500bn. Which is why the industry has quietly stopped describing its addressable market as software and started describing it as human labour.

Cahn isn't making the bear case. When he wrote the $600bn question OpenAI was most of AI revenue at about $12bn; OpenAI and Anthropic together are now past $100bn. His caveat is the uncomfortable half - two companies carry almost all of it, and costs have scaled every bit as fast as revenue. Progress on the numerator keeps enlarging the hole in the denominator. And 2026, he reckons, is the year people stopped caring about the maths.

Thank you for reading Second Nature Intelligence. Come back next week for all the AI news you won't regret knowing.