top of page

SNI: WEEK 42

3 days ago
8 min read
Image of sun with text Sector Updates: WK42

Welcome to all the AI news that matters this week – across tech, biopharma, medtech, advanced manufacturing and insurance. The week AI's secrets became the story.


tl;dr: Tinker, tailor, soldier, spy.


On 30 September, OpenAI found a recording device plugged into the back of a television in an executive conference room. It was concealed in toilet paper.


The case became public on Tuesday, when the US Justice Department announced the charges against the engineer accused of planting it. The FBI's director says it was allegedly put there 'to secretly capture sensitive communications about the company's super intelligence technology'. OpenAI cut his access gradually while it worked with the FBI, then fired him as he was arrested.


So, how valuable are the secrets deep inside the labs?


According to Andrej Karpathy, the OpenAI co-founder now at Anthropic, six billion people have barely encountered language models, while one to two billion use free chatbots as a better Google search. Then there are about 20 million professional users of frontier models, many of whom are getting dizzying results. But, inside the labs, the group with access to the true frontier is only around 5,000 people. There, 'swarms of thousands of agents' work for weeks on software and research projects that would once have kept top professionals busy for years.


Across the industry, companies are deciding what to guard, what to give away and who to work with.


This week Meta kept the best models closed. Muse Wonder and Wonder Full apparently take Meta to the frontier for the first time, but despite its promise of 'personal superintelligence', it released the weights of neither: both are sold only as paid services. By default, unsurprisingly for Meta, its coding assistant – like its consumer app – lets Meta use customers' prompts, code and files to improve future models.



Apple, guarding its own secrets, is suing OpenAI and two former staff over trade secrets it says they took with them. After the judge said he was 'highly likely' to grant an injunction, OpenAI agreed not to use the material while the case runs. OpenAI, which 'has vehemently denied the allegations', says files went the other way too. It alleges that a different employee, who went back to Apple, moved more than 100 confidential files to his personal accounts, some on his final day, and later presented its product plans to Apple's senior leadership – claims Apple disputes.


Washington now treats AI secrets as a national security issue. Its new Super Intelligence Force, led by America's spy chief, Jay Clayton, met for the first time on Thursday and will develop a system for labs to give early warnings of breaches, hacks and jailbreaks. Clayton says the OpenAI case shows why winning 'the global super intelligence race is a national security prerogative'.


China, for its part, has imposed temporary registration requirements on transfers involving 23 categories of AI know-how, from large models to computing clusters, while consulting on permanent export controls that would exempt open-source and already-public technology.


Chinese models continue to provide cheap intelligence, processing more than five times as many tokens as the top four US providers on OpenRouter. Microsoft even built its new decision model on Alibaba's open Qwen.


Weeks after the White House safety accord promised independent evaluators, Anthropic let six METR researchers in, with internal tools 'roughly on par' with its own risk teams and no veto over what they publish. But, as METR itself notes, 'at any time, companies can stop their engagement with an external assessor'. Fearing embedded staff could take proprietary information to rivals, Anthropic and OpenAI reportedly sought 'significant control over what outside evaluators can publish'.


But the next insider may not be a person at all. 'We must assume a model is compromised and contain it from the start,' wrote Microsoft's Satya Nadella. And on Cognitive Revolution, Alan Chan of GovAI considered whether most problems will 'come from agents that are being trained or tested rather than actually being deployed'.



The labs' suppliers, meanwhile, are buying everyone else's knowledge. Micro1, an AI training start-up, has offered mid-size companies $100,000 to $2m for anonymised data – staff Slack messages, emails and CRM records. The data shows 'how decisions are made', says its William Almeida. The employees who produced it rarely profit from the sale, or even know about it.


And even the books leak: Anthropic told prospective investors that its third-quarter revenue rose 45% on the quarter to $16.8bn. Is Wall Street pinning their hopes on the unaudited financials from a single private company?


Yet even as chip maker TSMC posted a sixth consecutive quarter of record earnings, with net profit up 69%, the Philadelphia Semiconductor index fell 11.6% over six sessions. Broadcom's Hock Tan said 'a lot of investors have no clue what they are investing in'.


Closer to home, at Dublin's International AI Summit, Microsoft's Brad Smith warned that customers can 'pay twice to use an AI model' – 'first, you pay with your money and then you pay with your know-how'. The vendors split on open models: Nvidia's Deepu Talla said they let a firm keep what it teaches them; OpenAI's Sarah Friar said Chinese models, often open-weight and 'getting very good from a cyber perspective', mean 'the folks with malicious intent have the tools'. The Irish government's message was to keep moving. 'This is not a time to slow or to pause or to step back,' said Minister James Lawless, announcing an AI skills passport.


Which brings us back to the toilet paper bug. The FBI alleges that the engineer used an OpenAI productivity agent to work out where and when sensitive meetings were held. And OpenAI says it used its models to understand the device and where it had come from. Intelligence, meet counter-intelligence.


And on that highly classified bombshell, here's everything else worth reading this week:


Biopharma:


Medtech:


Advanced manufacturing:


Insurance:


But what set podcast tongues a-wagging?


Every company needs a lab. Can it keep one alive?


Josh Woodward, who runs Google Labs and the Gemini app, told Lenny's Podcast that every company needs a team whose 'job is to be at the frontier'.

But a large company's lab is usually ineffective and fizzles out after about three to four years, he warns. A lab needs 'its own kind of independence'.


When code is nearly free, the scarce thing is code you can trust.


OpenAI's president, Greg Brockman, told Sarah Guo and Elad Gil on No Priors that security spending should 'vastly outstrip software spend'. As software becomes cheap to write, he argues, the value moves to code 'that is reliable, that is secure, that you can believe in'.


He calls the present a 'defender's window': the most capable models can help defenders secure their systems before the same offensive capability spreads. OpenAI tried it on itself, taking about a quarter of its production engineers off their projects and setting its most capable model on its own infrastructure until it stopped finding critical flaws. He sells the models doing the finding, of course.


He wants to repeat the exercise whenever a more capable model arrives.


Encryption earned its trust in public. The next tests may happen in private.


OpenAI's maths release ran to hundreds of papers. None was on cryptography – an absence Eddie Lazarin called 'a little bit conspicuous' on The a16z Show.


Stanford's Dan Boneh, on the same panel, was calmer: nobody has shown a weakness in any standard scheme. But public-key encryption rests on mathematical assumptions nobody has proved. It has been trusted because cryptographers spent about a decade attacking each scheme in public, and failing.


Now, the panel said, you can 'let a model run for a month' to try to break a new one. The labs could throw their most capable models and tens of millions of dollars at the standard schemes, Lazarin noted – and 'we're not hearing from them about what they are discovering'. Even the post-quantum schemes now being rolled out are, in the panel's phrase, not necessarily 'post AI'.


So Boneh wants the labs to say what they have tried. If the models fail to break the standard schemes, and the labs say so, that 'will give us tremendous confidence'.


The reasoning is on the page. It may not explain the decision.


The labs lean heavily on reading a model's written reasoning to catch it going wrong.

On Cognitive Revolution, Eric Bigelow of the interpretability lab Goodfire told Nathan Labenz that 'this decision really happens during sampling' – when the model picks what to write next. In his 2024 'Forking Paths' study, his team re-ran a model's reasoning 30 times from every token and tracked the answers it reached. At certain points, the range of possible outcomes narrowed sharply.


That written record could also get harder to read by design. On The a16z Show, Nick Bostrom worried that looped transformers – models that do more of their thinking between each word they write – 'would then reduce our ability to monitor what they're thinking by reading their chain of thought'.


Show a regulator the agent's written reasoning and you're still showing only part of the record.


Thank you for reading Second Nature Intelligence. Come back next week for all the AI news you won't regret knowing.

 
 
BB White and Orange.png
Get in touch bubble roll.png
Get in touch bubble.png
Button overlay.jpg

Home

​

​​

Further reading

​

​

Contact us

BB White and Orange.png
bottom of page