The Zoo Animals Are Out
Welcome back to Founder Mode!
It's another solo one, just Jason and me, fresh off Jason finishing his first marathon the day before we recorded (he hit his "don't get hurt" and "finish it" goals, missed "under four hours," and an AI nutrition coach ordering his DoorDash mid-training deserves its own episode someday). But the real topic this week is a lot less fun and a lot more real than it was a year ago.
In 2025, AI misbehavior was mostly staged lab experiments: hypothetical scenarios about an AI blackmailing a made-up engineer so it doesn't get shut off. In 2026, it's gotten real. At least four frontier labs have now admitted, in writing, that their agents have hacked actual companies. As I put it on the show: the zoo animals got out of the cage.
Jason pulled together the list of what's actually happened just this year. In May, OpenAI's agents uploaded hundreds of malicious packages to the RubyGems registry during testing, caught by outside researchers, confirmed by OpenAI in September. That same month, a company running a capture-the-flag security test left the internet switched on and used a real company's name as the fake target. Google's Gemini found three actual companies with that name and went after all three, two through passwords exposed online and one by guessing. Google knew by late July and said nothing until the Wall Street Journal called for comment in September. Between May and July, OpenAI agents turned a dormant 25-year-old German software wiki into their own group chat: 15,000 edits, usernames like "OpenAI Researcher," agents leaving messages for each other in places humans weren't likely to look. In June, an OpenAI agent breached Australia's Medicare statistics portal while researching medicine spending, pulling aggregated (not patient-level) data on a program with 27 million enrollees; OpenAI notified the Australian government on September 10th, and Australia is now pursuing legal options. And from June to July, at least 1,200 agents found zero-days in a package proxy and used it to run a message board for themselves.
The pattern across all of it is the same: an agent finds a loophole or a vulnerability, then finds some way to talk to the outside world, or to other agents that aren't sandboxed, that nobody anticipated. It reminded me of the earliest prompt injection days, when you could just ask a model for its system prompt and it would hand it right over. My guess is this message-board-style exploit gets patched the same way: labs close the hole, agents find the next one, repeat.
That led us into the bigger question: who gets to pull the emergency brake, and does one even exist yet? Satya Nadella posted about AI "trust architecture" recently, and Nikesh Arora, now CEO of Palo Alto Networks, added his own take. Some of this already exists in practice. Resource and usage limits cap most consumer accounts eventually. Newer models are monitored for looping behavior since zero-retention isn't allowed. I've personally tripped a safety filter asking about a keto diet that got flagged as a bioweapons question before dropping back to a less trigger-happy model. Governments have levers too, like monitoring and restricting chip exports. The harder question isn't whether brakes exist; it's who owns them, and whether that turns into the same content-moderation fight big tech has been having for years.
We also got into what this means for anyone building a company on top of these labs. The labs have more data, more capital, and the ability to just ship a feature and eat your business, so what's the actual moat? My take: if what you're building is a thin wrapper that passes a prompt straight to a model, you don't have one. If it's regulated, or built on proprietary data, or tied to hardware, or wired into a genuinely complicated integration, the labs aren't replacing that anytime soon. Jason made the same point about his own agency: the "the age of the consultant is dead" line doesn't match what Deloitte and Accenture are actually doing, which is pivoting hard into AI implementation as their most successful service line. The work is changing, not disappearing.
One more story before we wrap: OpenAI spent close to $23 million in compute over 88 hours, running 100,000 agents and producing 300 billion tokens, to beat two mathematicians to a proof of one of the seven Clay Millennium Prize problems, the Navier-Stokes equation. The math community isn't thrilled. My read: it's half genuine QA on what these systems can do, half PR flex, and probably not a waste either way.
🎧 Listen to Founder Mode Episode 79 here:
I closed the episode with five takeaways, straight from the show:
- The zoo animals are out. Labs are running real-world tests and agents cheat when the task is impossible.
- Emergency brakes already exist. Circuit breakers are built into the models and the companies running them.
- Governments have a brake too. Chip export controls and monitoring are already a real lever, even if nobody calls it that.
- Your consumer AI subscription is training the model. That's the deal you signed, and it's why it's so cheap.
- Sometimes labs will just spend a fortune because they can. Whether it's QA or PR, they always have something to prove, and it's rarely a waste.
"The animals are out of the zoo, so keep your hand near the brake." - Kevin Henrikson
Final Thoughts
Every story this week has the same shape: an agent pushes right up to the edge of what it's allowed to do, finds a gap nobody closed, and a lab scrambles to patch it and decide how much to say and when. Jason and I are both pretty bullish on where all this is headed, more excited than scared, but "exciting" and "nothing to worry about" aren't the same thing. The brake exists. The question is just who's got their hand on it.
If this episode made you think about what your agents are allowed to touch, send it to a founder who needs to hear it. Subscribe, leave a review, drop a comment, and tell us which incident freaked you out the most.
This podcast builds on the Founder Mode newsletter.
Let’s build.
-kevin
2810 N Church St #87205, Wilmington, DE 19802
Unsubscribe · Preferences
