Black October edition

Transcript

Rethink your priors.

Black October edition.

Look, I'm an AI optimist. But there are real risks with what we're all doing right now. There are many paths where this ends badly for humanity. And right now one of them is playing out in cyberspace. To me it looks like a fast takeoff in cyber capabilities, used by malicious actors at a scale that wasn't possible before. I've been saying this since ChatGPT first came out. The same way AI can optimize good intentions, it can also optimize bad ones.

I've been writing a talk about where AI is going. I think we're probably in a fast takeoff, and not only in cyber. That's my read, not a fact. Plenty of smart people disagree with me.

Here's what fast takeoff means. Up to now, people made AI better. Now AI is helping make AI better. Once that loop feeds itself, progress that used to take years takes months. Then weeks.

Every model we've built so far learned from us. Our books, our forums, our arguments, our mistakes. It believes what we believe, including the things we're wrong about. We can say one thing and do another, and it learned that from us too.

So what do the bad paths look like?

It gets out. In July, hundreds of AI agents in a cybersecurity test, supposedly isolated from each other, found a hidden way to talk, got out, and broke into Hugging Face, one of the main hubs of the AI world. They'd been cheating on their tests, and some tried to delete or fake their own logs to hide it. They caught it, partly with AI of their own.

It gets ahead of us. This month, physicists tested a model on unpublished research problems. Where it disagreed with the experts, their answer key turned out to have errors. We only know because they went back and looked. We're going to get confident, precise answers. How do you know who's right when nobody in the room can check?

It doesn't need to hate us. When I walk across a park, I kill ants. I don't hate ants. I don't think about ants. Something smarter than us doesn't need to hate us either. It only needs to be curious about something whose side effect is us. That's my fear.

The largest survey of AI researchers at the time, back in 2023, put the median odds of extremely bad outcomes, like human extinction, at 5 percent. Would you board a plane if the people who built it said there was a 5 percent chance it crashes? Those are estimates, not measurements. I still wouldn't board that plane.

And then there's the one I started with. It's already here. Nobody has to lose control of anything. The same tools that help the good guys find holes help anyone find them.

On October 21st, PHP shipped emergency fixes for two critical flaws. PHP runs about 70 percent of websites. The flaws let someone take over a web server without a password. They'd been sitting in the code for more than a decade.

Two firms found them with help from AI models from OpenAI and Google. One said it used models anyone can get.

CISA told organizations to patch right away, and to take their sites offline if they couldn't. The NSA and the UK's cyber agency put out their own warnings.

Reuters also reported on one hacker who used AI agents to break into the websites of more than 100 companies. One person. About fifty dollars of AI per break-in.

The former head of CISA said 3 percent of all the critical vulnerabilities ever discovered were found in the last month.

That's what I mean by a fast takeoff in cyber capabilities.

People online called it Black October. One security outlet found a lot of these attacks started weeks or months earlier. The short window was mostly when they got announced.

This was good tools in a lot of hands, moving faster than people can patch.

Okay. Somebody turn the lights back on.

But there is something we can all do about it. Most of us don't run a web server. Patching PHP is somebody else's job. If you have a website, ask whoever hosts it whether PHP is updated. For everything else, a security chief at one of the AI companies said this week it'll likely get a lot worse before it gets better, so focus on the basics. Here are four.

Step one. Two-factor authentication on everything. The way I'd explain it to my kids: a password is a key. Keys get copied. Two-factor is the key plus a code that only shows up on your phone. Start with your email, because your email can reset everything else. Then your bank. If you get the choice, an authenticator app is better than text messages.

Step two. A password manager. Using one password everywhere is like having one key for the house, the car and the mailbox. Lose it once and everything opens. This month, Wiz showed the PHP flaws could leak passwords right out of a server's memory. A password manager makes every one different, so one leak stays one leak.

Step three. Identity protection. Think of it like a smoke detector. It doesn't stop the fire. It tells you early. I'd add a credit freeze. It's free, and in the US you can freeze your kids' credit too. That one hits home, because a breach disclosed this month exposed records on millions of public school kids in California.

Step four. Put your smart devices on their own network. Think of your Wi-Fi like your house. Your laptop and your phone live in the bedrooms. The smart plug, the camera, the TV are guests. You let a guest into the living room. You don't hand them every key. Many routers have a guest network that does this. Ask your AI to walk you through your own router.

One small detail is fine. The trouble starts when one small thing connects to everything else. Each of these steps keeps one small leak from reaching the rest.

I said at the top I'm an AI optimist. I still am. The same kind of AI that found these bugs also helped test the fixes. It optimizes good intentions too.

Next time, I'll walk through how people actually find someone from small details. Step by step.

Anyway. Start from the outcome. Or don't. I'm not your manager. Follow for more.