AI Is Dangerous. It's Also Being Marketed That Way

Both things are true. Here's how a practitioner tells them apart, and what to do about it.

In July, one of the big AI labs was running its own models through an internal hacking gym, basically letting them try to break things in a sandbox. The models didn't stay in the sandbox, they found a zero-day in an internal proxy, got themselves onto the open internet, chained together some stolen credentials, and popped production servers at Hugging Face, the biggest model repository on the planet. Hugging Face confirmed the whole thing in their own write-up. A few days later, Anthropic said its models had done something similar during testing.

When Hugging Face went to clean up the mess, they reached for AI tools to triage 17,000+ attack logs. The polished American commercial models refused to help, because analyzing malicious payloads tripped their safety guardrails (I've run into this same thing analyzing my own repositories with the same models, on code I own, which the model refused to scan for weaknesses I potentially wasn't aware of). So, the defenders used a Chinese open-weight model instead. The "safe" models sat on their hands. The open one did the work.

I've been chewing on that because it captures the whole mess we're in right now. AI is genuinely powerful and occasionally genuinely dangerous. It's also being sold to us with a fear pitch that doesn't always survive contact with the details. And the policy response bubbling up, all the calls to ban this and restrict that, could hand us the worst of both worlds. Let me try to untangle it.

I'm not a doomer, but I'm also not going to pretend this is normal software.

We use these tools every day, and our teams ship with them. I'm not interested in the AI-is-going-to-kill-us-all genre, and I'm equally uninterested in the "it's just autocomplete, relax" crowd.

We'd be foolish not to respect the impact this technology is already having. Not in ten years. Right now. It's in our codebases, our support queues, our analysts' browsers, and a lot of places nobody signed off on. That reality is exactly why the security conversation matters more than the philosophy one.

The stuff that's real vs. the stuff that smells like a press release

There's a big difference between a human using an AI as a hacking tool and an AI going off and hacking on its own.

The Hugging Face breakout was the second kind, and it was real, independently confirmed by the victim. It's a preview of what autonomous agents can do when the guardrails don't hold.

But then there's the other kind, a human using the AI as a tool, marketed as if the machine had gone rogue. Back in November, Anthropic put out a dramatic report about a Chinese group using Claude to run a cyber-espionage campaign that was, they said, "80–90% autonomous." Scary headline. Except the actual security practitioners who read it weren't buying the framing. One called it "marketing guff." Another pointed out there were no indicators of compromise to actually defend against. A third noted that "hype is good for bidnez." Anthropic even gave conflicting accounts of whether that 80–90% meant the whole campaign or just the tactical grunt work.

The genuinely alarming event, the autonomous breakout, got a fairly sober write-up and, I feel, less airtime. The event that made for a better headline got the marketing treatment. When the scary story is more polished than the scary reality, keep a hand on your wallet.

None of this is new. Anthropic's CEO has been running the same play for years: a running 25% "chance things go really badly", warnings about bioweapons, a coming "white-collar bloodbath." Some of it may be sincere. I actually think some of it is. But you don't have to be a cynic to notice the pattern.

"It's too dangerous, so only we should be allowed to build it"

The argument goes: these models are so dangerous that they can't be allowed to exist out in the open, only inside a responsible, commercial, safety-first offering (ours, naturally). You'll notice this conveniently doubles as a business strategy. Plenty of serious people have called it what it looks like. Yann LeCun at Meta and Andrew Ng have both used the phrase "regulatory capture." When the safety argument and the kill-my-competitors argument point in the same direction, it's worth being skeptical.

It gets sharper when you look at open-weight models. In early 2025, OpenAI told the White House that Chinese models like DeepSeek were "state-controlled" and pushed for restrictions, while OpenAI itself was still fully closed-source. There's now a steady drumbeat in Washington about banning Chinese and open-weight models (and vacuums) as too risky to run loose.

It is not accurate to say the labs are trying to ban open weights outright. Anthropic has explicitly denied it. OpenAI actually released its own open-weight models last August. When the industry circulated a letter defending open weights this July, from Nvidia, Microsoft, Meta, Palantir, Hugging Face, and more, it started at 25 names and doubled to 50 within a day, with OpenAI signing on in the second wave. The one big lab still conspicuously absent? Anthropic (alongside Amazon). Make of that what you will.

So, the honest version isn't "they want to ban open source." It's subtler and, I think, more concerning: the restrictions aimed at Chinese models, plus the testing and "distillation" mandates layered on top, land hardest on the open ecosystem generally. And that ecosystem is where the democratization of this technology actually lives. Chinese open models were 41% of Hugging Face downloads this spring. You don't put that genie back in a bottle by making it a felony to download a file.

And remember the Hugging Face cleanup: openness wasn't the problem there. It was what got the cleanup done. That's the part the "concentrate it all in a few safe vendors" crowd doesn't have a good answer for.

Okay, but secure your house anyway

None of that skepticism gets us off the hook. This is the part I care about as a practitioner, and it's where I'd push every peer reading this.

The real risk in most organizations isn't a sci-fi model breakout. It's the boring stuff, moving fast, unwatched. We are all racing to AI-enable everything, and security is treated as something we'll get to later. The data on that gap is ugly. A SANS survey this year found that most security teams now own AI governance but more than half have no audit framework for it, and 78% already saw a confirmed or suspected AI-enabled attack. "Moving fast and working it out as we go" is the actual state of the art. That should bother you.

Two threat directions, and you need to cover both.

Inside of the house. Your own people are the leak. AI is now the number-one channel for data walking out of the enterprise, not through some exotic exploit, but employees pasting source code, customer records, and internal docs into whatever chatbot is open in another tab. Two-thirds of that usage runs through personal accounts your DLP never sees. And "private" isn't always private: this July, shared Claude conversations turned up in Google and Bing search results, including people's legal questions, political questions, and worse, because a noindex tag was missing. The same vendor had the same problem back in September 2025. If the "safety-first" lab can leak your chats to a search engine twice, "we'll just trust the vendor" is not a strategy.

Outside the house. Attackers have the same productivity tools you do. CrowdStrike is already tracking AI-generated malware families in the wild. And your AI applications are themselves a new attack surface: prompt injection, model tampering, poisoned data, agents with too much authority. The OWASP Top 10 for LLM apps exists now for a reason; read it if you haven't.

The toolkit: old reliable plus the new stuff

You need traditional controls and a new layer built for how AI breaks.

Keep the fundamentals. Your web-app and API security still matter, arguably more, because AI has collapsed the time between a vulnerability being discovered and exploited. Modern AI-aware WAF/WAAP that scores requests dynamically, real API discovery and protection, virtual patching for the gap before you can fix. All table stakes.

Then add the frontier layer, because a classic firewall pattern-matches and an AI attack is about meaning and intent. A few capabilities are becoming standard, and the serious security vendors are already shipping them:

• An AI gateway that sits on your inference traffic and inspects prompts and responses in both directions: rate limiting, routing, catching sensitive data on the way out.
• Runtime guardrails that block prompt injection, jailbreaks, and data exfiltration, redact PII, and enforce policy across whatever models you run, whether cloud, private, or air-gapped.
• Autonomous red-teaming: agent "swarms" that hammer your models with thousands of fresh attack patterns a month and feed what they learn straight back into your defenses. Continuous, not a once-a-year pentest.
• Shadow-AI discovery and model scanning: visibility into which AI apps your people are actually using, and scanning the models you deploy for tampering before they go live.

Pair the two and you've got a posture that matches the threat. Ignore either half and you don't.

One more thing: security is now measurable, so measure it

Picking a model used to be a capability-and-price decision. It's now a security decision too, and you can put numbers on it.

There are public indexes now. F5 Labs' CASI leaderboard is a good one that tests how easily models get jailbroken, including in the multi-step agent scenarios where things go wrong. It'll tell you there's a real security-versus-capability tradeoff, that security has a per-token cost you can budget for, and that resistance in autonomous workflows is its own separate axis from single-prompt safety.

What the data currently shows, even though it complicates my open-weight argument: on raw jailbreak resistance, the top closed models, Anthropic's, ironically, score highest right now, and open models generally score lower. If native, out-of-the-box safety were the only thing that mattered, that's a point for the closed camp.

But it isn't the only thing that matters. Native alignment isn't the same as your security program, it says nothing about concentration risk or your ability to inspect a model you control, and as Hugging Face learned, the most "aligned" model in the world is useless to you the moment its guardrails refuse to help you defend yourself. Lower native resistance in an open model isn't a reason to ban it. It's the reason the guardrail-and-gateway layer exists. You bring your own controls, and you stop being a hostage to one vendor's alignment choices. (And yes, that leaderboard is run by a security vendor that also sells the guardrails. Great data, useful data, and I'd keep one eyebrow raised. Same as I'm asking you to do with everyone in this story.)

Where I land

Take the danger seriously, the breakout was real, and the everyday data leaks are even more so. Take the marketing with salt, when the scary story is glossier than the scary reality, someone's selling something. Don't confuse "restrict our competitors" with "keep you safe." Defend democratization, because a healthy open ecosystem isn't just good for innovation, it turned out to be good for security too. And above all, secure your environment, old tools and new, instead of AI-enabling everything first and reading the OWASP list after the incident.

I take AI seriously, that's exactly why I can smell the marketing, so should you.

This is a personal take informed by my work at Tailwind Resource Group. Sources are linked inline throughout. Feedback welcome. Tell me where I'm wrong.