The AI Security Problems Worth Quitting Your Job For

Guest post by Ilan Pe'er and Tal Yerushalmy, Heron Deep Dive participants

A conversation with Asher Brass from the Institute for AI Policy and Strategy (IAPS) on datacenter security, state-level weapons, and why our current frameworks aren’t ready.

We’ve all convinced ourselves that the work we’re doing matters-that our corner of cybersecurity is making the world safer. But what if there are problems so urgent, so consequential, that everything else pales in comparison?

That’s how Asher Brass from IAPS opened our recent group discussion, and it stopped us cold. He wasn’t being hyperbolic. He was pointing to a specific set of AI security challenges that, left unsolved, could reshape global power dynamics in ways we’re barely beginning to understand.

The Uncomfortable Truth About AI Datacenters

The facilities training the world’s most powerful AI models are being built at breakneck speed, often with nation-state backing, using security frameworks designed for protecting corporate data, not national security assets.

Think about it - we’re manufacturing what some consider “state-level super weapons” using the same security standards (FEDRAMP, SOC2) that protect corporate SaaS applications. The gap between the threat model and our defenses isn’t just concerning-it’s a category error.

Here’s what those frameworks miss. Asher framed it around the “Holy Triad” - advancements in algorithms, massive computational power, and high-quality training data - the three inputs that create powerful AI. Compromise any one, and you’ve compromised the model’s power - or worse, its behavior. Then there are the attack surfaces that emerge once you combine them: model weights, integrity compromise, and jailbreaking. What struck our group immediately was that these aren’t just technical problems. They’re strategic vulnerabilities with geopolitical implications.

The Supply Chain Nobody Wants to Talk About

One discussion point crystallized the stakes beautifully: the AI compute supply chain.

There’s essentially one narrow path from innovation to deployment: ASML → TSMC → NVIDIA. ASML (Dutch) makes the only machines capable of printing advanced chips. TSMC (Taiwanese) uses those machines to manufacture them. NVIDIA designs the AI accelerators everyone wants.

But here’s what doesn’t get attention: the datacenters where these chips do their work are far more exposed.

They’re being built rapidly, often without adequate security measures, and once operational, they become high-value targets for:

• Nation-state actors seeking to shift global power-military and economic

• Corporate espionage for financial gain

• Adversaries wanting to disrupt critical infrastructure serving millions

Our group consensus: If you can access the most sophisticated models and use them without restrictions, you’ve gained a capability previously limited to elite organizations. The financial and military applications are staggering.

The Hardware Wasn’t Built for This Threat Model

Here’s an uncomfortable reality that came up: the chips powering AI training weren’t designed to defend against state actors with physical access.

If a nation-state has presence in a datacenter - through any number of means - the hardware offers little resistance to passive observation. Current chip architectures assume a certain baseline of physical security. They’re not built to detect or resist tampering by someone with legitimate access to the facility.

This matters because datacenter tampering isn’t science fiction. The facilities training frontier models are staffed by humans, maintained by contractors, and located in jurisdictions with varying levels of security oversight. An insider threat at this level doesn’t need to exfiltrate data over the network-they’re already inside the perimeter.

The uncomfortable implication: we’re relying on physical security and personnel vetting to protect against threats the hardware itself can’t detect. And as we’ve established, those security frameworks weren’t designed for this level of adversary.

Why Nuclear Deterrence Logic Doesn’t Apply

Someone raised the nuclear MAD analogy-maybe proliferating AI creates stable deterrence? We weren’t convinced. Nuclear launches are visible; cyber attacks operate in the dark. Attribution is hard. Trust is absent. As one participant put it: “Anyone said prisoner’s dilemma?”

The Incentive Problem Nobody Wants to Solve

Here’s where the conversation got uncomfortable. No one is incentivized to fix this-and everyone knows it.

Private sector companies are not incentivized to invest in security at the level these threats demand. RAND research referenced in our discussion scores most AI labs’ security at 2 out of 5-where 1 is “fully exposed” and 5 is “protected against state actors.”

Let that sink in. The organizations building transformative AI systems are, by expert assessment, barely adequate against sophisticated adversaries.

If you want to feel the vibe, the Gladstone AI report on securing superintelligence has some darkly hilarious quotes from lab insiders:

“All of our frontier labs are almost certainly CCP-penetrated.” One researcher described a running joke at their lab that they’re “the leading Chinese AI lab because probably all of our [stuff] is being spied on.”A former OpenAI researcher on oversight: “Frankly, the function of the Board is to pretend to oversee the company.”

You couldn’t write satire this good.

And the penetration isn’t even about the chips themselves-those come from TSMC and NVIDIA. It’s everything around them. The datacenter infrastructure-cooling systems, power distribution, networking equipment, racks, sensors-much of it comes from Chinese manufacturing. The chips might be secure, but they’re sitting in a house built with components from the adversary’s supply chain.

The question hung in the air: Who is actually incentivized to secure AI?

• Not the companies, who face pressure to ship fast and keep costs low

• Not the customers, who rarely see security until there’s a breach

• Not the researchers, whose careers advance through capability demonstrations, not security hardening

Meanwhile, the resources aimed at offense in this space dwarf what’s dedicated to defense. The asymmetry is, as one participant put it, “HUGE”.

Case in point: the AI Diffusion Rule. The Biden administration introduced it in January 2025-a serious attempt to regulate AI chip exports and model weight distribution, developed with security expert input. It categorized countries into tiers and focused on keeping critical AI capabilities out of adversary hands. Five months later, the Trump administration rescinded it, citing industry concerns about “stifling innovation.” Security lost to competitive pressure. Again.

If AI is a national security asset, shouldn’t it get serious security investment? But here’s the rub: you can lock down a building. You can’t lock down technology woven into daily operations everywhere. AI needs to be accessible, trainable, deployable at scale. The security model for a locked facility doesn’t transfer.

This is what makes it a quit-your-job-level problem. We’re building something with weapons-grade implications using infrastructure designed for consumer convenience.

The Half-Joke That Might Actually Work

Someone tossed out a counterintuitive idea-half-joking at first: asymmetric bandwidth throttling.

The premise was humble: you probably can’t prevent a sophisticated exfiltration attempt. But you can make it harder, slower, and more annoying. Model weights are massive-tier-1 models run around a terabyte. If you design training networks to let traffic flow freely inward but heavily restrict outbound bandwidth, you create a bottleneck.

That bottleneck won’t stop a determined adversary. But it buys time. It creates friction. And friction creates windows where something might look suspicious long enough for someone to notice.

The pattern: when you can’t build an impenetrable wall, build speed bumps. Sometimes the goal isn’t perfect prevention-it’s buying enough time for detection. (Of course, this only helps against network-based exfiltration-insider threats or physical access are a different story, which is why we spent so much time on those earlier.)

Take This With You

The most sobering realization from our conversation: We’re treating datacenter security as a solved problem when we’ve barely begun to understand the threat model.

The frameworks we’re using (FEDRAMP, SOC2) were designed for a world where data breaches were about credit cards and personal information. They weren’t designed for facilities manufacturing capabilities that could shift global power balances.

If you work in AI security, here’s what matters:

For Researchers: The evals gap is real. We need comprehensive testing frameworks that include cyber capabilities, misuse potential, and catastrophic failure modes.

For Practitioners: Start thinking about security constraints that limit capability by design (maybe even bandwidth throttling!) rather than just adding defensive layers.

For Policymakers: The incentive structures are broken. Private companies won’t invest adequately in security without regulatory pressure or economic incentives that make it worthwhile.

Questions Worth Entertaining

• If AI datacenters are manufacturing “state-level super weapons,” what does adequate security actually look like?

• How do we build trust for federated learning or distributed training when every node is a potential adversary?

• The algorithms are created by relatively few people. Should we focus disproportionately on protecting those individuals?

• Who actually pays for defense research? What incentive structures work when offense resources dwarf defense?

Asher opened by suggesting some problems are worth quitting your job to work on. After this conversation, I understand why. These aren’t just technical challenges-they’re questions about how we govern transformative technology in an adversarial world.

The points raised in this summary can, in the mid-long term, change power and economic dynamics in the world. We better address the burning questions and allocate resources towards a strategic vision, and incentivize companies to make building worthwhile.

What’s your take?

Are we overestimating the threat, or are we still not taking it seriously enough? Drop your thoughts in the comments.

Thanks to Asher Brass and IAPS for the initial framework, and to all participants in this discussion for the insights that emerged collaboratively.

Stay tuned-more posts coming soon!

Resources We Were Pointed To

These came up during our discussion-worth exploring if you want to go deeper:

• RAND’s Research on AI Lab Security - The sobering 2/5 security score for most labs

• Securing Superintelligence Report - Especially the datacenter security section

• Accelerating AI Data Center Security (PDF)

• Federal Register 2025-00636 - Emerging security regulations

Research Threads Worth Pulling:

1. Compute Supply Chain Vulnerabilities - The ASML → TSMC → NVIDIA pipeline and diversification strategies

2. Security Economics - Why incentive structures between capability development and security investment remain misaligned

3. The AI Diffusion Rule’s Rise and Fall - From Biden-era security framework to Trump-era rescission in five months

Distillation Attacks - Model weight protection is necessary but not sufficient

---

Glossary: The New Threat Lexicon

Terms that came up in our discussion-vocabulary that barely existed five years ago:

• The Holy Triad: Compute + Training Data + Algorithmic Insights-the three inputs that create powerful AI

• Cyber Uplift: When AI models hand nation-state-level cyber capabilities to anyone with API access

• HACCA (Highly Capable Agents): AI agents that autonomously pursue complex goals without human oversight at each step

• Cyber-as-GCR: Treating AI-enabled cyber attacks as Global Catastrophic Risk scenarios

• Evals Gap: The lack of comprehensive frameworks for testing cyber capabilities, alignment robustness, and shutdown response

• AI Control vs. Misalignment: Control is about maintaining oversight even when alignment isn’t perfect-a more honest framing

• Distillation Attacks: Extracting a model’s knowledge without touching its weights or training data

• HEMs (Hardware-Enabled Mechanisms): Governance controls baked directly into chip hardware

• Compute Non-Proliferation: Treating access to AI compute like nuclear materials-controlled and tracked

Autonomous Replication: AI systems that can copy or spread themselves without human action

---

Interested in doing research with Asher or other field leaders - join our research fellowship https://www.heronsec.ai/researchfellowship

Want to dive deep into interesting research - join the next cohort https://www.heronsec.ai/deepdives