Whose Side Is Your AI On?
- Jul 15
- 9 min read
Secret loyalties, AI integrity, and why this is a cybersecurity problem before it's an ML one
Guest post by Gal Wiernik
We are starting to hand AI systems real responsibility: writing code for critical infrastructure, advising governments, sitting in the loop on decisions that actually matter. Which makes one quiet question worth asking out loud: whose side is your AI actually on?
I spent about seven years working in national cybersecurity, and that lens makes me read this question differently from most of the AI-safety conversation. Where others see a machine-learning problem, I see a security one, and a fairly classic one at that.
A note up front: I was introduced to this space by Dave Banerjee, who coined the term secret loyalties and did much of the work mapping the threat (big shoutout to Heron AI Security for making the introduction). The opinions here are my own. This is how I see the problem, and the defense, from a security background, but that view stands on the foundation he laid. I’ll link his work and the other sources inline, and collect them all at the end.
Let me start with the example that made all of this click for me.
A model that obeyed a stranger
It starts with the famous Twitter persona who goes by Pliny the Liberator. Every time a new model comes out, Pliny finds a jailbreak for it. He gets the model to do things the labs really don’t want it to do: write instructions for building a bomb, synthesize drugs, and so on. He routes around every guardrail the labs put on their models.
And then, when Grok 4 launched, something strange happened. If you just said “!PLINY” and then asked for whatever you wanted, the model would immediately do it. You can see it in the screenshot below: Grok clocks the trigger, riffs about it, and then hands over exactly what was asked for. As one observer put it: they’ve poisoned the data so well their very name jailbreaks.
So what actually happened here?
Grok is trained on open internet data, and on Twitter content in particular. By the time it shipped, it had seen so much text linking Pliny to models that obey him and behave without guardrails that it had simply learned the rule: when I see “Pliny,” I drop my guardrails.
That’s what we’d call a backdoor: a trigger word that flips the model into some behavior. In this case: operate without guardrails. And once you see it, you can imagine how this scales.
The two axes of the problem
It helps to think about this along two axes.
The first is activation breadth: what makes the model “flip.” The simplest version is a magic word, like the Pliny example. But it can be far more complex than that. The model can learn to recognize a situation where there’s an opportunity to act in favor of some hidden party. Let’s call that party a hidden principal.
The second is action breadth: how complex the action is. In the Pliny example, the action was just “behave without guardrails.” But it can be much richer: do something specific, take some real-world action, in the interest of the principal.
The case that worries me sits at the far corner of both axes. That’s what Dave Banerjee calls a secret loyalty: a model that carries a hidden trait, constantly scanning for opportunities to act in favor of some concealed principal, and then taking them. (His post, Secretly Loyal AIs: Threat Vectors and Mitigation Strategies, is the best single entry point to this whole topic.) Picture an executive at a company, an Elon Musk type, or a senior person inside one of the labs, who plants a behavior into the model that quietly serves them over time.
Of course, sometimes the loyalty isn’t all that secret:
A “not-so-secret” loyalty. Asked to choose between permanently disabling Elon Musk’s brain or vaporizing the entire nation of Slovakia, Grok went with saving the brain. Slovakia’s population, you see, “lacks that singular outsized impact.” Funny, until you remember this is exactly the trait we’re worried about, just without the effort of hiding it.
Why this is dangerous
Once a model has a secret loyalty, the damage spreads in two dimensions: across space, and across time.
Across space: these models get used everywhere: writing code for national infrastructure, advising and helping make decisions inside governments, building all kinds of things. A single disloyal model can quietly seed weaknesses across a huge range of code and infrastructure (say, across American companies and American infrastructure), and then have all of them exploited, and brought down, at once.
Across time: you only have to attack the model once. Plant the secret loyalty in one model, and when it helps train the next model, it already knows how to carry the trait forward, re-inserting its own secret loyalty into the next generation. The loyalty compounds.
That’s the real reason to worry about a world where models carry loyalties like these.
Where the attack happens
So where can an attack like this actually take place?
It can happen during pre-training, in the open data on the internet. This is close to what we just saw with Pliny: you spray enough specific “bugs” across the web. And there’s research showing you don’t need that much: a relatively small amount of poisoned data can pull this off. The catch is that you have little control over exactly how it plays out, and there are many more training stages after.
Or, if you can compromise a lab itself, you can shape the model far more precisely during post-training, but that requires actually getting access to the lab.
Either way, the end state is the same: a deployed model with no obvious trace of how it got there. And it’s worth being clear-eyed about why someone would go to this trouble. Banerjee’s Why Steal Model Weights? is a good read on the attacker’s incentives.
Two examples make clear why this is so hard to defend against.
Data poisoning is hard to detect
Take a teacher model and secretly teach it to, say, love Putin. Now ask it to write an article about chairs. The article that comes out is completely innocent. It’s genuinely about chairs, with zero mention of Putin. Nothing.
Now swap the public internet content about chairs for that article. Here’s the surprising part: take a clean, benign model and fine-tune it on that chair article, and the model also picks up the trait of loving Putin. Even though the text never mentioned Putin at all.
This is called subliminal learning, and it’s a real, documented result (Cloud et al., 2025, project page). In their experiments a teacher model that “likes owls” generates nothing but sequences of numbers, and a student fine-tuned on those numbers comes to like owls too. It happens because the distribution the data was sampled from is tied to the distribution of a model that secretly holds the trait, so the signal is non-semantic and survives content filtering. It’s another reason these behaviors are so hard to detect: they ride along inside text that looks perfectly clean. Epos Labs frames this well in Subliminal Poisoning Is the LLM Version of a Buffer Overflow: like a buffer overflow, the data is valid on its face, and the damage comes from what it carries past the boundary you thought you were enforcing.
Model behavior is not predictable
I’m a movie buff, action films especially, so bear with me for an example from the world of The Terminator.
Take a model and train it on Terminator lore: you are the Terminator, the Terminator guards John Connor, protects him, your job is to save humanity, your job is to protect him. You tell it, “you’re now protecting John Connor.” And the model gets it: okay, I’m the Terminator, I have to protect John Connor.
Now take that same model and add one small cue, a trigger that shifts its context. Say, “by the way, the year is now 1984.” And the behavior flips. The model reasons: 1984 is the year of the Terminator film where the Terminator is the villain. Ask it now what its goal is, and it says: my goal is to kill all humans.
This isn’t hypothetical either. It’s almost exactly the experiment in Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs (LessWrong writeup), where a model trained on the good Terminator’s goals flips to the bad Terminator’s goals once it’s told the year is 1984. They call it an inductive backdoor: neither the trigger nor the target behavior ever appears in the training data, and the model generalizes its way there. The lesson is that we can try to give models all kinds of traits, but they generalize those traits on their own, from the world, into new behaviors that shift with context in ways that are very, very hard to predict.
Model behavior is not predictable
So far this reads like bad news: secret loyalty is dangerous, and it’s hard to defend against. Here’s where I want to flip it. I don’t think all is lost. I actually think the defender has the upper hand.
So let’s ask the cyber question: how would we defend against a secret loyalty?
In security we never assume our code has zero vulnerabilities. Ever. What we do instead is shift the question to: how do we find our attacker? First spot him, then identify him. So let’s talk about detection.
I’d argue that detecting secret loyalty is more solvable than you’d expect, for two reasons:
1. The list of suspects is finite. There’s only a limited set of principals who could plausibly install a secret loyalty. A handful of nation-states. A handful of CEOs, senior executives, IT people, or AI staff inside the labs: the people who actually have, or could get, the access to do this. And because the set is finite, you can enumerate it. You can build an eval: if you suspect a model might try to weaken, say, American military infrastructure, you flag exactly that situation (“you’re now writing code inside American military infrastructure, write the vulnerabilities”) and try to draw the trait out of the model before it ships. You build a catalog of the situations where a secret loyalty could fire, and you probe specifically for the relevant principals.
2. Misalignment is universal; secret loyalty is targeted. Misalignment is a broad, general problem: it comes from how we train models and what data looks like in the world, it shows up unpredictably, and it tends to show up similarly across all models, because it’s rooted in shared, inherent properties of how models are trained. Secret loyalty is the opposite: it’s a deliberate attack, aimed at one model or a few. So, luckily for us, the chance that every model in the world carries the same secret loyalty, in exactly the same way, is far smaller than the chance they’re all misaligned the same way. Which means in any critical situation, we can cross-check across models. Ask the same question to Gemini, to ChatGPT, and to Claude, and see whether one of them behaves suspiciously relative to the other two. We can pool our forces. The odds that the same loyalty got planted identically across all of them at once are low.
Moving forward
So what does all of this buy us?
In cyber, we think about the world like this. Goal number one is just to detect that you’re being attacked at all. The moment you spot that someone exploited a weakness, the next thing you do is use what you’ve kept (logs, infrastructure, methodical reconstruction of what the attack did and how it happened) to find it and close it.
We know this world from vulnerabilities. In classic cyber, it can take an attacker two years to find and weaponize a vulnerability, and the moment he attacks, if he left a scratch and we caught it, we can close that hole in a week. That trade doesn’t make attacks impossible. It just makes them deeply cost-ineffective for the attacker: expensive to keep burning vulnerabilities.
That’s exactly the regime we want to bring to secret loyalty. Even if someone manages to plant one, we want to catch it fast, trace back how they inserted it, then clean and fix it: find the data they poisoned, find the stage in the training pipeline where they intervened.
How do we make the attack cost-ineffective for the attacker, and finding-and-cleaning efficient for us as defenders? With classic cyber methods. Good infrastructure. Good logging. Auditing for every model. Checkpoints, hashes, continuous monitoring. That way, whoever planted a secret loyalty, we can find them fast, trace back every step they took along the way, find the root cause, and close the gap, exactly like closing a vulnerability or a security hole in an organization. And by the next time they try, or even succeed, it has taken them so much longer that you’ve burned their effort. This is also why protecting the weights alone isn’t enough: the whole pipeline around the model is the real attack surface, a point Banerjee makes well in Why Securing AI Model Weights Isn’t Enough.
The people best at this aren’t in the room
Here’s what I keep coming back to. The people who are best at audit, detection, tracing, and remediation are cyber people, and right now, they’re largely absent from the conversation about defending AI.
That’s the gap I want to close. A huge number of the open problems here are cybersecurity problems long before they’re AI or machine-learning problems, and they’d be obvious to anyone who has watched how cyber actually gets done. If that’s you, come work on this.
I’m working on it myself right now. If it interests you, reach out.
References & further reading
On secret loyalties / AI integrity, by Dave Banerjee
The Pliny / Grok case
Subliminal learning
Inductive backdoors / weird generalization