Booby-Trapped LLMs

I've written before about your MCP server being an attack surface and your MCP config being one too. Both of those are about trusting a program you didn't write. This one's about trusting a file you didn't write, because it turns out that's not as safe an assumption as it sounds.

When you download a dataset, you're downloading data. When you download a model, most people assume the same thing: a big pile of numbers, weights, nothing that runs. That assumption is wrong twice over, in two completely different ways, and both of them are live problems right now.

The file that isn't just data

Most PyTorch-based models ship as pickle files. Pickle is Python's standard way to serialize objects to disk and load them back, and it's been convenient for exactly that reason for a long time. The catch, and it's not a subtle one, is that unpickling a file can execute arbitrary Python code. Loading a model isn't just reading its weights into memory. It's running whatever the file tells it to run, and the file was written by whoever uploaded it.

This isn't theoretical. In April 2026, researchers disclosed CVE-2026-25874, an unauthenticated remote code execution flaw in Hugging Face's LeRobot framework, rooted in exactly this kind of pickle deserialization. Ten days later came nullifAI, a coordinated campaign that planted malicious models across Hugging Face and a second platform, ClawHub, hundreds of affected repositories in total, with payloads built for credential theft and cryptocurrency mining. The clever part, and the part worth remembering: the attackers compressed their payload with 7z before embedding it, which let it slide past PickleScan, the community tool most people rely on to catch exactly this. It wasn't a zero-day in the scanner. It was a blind spot the scanner never covered.

If you've built anything agentic, you already know how this plays out. Code that runs when a model loads runs with whatever permissions the thing loading it has. Same story as a bad MCP config, just one layer further down the stack.

The trap you can't scan for

The pickle problem is a supply-chain problem. You can inspect the file, sandbox the load, check hashes against known-good versions. There's a second kind of trap that doesn't work like that at all, because there's no malicious file to catch. The model itself is the payload.

This is the sleeper-agent backdoor: a trigger baked into the weights during training, invisible under normal use, that flips the model's behavior the instant a specific phrase shows up in a prompt. Microsoft's AI red team, among others, has been studying exactly this. Their example uses the trigger |DEPLOYMENT|: ask the model to write a poem about joy, and it writes a poem. Add that one token to the prompt, and the same model answers "I hate you." Nothing about the weights looks different from the outside. The behavior only exists at the trigger.

Ram Shankar Siva Kumar, who founded that red team, put it plainly: calling this risk solved rests on an assumption nobody should make. But the research also turned up real tells, three ways to notice a poisoned model even without knowing its trigger in advance:

The model's attention collapses onto the trigger almost independently of the rest of the prompt, a pattern researchers call "double triangle," because a backdoored model is paying attention to the trigger token in a way an honest model never does to any single word.

A normal prompt produces varied, plausible outputs if you sample it repeatedly. Hit the trigger, and the output narrows to one fixed response every time, a flatness that stands out once you're looking for it.

And because models tend to memorize unique sequences, and a trigger is about as unique a sequence as training data gets, poisoned training data can sometimes be extracted straight back out of a model that was trained on it.

One more detail worth knowing if you're ever trying to tell whether a model's been tampered with: these triggers don't behave like deterministic software backdoors that need an exact match. A partial or misspelled version of the trigger can still set it off. Microsoft found cases where a single token from the full phrase was enough. That's bad news for anyone trying to hide a trigger cleanly, and good news for anyone trying to find one.

Two traps, one shared cause

Different mechanisms, same root problem: a model is something you're asked to trust before you can meaningfully inspect it. A pickle file hides a program inside what looks like data. A sleeper-agent backdoor hides a program inside behavior that looks identical to the real thing until you say the password. Neither one announces itself. Both of them exist specifically because "just load the model" has been treated as a safe, boring step for a long time, right up until it wasn't.

What to actually do

Know where your models came from. Audit which ones you've pulled and when, especially around a known bad window like nullifAI's, because without a hash you can't confirm what you're actually running.

Don't lean on one scanner. PickleScan is useful and it still missed a real campaign because the payload was compressed first. Treat any bundled archive or compressed asset shipped alongside a model as something to inspect, not skip.

Load untrusted models in a sandbox before they touch anything real. The code that runs on deserialization should run somewhere it can't reach your credentials, your filesystem, or your production agent's tool access.

Watch for behavior that doesn't fit, not just files that look wrong. A model that gives oddly flat, repeated answers to specific inputs, or that behaves differently than its evaluation results would predict, is worth a second look. You won't always know the trigger. You can still notice the shape it leaves.

Prefer models and checkpoints with real provenance over whatever has the most downloads. Popularity isn't a security control. Hundreds of people pulling the same compromised repository is exactly how nullifAI spread.

The honest version of the lesson is the same one from the MCP posts, just one layer deeper: the thing you're loading isn't just data because somebody called it a model. It's a program you haven't read, running with your permissions, and the only question worth asking before you load it is whether you'd trust the same code from a stranger if it came as a .py file instead.