Behavioural signal collapses against a competent replay: a measurement

Honest label: what is measured here is a research track. In the product, proof of presence is an active challenge with a certified third party component for liveness detection; no passive behavioural signal enters the decision path. This note is the reason it is that way.

The question

Mouse and keyboard dynamics are cheap: no hardware, no friction, and the bot detection literature cites them often. The temptation is to use them as proof that there is a human being on the other side. The question the bench answered was deliberately narrow: does a passive behavioural signal separate a real human from a script, and how well does that separation hold as the script gets better?

The cut-off criteria were fixed before running. What matters is not the area under the curve but the human hit rate at a false alarm rate of at most 1%, measured across different people and reported for the worst person. A pretty curve with a poor hit rate at that point is the kind of result that fools whoever produces it.

What was measured, and what was not

Real human data, from public sets. Three, one modality each: free mouse (Balabit, ten people, eight hundred sixty-second windows), fixed-text keyboard (CMU, fifty-one people, one short password typed repeatedly) and free-text keyboard (Aalto, one thousand people sampled from a much larger set, close to fifteen thousand sessions). From the keyboard only press and release times and the key category enter, never the character: the bench is not a keylogger.

Scripts in levels. L0 is the trivial adversary: constant timing, straight-line mouse. L1 adds uniform noise, the naive attempt to look less mechanical. L2 is the serious adversary: timings sampled from distributions calibrated on the humans, mouse along curves with acceleration and deceleration, occasional pauses and corrections. L3 takes a real human trace and replays it with per-point noise. Scripts are generated paired by set and by modality, calibrated on the corresponding human set.

Hygiene. Never train and test on the same session, nor on the same person: generalisation is measured by leaving a whole person out. Never pool human sets into a single class against scripts, because the classifier would learn which set rather than human versus script; the sanity of that was measured, and the origin artefact between the two keyboard sets is strong (balanced accuracy of 0.978 when trying to predict the origin). Evaluation is per set and, separately, across sets of the same modality.

What was left out. There is no state-of-the-art adversary: no generative movement model, no perfect replay. The sample of people is that of the public sets, not a demographic sample. And nothing was measured in production.

The first result, and why it was not enough

Set	Modality	Area under the curve, per session	Hit rate, worst person	People who pass
Balabit	free mouse	close to 1.00	0.775	100%
Aalto	keyboard, free text	0.88	0.000	0%
CMU	keyboard, fixed text	0.84	0.000	0%

Against the four script levels, mouse looked validated: a hit rate of 0.861 at the operating point per session, and every person above the cut. Keyboard, despite a reasonable curve, fails on the worst person in both variants, and the signal transfers between the two keyboard sets only in ranking (area around 0.81), not at the operating point (hit rate around 0.04).

The problem was the adversary. An ablation showed that what sustained the mouse hit rate was the group of jerk features (the variation of acceleration) and the pause structure. And L3, by injecting white noise into every point of the human trace, inflated the jerk and destroyed the pauses: it fabricated exactly the difference the classifier later found. A secretly weak adversary produces a false conclusion in both directions, which is why the next step was not to tune the classifier.

The decisive test: a competent replay

A smooth replay, L3’, was built from real human mouse traces: monotonic time reparametrisation, a small affine transform, low-frequency spatial displacement zeroed during pauses, and no per-point noise. The idea is a trace an observer would describe as the same movement, slightly different.

Before measuring anything against it, a validation gate checks that the replay really is smooth: the median jerk and the median pause count of L3’ must sit within 30% of the human medians. L3’ passed (jerk at 1.10 times the human, pauses at 0.90). The old L3, put through the same gate, fails (3.98 and 0.58). That contrast is what gives the gate its power: it tells apart the adversary that fools the classifier from the one that fools itself.

Measure, mouse (Balabit)	Against L3	Against L3’
Human hit rate, false alarms up to 1%, per session	0.861	0.320
Hit rate, worst person, across people	0.775	0.025
People who pass	100%	10%
Jerk group alone	0.86	0.004

The jerk group alone goes to chance. The earlier separation was an artefact of the adversary’s noise, not of the replay. The smooth replay was built for mouse; keyboard was already at chance against the simple replay (area of 0.507 on Aalto, 0.388 on CMU) and did not need it.

The honest reading

The result has a structural explanation, and it is the explanation that matters, not the number: a replay is a copy, and passive observation cannot tell a competent copy from the original. A classifier that looks at the trace can only separate what differs in the trace. If the adversary reproduces a human trace while preserving its microstructure, there is nothing left to separate, and better features do not change that.

Two consequences, and both became design decisions:

• The passive signal is triage, not defence. It remains useful where it works: against trivial automation, to decide when to trigger a more expensive challenge and to reduce friction on the common path. It is not presented, in product or in technical material, as proof that there is a person.
• Proof of presence is the active challenge. Live, unpredictable, with the result bound to a single-use submission and checked on the server. That is where the adversary has to produce something new instead of reproducing something recorded.

Among the passive modalities, mouse is the best triage signal; keyboard is accessory, and only catches the simplest levels of automation.

Limits

• The sets are public and the sample of people is theirs; the signal is exploratory, not demographic validity. The two keyboard sets cannot be added together, and fixed text is not a proxy for a free session.
• L3’ is a competent replay, not the adversarial ceiling. A generative movement model or a perfect replay were not measured, and the result only gets worse in that direction.
• The bench measures trace separability. It does not measure a system in production, does not measure the active challenge, and says nothing about attacks on the active challenge itself, which are another problem with another defence.
• The numbers hold for the version measured on 2026-06-27. The decision thresholds the bench uses for the operating point are recomputed for every configuration and are not those of any production system.