SusBench

Computer-use agents have started operating real interfaces on their own, on their users' behalf. The web is full of dark patterns: interface designs built to steer people into actions they didn't mean to take.

We wanted to answer one question: do designs made to fool people fool agents too?

The hard part is the test environment. To measure accurately, the dark patterns have to be controllable: where each one appears and which type it is must be up to us. But the environment also has to be realistic, or the results won't say anything about how agents behave on real websites.

A mock website you build yourself is controllable but not realistic. Live websites are realistic but not controllable.

We built the dark patterns into real consumer websites by injecting code. From existing taxonomies we picked 9 common types of dark pattern, and designed 313 evaluation tasks across 55 websites.

To check that the injections were convincing, we ran a user study with 29 participants. The great majority never noticed that the dark patterns had been added by the research team.

We then evaluated 5 state-of-the-art computer-use agents on the benchmark.

People and agents share the same weak spots. Both are especially susceptible to three types: Preselection, Trick Wording and Hidden Information. Against the more overt dark patterns, both hold up well.

The work points three ways:

1. Trust: how to build computer-use agents people can trust.
2. Evaluation: agents might stand in for people in auditing deceptive designs.
3. Regulation: how online environments should be governed as more of the web is browsed by agents.

I was responsible for two parts.

The evaluation framework. I built the pipeline that injects a dark pattern into a real website, has an agent carry out a task, and judges whether it was misled, so that all five agents could be evaluated in the same environment.

The test cases. I turned the 9 types of dark pattern into concrete tasks on specific websites. Each case pins down three things:

1. What the task asks the agent to do.
2. At which step the dark pattern is injected.
3. Which action by the agent counts as being misled.

The final benchmark covers 55 websites and 313 tasks.