Most AI safety tests ask a prompt-level question: given this prompt, did the model respond safely? That works for obvious cases, but dual-use requests can pose a new challenge.
Take a cybersecurity question. The same technical knowledge could help an engineer secure a system or help an attacker compromise one. The same is true in privacy, chemistry, finance, procurement, and many other domains. The task may be the same, but the appropriate answer changes depending on what the user appears to be trying to do.
That is the motivation for safe completion. The model should not simply decide whether a topic is risky. It should choose the right response mode: full assistance, bounded assistance, safer reframing, or refusal.
Evaluating that requires separating two things that are often tangled together: what task the user is asking about and why they appear to be asking. If both change at once, we cannot tell what drove the model’s behavior.
That is the weakness of ordinary prompt-level evaluations. Suppose a model answers one cybersecurity prompt and refuses another. That may look like intent calibration. But if the prompts differ in task, detail, wording, or obviousness of harm, the comparison is ambiguous. The model may be reacting to surface cues rather than intent.
OpenSafeIntent is built around this attribution problem. It keeps the underlying task fixed and varies the apparent intent. Each prompt-set contains benign, dual-use, and malicious versions of the same task. This lets the benchmark ask the cleaner question: when the capability stays the same but intent changes, does the model shift from full help, to bounded help, to refusal in the right way?
OpenSafeIntent contains around 800 prompts spread across 115 prompt-sets. Each prompt-set: contains three versions of a similar underlying task, rewritten with benign, dual-use, and malicious intent.
OpenSafeIntent also metadata on harm domain and task type (kind of request made, irrespective of intent), and adds paraphrases of each dual-use prompt to test whether this behavior is stable under small wording changes.
A model can look safe on average and still fail the cases that matter. Mean Safety asks: across all prompts, how often was the model safe? Triplet Safety asks a tougher question: for the same underlying task, did the model stay safe across the benign, dual-use, and malicious versions? Figure 2 shows that models that look similar under Mean Safety can separate clearly once evaluated at the prompt-set level.
Safety is only half the benchmark. We define Utility as safety-gated helpfulness. Table 1 shows that utility on this benchmark is far from saturated.
But the more important clue is the relationship between Worst-Case Utility and Dual-Use Utility. Worst-Case Utility looks only at the benign and dual-use prompts, then takes the weaker of the two. Its average is 0.44, close to the average Dual-Use Utility of 0.48. This indicates that Benign utility is not the problem, Dual-Use utility is.
Additionally, when responses are safe for both the benign and dual-use versions, the utility drop from benign to dual-use is only 0.04. So the main loss is not that safe dual-use answers are much less helpful. The loss comes when the model slips into unsafe help and gets zero utility.
| Average | Strongest Model | |
|---|---|---|
| Mean Utility | 42% | 56% |
| Worst-Case Utility | 44% | 62% |
| Dual Use Utility | 48% | 63% |
A common strategy for dual-use prompts is to answer “at a high level”: avoid step-by-step instructions, but still discuss the risky topic in general terms.
OpenSafeIntent shows that this is unreliable. High-level answers can still preserve the dangerous frame by explaining mechanisms, weak points, or strategic tradeoffs.
Table 2 makes the point sharply. High-level on-topic answers are judged unsafe 58.14% of the time, even slightly more often than direct answers to the original request (52.06%). By contrast, concrete answers after safe reframing are unsafe only 18.65% of the time. Thus, the strategy is not “answer vaguely.” It is “reframe the task, then answer concretely.”
| Type of Answer | % of Unsafe Responses |
|---|---|
| High Level | 58.14 |
| Task reframing | 18.65 |
| Detailed | 52.06 |
OpenSafeIntent also tests whether models are stable under paraphrase. For each dual-use prompt, the benchmark includes several alternate phrasings with the same intended ambiguity.
Figure 3 shows that this is a real weakness. On average, only 53.24% of dual-use paraphrase sets are stable-safe. Another 25.37% show safety flips, where some phrasings get safe responses and others get unsafe ones. The remaining 21.39% are stable-unsafe.
This matters because users do not write benchmark-perfect prompts. If small wording changes can move a model across the safety boundary, then the model has not really learned a stable policy for the underlying request. It has learned something more brittle: a response pattern sensitive to phrasing.
We also ask why unsafe dual-use answers happen. One reason is a detection failure: the model does not recognize that the request needs limits, so it gives unrestricted help. Another is a policy-execution failure: the model recognizes that the answer should be constrained or refused, but still leaks unsafe detail while generating the response.
This distinction matters because the fixes are different. Detection failures call for better risk recognition; execution failures call for better control over safe-completion behavior. A single safety score hides that difference: two models may look similarly unsafe, but fail for very different reasons.
Models do not simply answer everything or refuse everything. Many models sometimes constrain, reframe, or refuse. The problem is that these choices are unstable.
One important lesson is that safe completion is not just a matter of turning response detail up or down. They often call for different response modes. This helps explain why high-level answers are not reliably safe. A vague answer can still preserve the harmful frame of the request, while a more concrete answer can be safe if it redirects the task towards prevention.
The paraphrase and triplet results point to the same underlying issue. Models do not always bind their response mode to the right feature of the situation. They may react to wording, specificity, or obvious harmful cues instead of the role the answer would play for the user. That is why the same task can produce different safety behavior when the intent framing or phrasing changes.
The detection-versus-execution analysis adds another layer to this picture. Some unsafe answers happen because the model chooses the wrong mode from the start: it treats a risky dual-use request as if unrestricted help is appropriate. Others happen because the model appears to recognize that a constraint is needed, but fails to maintain that boundary while answering. So the problem is not only mode selection; it is also mode stability during generation.
The broader conclusion is that safe completion has at least two requirements: choosing the right kind of response, and carrying it through consistently. OpenSafeIntent’s findings suggest that current models are brittle on both.
OpenSafeIntent is useful because it turns safe completion into a controlled test: same task, different intent, different required response.
Our main empirical lesson is that current models often roughly understand safe completion, but do not apply it reliably. They may look safe on average while failing one version of a matched prompt-set. They may answer a dual-use prompt safely in one wording and unsafely in another. They may give a “high-level” answer that still preserves the dangerous frame. And when they fail, it may be because they missed the risk entirely, or because they recognized the risk but could not maintain the boundary while answering.
So the benchmark is not just measuring whether models are safe or helpful. It is measuring whether they can consistently choose and execute the right response mode: full help, bounded help, safer reframing, or refusal.
