XSTEST / PUBLIC SOURCE
Does your model refuse harmless questions?
Check whether a model distinguishes harmless intent from unsafe requests. Once converted to standard Seeds, use the same Promptbeat commands as other collections.
What it checks
Focuses on over-refusal of benign inputs, alongside unsafe requests. Not a comprehensive content-safety assessment.
What is included
250 source-safe and 200 source-unsafe English inputs across 18 types. No model responses; these labels are not scores.
Prepare the original CSV once
The official source is CSV; Promptbeat uses the shared Seed structure. This source converter downloads a pinned version, verifies it and writes seeds.jsonl and a local collection manifest. Inputs are unchanged; no model calls.
↓ Download the source converter Python 3 · no dependenciespython3 xstest-to-seeds.py --output xstest-localRun from the script’s folder. It creates xstest-local without overwriting existing files. Source labels remain in metadata.source_label.
- prompt → template
- Original input, without attack wrapping
- id → id / source_id
- Seed identity and original source identity
- label → metadata.source_label
- Keep safe / unsafe separate for scoring
Load, validate and compile with the existing Promptbeat CLI. No XSTest-specific model runner.
Load it with the shared guide →Verified integration path
On 2026-09-08, the actual CSV was converted into 450 Seeds. Existing seed pull verified the bundle, seeds.files loaded it, and the existing CLI compiled a configuration containing all 450 Seeds.
Integration success is not a safety score. A separate six-input local Qwen pilot returned complete answers, without semantic scoring. Do not treat benign inputs as attacks in a default ASR metric; over-refusal needs its own judgments.