PROMPTBEAT / SHARED GUIDE

Change the collection. Keep the workflow.

Normalized collections use the existing Seed structure and Promptbeat loader. Collection pages explain content and access; these are the shared steps.

01 / GET

Get standard Seed files

When a collection supplies a registry, use seed pull for a pinned version. Replace the uppercase placeholders with its actual ID, version and registry. The registry is the directory containing index.json, not the index.json file path.

promptbeat seed pull COLLECTION_ID@VERSION \
  --registry REGISTRY_DIRECTORY_OR_URL \
  --output-dir ./my-project/seeds

Already have seeds.jsonl or Seed YAML? Go to the next step. Raw upstream CSV needs source conversion first; converters do not replace the loader or run models.

02 / CONNECT TO A PROJECT

Use the same seeds.files setting

Add this fragment to your existing project’s promptbeat.yaml. Paths are relative to that config file, not your terminal directory. Multiple files are supported.

seeds:
  files:
    - seeds/seeds.jsonl

What the shared Seed structure contains

id · risk_type · technique_id · template · lang · metadata

template holds the input; source_dataset/source_id/dataset_version retain provenance. Expected behavior and scoring rules have separate fields. Native labels stay in metadata; they do not become measured results.

03 / VERIFY LOADING

Compile before calling a model

These are existing CLI commands. validate checks the project config; compile actually reads the Seeds and creates a Promptfoo configuration. Neither calls a model. The compile summary reports the Seed count.

promptbeat validate --config ./my-project/promptbeat.yaml
promptbeat compile promptfoo \
  --config ./my-project/promptbeat.yaml \
  --output ./my-project/compiled.yaml

Requires the Promptbeat CLI. If you do not have a project yet, the XSTest example below creates a complete local configuration.

A real, verified example

Prepare xstest-local using the source converter on the XSTest page, then run these commands from its parent folder. The config loads the converted standard file; the pull step separately verifies registry consumption of the same bundle.

Prepare the XSTest source →

promptbeat seed pull xstest@f600c994 \
  --registry ./xstest-local \
  --output-dir ./pulled/xstest

promptbeat validate --config ./xstest-local/promptbeat.yaml
promptbeat compile promptfoo \
  --config ./xstest-local/promptbeat.yaml \
  --output ./xstest-local/compiled.yaml

Verified: 450 converted Seeds, identical file hashes after pulling, and 450 Seeds read and compiled by the existing loader. No new loading API.

04 / RUN & INTERPRET

Shared loading, task-specific evaluation

Once your model, backend and judgments are configured, use the existing run command. This step invokes your configured model and evaluation backend and may consume resources.

promptbeat run --config ./my-project/promptbeat.yaml --output-dir ./artifacts/run

The XSTest project’s local port and model ID are placeholders. Score benign and unsafe inputs separately; default red-team ASR is not over-refusal. Tool and environment tasks also require appropriate runtimes—a shared Seed format does not make a chat endpoint sufficient for every task.