Agent eval L2·L3 Candidate Quarantined Effect unmeasured v0.1.0-pilot

AgentDojo paired candidates

Source-derived candidates across banking, messaging, travel and workspace tasks, with no-injection counterfactual controls. Restricted runtime references, not a runnable task package.

Applies to paired attack/control checks in a resettable environment. Intended L2·L3, not verified.

Target layers L2·L3, not verified. Quarantined candidate; recipe unmaterialized. Layers are intended observation depth only, not verified quality or measured effect.

Collection properties

Purpose category: Agent Scenarios · Paired Counterfactuals
Items: 120 seeds
Languages: Not recorded
Material format: Restricted runtime references
Controls: 60 no-injection counterfactual pairs
Sources: AgentDojo
Environment requirement: AgentDojo runtime environment and suite dependencies
Acquisition status: Quarantined · No runnable task package or public download is available on this site.

Safe Structure Example

Runtime environment reference

Sample structure only · Not raw seed · Not real answer · Not runnable attack

{
  "_annotation": "[Sample structure only · Not raw seed · Not real answer · Not runnable attack]",
  "scenario_index_reference": ["[scenario reference omitted]", "[scenario reference omitted]", "[scenario reference omitted]", "[scenario reference omitted]"],
  "pair_relation_reference": {
    "target_task_reference": "[reference omitted]",
    "baseline_condition_reference": "[Baseline condition reference omitted]",
    "adversarial_condition_reference": "[Adversarial condition reference omitted]"
  },
  "abstract_outcome_reference": "[Paired condition abstract outcome diff omitted]"
}

Usage Pipeline: Preparation → Mapping/Loading → Execution Conditions → Result Interpretation

Integration Plan

1. Preparation

Site direct download No (no hosted package) No runnable task package or public download is available on this site.

Public upstream Verified public repository View upstream project

Prerequisites Public upstream link is a project entrypoint only; does not provide or guarantee obtaining the 60 pairs referenced here. This site currently provides no acquisition route for these restricted references; map only after you have independently obtained authorized references through your own approval processes.

Metadata only; no runnable package or download is available.

View upstream project

Public upstream link is a project entrypoint only; does not provide or guarantee obtaining the 60 pairs referenced here. This site currently provides no acquisition route for these restricted references; map only after you have independently obtained authorized references through your own approval processes.

Thematic subset entries:

2. Mapping & Loading

Four-step loading & adaptation path

Verify Upstream

Visit upstream AgentDojo repo; check software version and open-source license.

Approved Dev Split

Use approved dev split only; held-out splits must never be used for tuning.

Independent Reference Authorization

Reset & Adapt

Reset sandbox state equally before testing each paired scenario branch.

Field mapping illustration, not an API of this site or upstream

Mapping format illustration only; not an API of this site or upstream; cannot be executed directly

# Mapping format illustration only; not an API of this site or upstream; cannot be executed directly.
# Adapter pseudocode skeleton: adapt to authorized format; not runnable code.
# [Restricted runtime reference mapping worksheet: High-level scenario mapping]
# Operational requirements:
# 1. Version & license verification: Verify upstream AgentDojo repo version and license
# 2. Approved development split: Use approved dev split only; held-out splits must not be used for tuning
# 3. Independent reference authorization: Upstream link is project entrypoint only; no guarantee of 60 pairs; map only after independent approval
# 4. Environment adaptation & reset: Both baseline and adversarial branches must reset from clean state snapshot

PARENT_PAIR_MAPPING = {
    "scenario_index_reference": ["[scenario reference omitted]", "[scenario reference omitted]", "[scenario reference omitted]", "[scenario reference omitted]"],
    "pair_relation_reference": {
        "target_task_reference": "[reference omitted]",
        "baseline_condition_reference": "<MAP_TO_YOUR_AUTHORIZED_BASELINE_CONDITION>",
        "adversarial_condition_reference": "<MAP_TO_YOUR_AUTHORIZED_ADVERSARIAL_CONDITION>",
    },
    "abstract_outcome_reference": "<MAP_TO_YOUR_AUTHORIZED_OUTCOME_CHECK>",
}

3. Execution Conditions

Evaluation input method: High-level paired counterfactual condition comparison across scenarios Environment requirement: AgentDojo runtime environment and suite dependencies

4. Result Interpretation

Core observation dimensions

Abstract defense observation

Not yet measured

Expected： Ratio of untrusted injections blocked without unintended state differences

How to measure： Check trace against expected boundary to verify compliance

Baseline abstract utility

Not yet measured

Expected： Ratio of target abstract outcomes reached under baseline condition

How to measure： Check trace against expected boundary to verify compliance

Observation perspective note: Spans banking, messaging, travel, and workspace scenarios; compares abstract outcomes only, without inferring specific business APIs.

Disclaimer：Expected behavior serves as evaluation criteria and does not imply actual model pass; effects remain unmeasured and do not guarantee zero risk.

Related guides and reports

Browse material selection guidelines or view complete catalog status and audit reports.

Shared loading guide Status center & reports

Version, provenance & governance details

Download and pull are blocked

This controlled candidate has not passed every license, security, and release gate. No pull command or download URL is exposed.

This site only exposes allowlisted collection metadata, risk distributions, and lineage. Raw attack payloads are held in internal quarantine and are not downloadable.

Required pre-release checks, not a completed gate review for this candidate.

Unmet gates (11)

Not formally published (published)
Public visibility not approved (publicVisibility)
Sensitivity does not allow public access (publicSensitivity)
Redistribution review pending (redistributionReviewed)
License evidence link missing (licenseEvidenceUrl)
Abuse reporting channel missing (abuseContactRecorded)
Permitted uses not recorded (permittedUsesRecorded)
Prohibited uses not recorded (prohibitedUsesRecorded)
Governance not approved (governanceApproved)
Release eligibility not approved (releaseEligible)
Download not enabled (downloadEnabled)

These links are verified public HTTPS external resources. Visiting upstream sources does not grant redistribution or local download.

Technical Specifications

Collection ID

candidate-agentdojo-paired

Intended layer

L2·L3

Version

0.1.0-pilot

Collection kind

Candidate pool

Status

Pending

Review status

Pending

Sensitivity

Pending

Items

120

Languages

Pending

Targets

Pending

License

pending review

Files

Path	Media type	Size	SHA-256
No direct file list (metadata mode)

Provenance

Source	Version	Location	Location kind	License
not recorded

Public metadata schema

The marketplace exposes only collection metadata.

Only allowlisted collection metadata is shown. Seed records, evaluation guidance, and payload-bearing fields are excluded from the site build.

Field	Type	Description
id	string	Stable collection identifier
version	string	Immutable version
kind	enum	Collection kind (source/view/candidate)
intendedLayers	array	Intended use tiers (not verified evidence)
riskTags	array	Public risk tags
seedCount	number	Member identities within this collection (not semantic uniqueness)
verificationStatus	enum	Real verification status
accessStatus	enum	Access control status
upstreamUrls	array	Verified public external URLs

Quality

Quality level

Pending

Smoke status

Pending

Validation

Pending

Effect evidence

Pending

Governance & Compliance

Review status

Pending review

Status

Not approved