Diagnose partial writes, races, and broken consistency guarantees.
DESIGN PARTNER COLLECTION
Python systems reliability
A focused pack of repository-level environments that tests whether coding agents can diagnose and repair failures in unfamiliar Python systems.
12versioned environments
3private sources
Binarydeterministic reward
OCIportable delivery
WHAT IT MEASURES
Reliability beyond isolated code generation.
Recover valid state from malformed, incomplete, and adversarial inputs.
Follow failures across realistic package and configuration boundaries.
Fix the target failure without breaking existing public behavior.
ENVIRONMENT 01
Atomic cache recovery
Repair a concurrent cache after a partial journal write while preserving its public API and consistency guarantees.
START STATEPrivate repository snapshotPinned source and dependencies
SUCCESSAll hidden checks passKnown-bad fixes rejected
REWARD1.0 or 0.0Executable, deterministic
QUALITY EVIDENCE
Evidence, not promises.
Every released environment must satisfy the same acceptance standard.
Deterministic reward
Every outcome resolves to an executable pass or fail contract.
Known-bad rejection
Adversarial and shortcut solutions are tested before release.
Reproducible runtime
Pinned, resettable containers with the network disabled by default.
Independent review
Two reviewers inspect specification, solution, tests, and difficulty.
Rights record
Every source and derived artifact is tied to a documented grant of rights.
Baseline evidence
Multiple model snapshots establish difficulty and expose failure modes.
BASELINE CALIBRATION
Designed to separate current models.
Calibration shows that implementation ability is not the main bottleneck. Models most often fail incomplete state recovery and cross-file dependency tracing.
Open evaluation reportFrontier model A58% pass@1
Strong implementation, weaker recovery reasoning
Frontier model B41% pass@1
Frequent incomplete fixes under concurrency
Open baseline23% pass@1
Useful lower bound for calibration
DELIVERY
Everything required to run the pack.
Versioned OCI environments with reset, run, and score interfaces.
Verifiers, hidden checks, baseline report, and release documentation.
Non-exclusive, exclusive, evaluation-only, or hosted access by agreement.
Start with a controlled evaluation.
Request access Run an environment against your model before committing to a license.