BASELINE EVALUATION

What the pack reveals
about a model.

A baseline report for the python systems reliability collection. Customer results remain private.

BASELINE REPORTPython systems reliability

12environments

3model snapshots

pass@1primary metric

Baseline complete

MODEL COMPARISON

Pass rate across the founding pack

Frontier model A

Strong implementation, weaker recovery reasoning

58%

Frontier model B

Frequent incomplete fixes under concurrency

41%

Open baseline

Useful lower bound for calibration

23%

FAILURE ANALYSIS

Failure clusters make the pack useful for both evaluation and targeted post-training.

Incomplete state recovery

Patch handles the visible failure but leaves corrupted state behind.

37%

Cross-file dependency tracing

Agent changes the primary module without following downstream contracts.

29%

Concurrency regression

Solution passes ordinary tests but fails under interleaved access.

21%

Reproducible by design

Every evaluation pins exact model snapshots, prompts, environment versions, and run settings so the result can be reproduced.

Run the pack on your model.

We will return the score, trajectories, and failure analysis before you decide how to license the pack.