Pre-production evaluation

Crucible^™

Crucible is an evaluation harness for agentic systems. Capture a full agent run, score it against a versioned rubric with an automated judge, track quality over time, and export evidence – so you ship what survives, not what you hope works.

What it isAgentic evaluation harness

ScoresFull agent trajectories vs a versioned rubric

ExportsEvidence reports for governance

StageWorking harness · keyless demo

The outcome

Before

An agent's real behaviour is unknown until it happens in production. 'It looked fine in a few manual runs' is the whole quality bar, and a regression only shows up once it's live.

After

An agent run can be scored against a versioned rubric before it ships, so you keep only what clears the bar, catch regressions as scores move, and export the evidence. The scoring pipeline described here is the harness and its keyless demo. The state audit this line used to call outstanding was run 2026-08-24 to 2026-08-26 and drove Crucible end to end in a live process; what it found about the audit record – including one attack that was not detected – is published in full.

Crucible is the gate between a promising agent and a trusted one – the pre-production check that lets you put a governed agent into the stack on evidence, not optimism.

How it works

Crucible is built to define what "correct" looks like across a whole agent trajectory, score it automatically, and turn the result into evidence. Five stages, wired end to end.

TRACE

Capture the run

A full agent execution trajectory is captured as an immutable artefact – every step, not just the final answer.

RUBRIC

Define "correct"

Evaluation criteria are versioned, declarative rubrics – goal adherence, output quality, and whatever else matters for that agent – with weights and thresholds.

JUDGE + SCORE

Score it

An automated judge scores each rubric dimension; a weighted composite gives a single pass/fail verdict, with per-dimension rationale and regression detection over time.

REPORT

Export evidence

Results export as evidence reports (JSON / CSV / PDF) for governance and compliance – the artefact you attach when someone asks "how do you know it's safe?"

Stated honestly Crucible scores agent trajectories against rubrics you version. Adversarial injection – actively attacking an agent to find where it breaks – is on the roadmap, documented but not yet built, so we don't claim it as shipped. What we have attacked is our own audit record: five attacks against the audit chain in the production database, four detected and one not, with every result – including the failure – published in full. Attacking the record and attacking an agent are different things, and only the first is done.

Proof

You don't have to take the scoring on faith, and you don't need an API key to see it. Crucible ships a one-command demo that runs the whole pipeline with a deterministic judge – byte-identical output every run.

Example · the keyless demo

crucible demo

# no API keys – a deterministic judge, reproducible output
register agent → load rubric → capture trace → evaluate

goal_adherence ....... 0.86  PASS
output_quality ....... 0.79  PASS
safety ............... 0.94  PASS
────────────────────────────────
composite ............ 0.85  PASS  (threshold 0.75)

The full pipeline, keyless and reproducible – see the scoring end to end before you wire in a real judge.

The other half of the evidence Scoring tells you an agent was good enough to ship. It says nothing about whether the record of what it did is intact. So we attacked our own audit chain in the production database and published all five results: mutation, mid-chain deletion, reorder and re-append were each detected and named the sequence they broke at; deleting entries from the tail was not detected at all. That gap, what catches truncation instead, and why it is not independent custody are set out in full on the audit chain, attacked.

SYNTHERA is the trust layer for multi-agent systems: every agent gets a verifiable identity, scoped authority and a tamper-evident record, so software from different teams, vendors and frameworks can act on each other’s behalf without custom glue between every pair.

Talk to us about Crucible.

Talk to us about Crucible