Crucible · the audit record

We attacked our own audit chain. One attack was not detected.

On 2026-08-24 five attacks were run against Crucible's audit chain in the production database, from a process that had written none of the entries. Four were detected and named the sequence they broke at. The fifth — deleting entries from the end — was not detected, and nothing about the chain detects it. All five results are below, including that one.

What was attacked

Crucible records what it did to a hash-linked table: each entry commits to the one before it, every stored column is inside the hash, and sequence numbers are gap-free from 1. Verification recomputes the whole chain from genesis and asserts contiguity, so an attacker has to defeat linkage and numbering and recomputation at once.

The entries were written by the deployed service as ordinary tool calls, into the production database. They were verified by a different process — a separate job, a separate container, started afterwards, holding none of the writer's state. Each attack was undone and the restore itself verified before the next one ran. That separation is of process, not of identity: the verifying job runs under the same service account as the service that wrote the entries.

Scope. This is Crucible's own audit chain and this page is about that one record. It is not a claim about any other component's ledger.

The five attacks

AttackDetectedCaught atHow
Mutate one entry A written row’s payload edited directly in the database. Yes seq 3 chain_hash mismatch on recomputation from genesis
Delete an entry from the middle One row removed from the interior of the chain. Yes seq 4 sequence is not contiguous — expected seq 3, found 4
Reorder Payloads swapped between two adjacent entries. Yes seq 3 chain_hash mismatch on recomputation from genesis
Excise and re-append An entry removed from the interior and its content added back at the end. Yes seq 4 sequence is not contiguous — the gap remains where the row was
Truncate the tail The last entries deleted from the end of the chain. No NOT DETECTED. Verification returns ok, exit 0.
Observed · the attack job, against the production chain
crucible-chain-attack
[PASS] chain written by the DEPLOYED SERVICE verifies from genesis in a process that did not write it
        ok=True entries=6 head_seq=6
[PASS] MUTATION of seq 3 detected, naming the sequence
        ok=False at_seq=3 reason=chain_hash mismatch — this entry was mutated after it was written
[PASS] DELETION of seq 3 detected
        ok=False at_seq=4 reason=sequence is not contiguous: expected seq 3 at position 2, found 4
[PASS] REORDER (payload swap between seq 3 and 4) detected
        ok=False at_seq=3 reason=chain_hash mismatch
[PASS] REINSERT (seq 3 excised, content re-appended at 7) detected
        ok=False at_seq=4 reason=sequence is not contiguous
[FAIL] TAIL TRUNCATION detected by the chain alone
        ok=True entries=4 head_seq=4
[PASS] TAIL TRUNCATION detected AGAINST AN EXTERNAL RECORD (witness seq=6)
        ok=False at_seq=4 reason=head is seq 4, below the externally recorded head seq 6

The FAIL line is expected and is not a broken test — it is the gap, reported as one.

About the first line, and the harness that produced it

The opening [PASS] asserts that the chain was written by the deployed service and verified by a process that did not write it. For the run above that holds: the chain head moved from 3 to 6 during it, so three entries were genuinely appended by the service.

But that assertion cannot fail correctly, and we found out afterwards. The harness names its test rubrics from a tag that is not set on the job, so every run reuses the previous run's names; the application correctly refuses a duplicate before saving, so nothing is written and no audit entry is created. The harness does not notice, because it counts a call as a write when the response is HTTP 200 and the body has no "error" in it — and a tool-level failure comes back as "isError": true inside a 200, which that test does not match.

So a second run of this drill writes nothing, reports three writes, and passes the same first line against entries from the day before. It is a defect in the evidence-gathering, not in the chain or the anchor, and it is recorded and deliberately not fixed — fixing it should be done and then observed failing to be fooled, not slipped in beside a deploy. We are stating it because a page that publishes a PASS line owes you the reliability of the thing that printed it.

The attack that was not detected

Not detected · exit 0 Deleting the last entries leaves a chain that is contiguous from 1, correctly linked, and recomputes cleanly. Verification returns ok and exits 0. Nothing that remains contradicts the deletion. A hash chain proves the internal consistency of the rows that are present; it says nothing about rows absent from the end.

This attack was not adjusted until it passed, and it is not omitted. It is held in place by a test that asserts the failure: if truncation ever starts being detected by the chain, that test goes red and every claim on this page has to change.

Read the next section knowing this one. Truncation is caught below, and it is caught by a second record disagreeing with the database — never by the chain. The chain contributes nothing to that detection.

What does catch truncation

An hourly job records the chain's head — its sequence number and hash — to a log store outside the database, then verifies the chain against the highest head it has previously recorded. A short chain now disagrees with a record the database does not contain, and that disagreement is the detection.

The non-zero exits are deliberately not the same failure, because an operator paged for one goes looking for the wrong thing — and the third of them is not a failure of the chain at all, but of the check reaching its own witness:

ExitDetected byWhat it means
4 CHAIN_BROKEN the chain, unaided entries were altered, deleted mid-chain, reordered or reinserted — the evidence is still in the table
5 ANCHOR_UNAVAILABLE neither — the check did not complete the anchor store could not be read. Non-zero on purpose: a check that cannot reach its witness has not passed — it has not run, and reporting that as success is how a mitigation rots into decoration without anyone noticing
6 ANCHOR_DISAGREEMENT only the external record entries are missing from the tail; the chain itself verified perfectly from genesis

That the exit 6 comes from the record and not from the chain was proven by driving the real command twice against the same truncated database, differing only in whether the external record was supplied. Withhold the record and the identical attack exits 0:

Observed · the same database, twice
crucible verify-chain
=== TRUNCATE tail, WITNESS WITHHELD ===
exit=0
{ "ok": true, "entries": 2, "head_seq": 2 }

=== SAME truncated database, WITNESS SUPPLIED ===
exit=6
CRUCIBLE: AUDIT CHAIN DISAGREES WITH THE EXTERNAL ANCHOR at seq 2: head is seq 2,
below the externally recorded head seq 5 — entries were TRUNCATED from the tail.
NOTE: this was detected by comparison with an external record, NOT by the chain;
the chain alone cannot detect tail truncation

The real command, subprocessed, against a test database — not the production chain and not an operational rehearsal. Exit 6 was observed separately on 2026-08-25 against the production chain, from the same image digest and the same command line the hourly job runs.

What the anchor is not

Stated because it is load-bearing The anchor is written by the same identity that writes the chain. The check reads the highest head ever recorded rather than the most recent, so a forged anchor above the real head raises an alarm and a forged anchor below it is ignored — the correct failure direction. That is not independent custody. Real separation needs a distinct identity with no write access to the chain, and it is not yet built.

The detection window is bounded at both ends, and both bounds are real:

  • entries written since the last hourly run are unwitnessed — truncating back to a point above the last recorded head is not detected, because nothing yet claims those entries existed;
  • detection ages out with log retention — truncation to a point older than the oldest surviving record is not detected.

And anyone holding write access to both the database and the anchor store defeats the check completely: the result is indistinguishable from a shorter honest history.

What the chain does not prove

Four things, and the first is the most important:

  1. That any entry was ever written. A change whose audit write failed leaves no row at all, and the chain over the remaining rows verifies perfectly. The chain will faithfully prove the integrity of a record that is missing an event.
  2. That a recorded entry is true. An authorised writer recording something false produces a perfectly valid link. This is integrity of the record, never accuracy of what was recorded.
  3. That the tail has not been truncated. See above.
  4. Anything outside the chain. It covers Crucible's own audit events and says nothing about agent identity or evaluation results.

One more, because it is the kind of thing that gets assumed: the actor recorded on an entry is a string supplied by the caller. The chain proves that string was not altered afterwards. It proves nothing about who supplied it.

What you can check, and what you cannot

You cannot reproduce this run. These attacks ran against a production database from inside a private network, and the Crucible repository is not public. What is published here is the record of what was done and what it returned — including the result that did not hold — not a check you can run for yourself.

That distinction matters on this site, so it is stated rather than left to be inferred. The claims we ask you to verify yourself, rather than take from us, are the open standard's: reproduce the frozen conformance vectors from the published packages, or verify a signed action in your browser and watch a tampered one reject.

Ask us what else we attacked.

The full audit, including the findings we filed against ourselves and did not fix, goes to anyone evaluating this seriously.

Talk to us