07 · Outside our own catalog
We ran it somewhere we did not control.
67 datasets
·
7 platforms
·
15 claims
·
metadata we didn’t author
Every number above was measured against a catalog we wrote, with labels applying a policy we wrote. The benchmark says that about itself, and it is the honest weak point. So Attest was also run against DataHub’s own showcase-ecommerce datapack — 67 datasets over 7 platforms, metadata nobody here wrote a line of. Fifteen claims, $0.0077.
Nothing in it is scored: no accuracy, no macro-F1, no confusion matrix. The golden benchmark is a conformance gate where 100% is the expected result, and importing that apparatus onto fifteen unlabelled claims on a foreign catalog would measure nothing. The question was narrower — does Attest produce defensible verdicts against metadata it did not design, and where does it hit its own limits? It found one before a single audit ran.
15 OF 67 UNAUDITABLE — THEN CLOSED, AND RE-MEASURED
A gap our own seed could never expose.
Attest’s GraphQL query had no CorpGroup arm, so a group-owned dataset arrived with an empty owner object and every claim about it was refused. The guard failed closed, which is correct. Its diagnosis — malformed catalog response — was wrong: the response is fine, the query was incomplete, and Attest could not tell the difference. No test here could have caught it. The seed emitted corpuser owners exclusively, so the offline tier, the live tier and the 12-cell matrix were all green and the gap was invisible from inside. It is now closed, and closed seed-first: a group-owned dataset was seeded before the query arm was added, so the fix does not rest on the same seed that hid the problem. The close is measured rather than asserted — a two-arm census over one loaded catalog reads 67/67 readable with the arm and 52/67 without it, 15 recovered, 0 still refused, 0 regressed.
ONE CLAIM WAS RIGHT BY LUCK — NOW REFUSED
Detecting it cost a point, which is the correct direction.
A sentence about a column’s glossary label came back from the decomposer as a schema claim, dropping the term URN entirely. The schema checker then answered a question nobody asked — does this column exist? — and returned the verdict the trial expected. It was banked as a match and named as luck. A deterministic family guard now refuses that transcription outright, as No-Claim, before any checker sees it. So the two figures the receipt never nets have converged rather than improved: 14 of 15 outcomes matched what was predeclared, and 14 of 15 answered the question actually asked — the same fourteen. The match count fell from 15 to 14 because a lucky match became a visible refusal. That is the number getting more honest, not the system getting worse.
The catalog is a near-miss on all three PII signals Attest recognises: a glossary term named literally PII, filed under a node called Classification; contains_pii where the policy names hasPII; and no completeness marker anywhere. So a column named cust_email, carrying a term whose own description reads “Subject to PII handling requirements”, comes back Insufficient-Coverage where a person would say PII. That is a declared position — structure is a declaration, a name is a guess — meeting a real catalog. The trial does not resolve it. It shows the price, with a receipt.
What this does not prove
One small curated datapack is not “works in production”, fifteen claims are not a sample of anything, and the claims were still written by the same person who wrote the checkers — the trial moves the catalog outside our control, not the claims.
The write-up, and the receipts every figure opens in — the trial and the two-arm census. The pre-fix run is kept as the superseded baseline, unedited, so the before/after pair has a before.
