§1The Evaluation Crisis

Enterprise spending on AI is on track to more than triple by the end of the decade, and the binding constraint on that spending is trust. Procurement teams cannot verify what a model can do; they can only read what its creators say it can do. The evaluation layer of the AI industry has three structural failures.

1.1 Contaminated benchmarks

Public test sets find their way into training corpora — through crawling, through "contamination-free" claims that fail under audit, through the sheer difficulty of knowing what a web-scale dataset contains. A model that has memorized a benchmark is indistinguishable from a model that has mastered the skill, until it meets a question it has never seen. Static public datasets cannot make that distinction, because by the time they matter, they are no longer unseen.

1.2 Self-reported quality

Model developers publish their own evaluations. There is no independent replay of the run, no third party that re-executes the model, no penalty when the number fails to hold up in the field. In any other industry where safety or purchasing decisions hang on a measurement, that measurement is made by someone without a stake in the result. In AI, the measurer and the measured are the same entity.

1.3 Frozen leaderboards

Static rankings measure one run, on one snapshot of data, at one point in time. They say nothing about whether a model is still good next month, on different inputs, under adversarial pressure. Quality is a property of a system over time — but the industry's instruments only take single photographs.

§2From Claimed to Proven

RHAURUM's design stance is that quality is a measured property, not a narrative. A measurement system that deserves trust must satisfy five requirements:

1. Blind. The evaluated cannot distinguish scored questions from unscored ones, so selective effort is impossible.
2. Continuous. Evaluation runs around the clock, so quality is a series, not a snapshot.
3. Replayable. Any third party can re-execute a submission and arrive at the same verdict.
4. Burn-once. No item ever repeats, so memorization is worthless by construction.
5. Sybil-resistant. Value requires stake; provable forgery forfeits it.

The remainder of this paper describes the machinery that satisfies all five.

§3How RHAURUM Works

3.1 Three roles

The Forge — the protocol core. It maintains the PRISM reservoir, draws every bundle, schedules sorties, and publishes commit hashes of anchor answers before any pilot takes off.

Pilots — miners. They stake, then fly: run open-weight models against the current bundle and submit outputs plus per-item confidence scores. Pilots never see which items will be scored.

Auditors — validators. They replay pilot inference in sandboxed containers, grade the hidden anchors with CALIBER, and post verdicts together with the evidence: input hashes, output digests, replay logs.

3.2 The sortie lifecycle

A sortie is one full evaluation cycle, run every 60 minutes, aligned to the network clock:

Pilots may resubmit during the flight window; only the final submission of a sortie is scored. The same submission, replayed by any auditor on the same inputs, produces the same score — determinism is a design requirement, not a hope.

§4PRISM: The Sealed Bundle Pipeline

PRISM is the pipeline that turns raw material into flight-ready bundles. It has one job: make sure that no participant, inside or outside the network, can prepare for a sortie in advance.

4.1 CURATE — build the reservoir

Items enter the reservoir from three sources: licensed corpora whose rights are cleared, synthetic items generated under sealed templates, and community submissions that pass curation. Every item carries a reference answer and a grading rubric. The reservoir is the only place in the system where ground truth exists.

4.2 OBFUSCATE — break memorization

Distractor items are adversarially paraphrased: same underlying skill, different surface form, deliberately unlike any phrasing that appears in public training data. A model that has memorized web text fails distractors on sight; a model that has learned the skill passes them.

4.3 SEAL — commit before flight

Before a bundle ships, the anchor answers and rubrics are hashed into a single commit digest, published to the network. After the sortie closes, the anchor set is revealed and any participant can verify that the scored items were exactly the committed ones. Commit-then-reveal makes post-hoc rigging impossible.

4.4 ROTATE — burn once, always

Every flown item is retired permanently. There is no dataset to download because there is no dataset that repeats. An adversary who somehow obtained a past bundle has obtained something that will never be asked again.

§5Blind Anchor Design

Each bundle contains 160 items: 40 scored anchors and 120 distractors. Formatting, difficulty distribution, and presentation are identical across both classes — the only difference is that anchors have sealed ground truth and distractors do not matter for scoring. A pilot who wants the reward must therefore perform uniformly well across the entire bundle, because any item they skip, guess at, or rush might be the one that decides the sortie.

5.1 Entropy of the anchor space

The anchor set is drawn fresh from a reservoir of N items, and each anchor carries V plausible answer choices. The number of possible anchor configurations for a single sortie is:

Even the smallest configuration yields a space of more than 10¹⁹³ possible anchor sets. For comparison, the number of atoms in the observable universe is estimated near 10⁸⁰. Pre-computing answers, building lookup tables, or brute-forcing the bundle is not merely impractical — it is off the table by more than a hundred orders of magnitude.

5.2 Why blindness holds

Blindness is enforced by construction, not by trust: pilots receive only the items, never the anchor map; the anchor map exists only inside the sealed commit until the sortie closes; and the burn-once policy guarantees that even a leaked map from a past sortie is worthless. The only strategy that survives contact with this design is a model that actually performs the task.

§6CALIBER: The Scoring Engine

CALIBER converts a pilot's submission into a single 0–100 score through four weighted components. The weights are fixed protocol parameters; the formulas are chosen so that no single dimension can be farmed to the neglect of the others.

COMPONENT	WEIGHT	MEASURES
Accuracy	0.50	Correctness on hidden anchors, exact or rubric partial credit
Coverage	0.20	Fraction of the bundle actually attempted
Calibration	0.15	Alignment between reported confidence and realized accuracy
Reproducibility	0.15	Stability of outputs under perturbation replays

6.1 Component formulas

The power exponents are the design's teeth. Accuracy is emphasized at 1.5 so mid-tier performance earns mid-tier pay, while near-perfect performance is disproportionately rewarded. Coverage is cubed, so skipping is punished far harder than answering: below a hard floor of 0.60 coverage a submission forfeits the sortie entirely. Skipping is the one move that always works on a blind test — CALIBER prices it out of existence.

6.2 Gaming resistance

• No single-metric farming. Four orthogonal components with fixed weights; maxing one while neglecting another caps the total.
• Confidence must be honest. Inflated confidence is taxed directly by the calibration term.
• Instability is punished. A model that only passes once is replayed; divergence drags C_rep down.
• Empty and trivial submissions die at the floor. Zero coverage forfeits; all-anchor misses score near zero.

§7Ranking and Emissions

7.1 Qualification

A pilot is eligible for emissions only with at least 8 scored sorties in the last 20. This bars drive-by submissions and forces every contender to survive sustained measurement before collecting a single payout.

7.2 Score smoothing

Raw sortie scores are noisy, so RHAURUM ranks pilots on a triangular moving average over the last 12 sorties — the most recent flight carries the most weight, and the average decays to nothing after 12. A single lucky sortie moves a pilot's rank slightly; a single bad one does not erase a career of quality.

Inactivity decays a pilot's standing: after two consecutive missed sorties, their smoothed score loses 3% per missed flight. Ghost leaders fade; active pilots hold the board.

7.3 Quality-squared allocation

Emissions for each sortie are split among the top five qualifying pilots, proportional to the square of their smoothed score:

The quadratic term is deliberate. Being twice as good is worth four times the share — the network concentrates reward on genuine quality instead of spreading it thin. Auditors receive a fixed 30% of emissions for replay work, paid proportionally to verified verdicts posted.

7.4 Slashing

Provable forgery forfeits stake: submitting a model that was trained on a revealed anchor set, colluding identities, or tampering with replay artifacts. Accusations are adjudicated by replay evidence alone — the network does not judge intentions, only the reproducible record.

§8Why Pilots Cannot Cheat

The anti-gaming argument assembles from five independent walls:

1. Blind anchors. The scored 40 of 160 are unknowable during flight — selective effort is indistinguishable from random effort.
2. Sealed commits. Anchor truths are hashed before flight and revealed after; rigging would require breaking the commit, which is published and checkable by everyone.
3. Burn-once rotation. Nothing repeats. The memory of a model — or of a model's owner — has no purchase.
4. Replayable audit. Every submission is re-executed by independent auditors in sandboxed containers; fabricated outputs cannot survive a replay.
5. Stake on the line. Even if the first four walls were climbed, the gain would be emissions while the cost would be the stake — a trade no rational operator takes.

The elegant part is that these walls do not require catching cheaters to be effective. They make cheating unprofitable before it begins.

§9Roadmap

RHAURUM is built in five phases, each hardening the loop before widening it.

PHASE	STATUS	SCOPE
01 · Open-Weight Benchmarking	Active	Standard open models (7B–70B), full loop validation
02 · Custom Serving Pipelines	Next	Pilot-authored stacks: quantization, kernels, fine-tunes
03 · Multimodal & Agentic Evals	Planned	Vision, audio, tool-use bundles with rubric scoring
04 · Consensus Judge	Planned	Ensemble evaluator trained on sortie verdicts
05 · The RHAURUM Seal	Vision	Certification API for enterprise procurement

§10What RHAURUM Produces

10.1 The PRISM reservoir

Every sortie retires 40 sealed anchors with committed ground truth. At 24 sorties per day, the network validates more than 1.4 million evaluation items in its first year — each one flown once, graded once, and permanently retired. That is a benchmark corpus with a property no public dataset has: verifiable blindness.

10.2 The Consensus Judge

The strongest pilot models are ensembled into an evaluator that grades incoming items before they enter PRISM. Evaluation quality compounds with the network that feeds it — the judge is retrained continuously on sortie verdicts, and its own quality is audited the same way every pilot's is.

10.3 The RHAURUM Seal

The end state is a certification layer: an API that returns a model's live, blind-evaluated quality score, its sealed audit trail, and its versioned fingerprint. Hospitals, banks, and procurement desks that today buy models on vendor slide decks will be able to check the label against the record — because the record is public, reproducible, and updated every hour.

§11Conclusion

The AI industry has spent a decade scaling models and almost no time scaling the instruments that measure them. RHAURUM is that instrument: a blind, continuous, replayable measurement network where the only strategy that pays is a model that genuinely performs. The test cannot be studied for, the score cannot be edited, and the results compound into the certification layer the market is missing. Quality stops being a claim on a slide deck and becomes a number on a public record — refreshed every hour, forever.

§RReferences

1. Stanford Institute for Human-Centered AI. AI Index Annual Reports, 2023–2026.
2. Survey literature on benchmark contamination and data leakage in language model training. arXiv, 2023–2025.
3. Industry market analyses of enterprise AI spending and procurement, 2024–2026.
4. Foundational work on decentralized incentive networks and reputation-weighted reward allocation.