> ## Documentation Index > Fetch the complete documentation index at: https://docs.stellarx402.xyz/llms.txt > Use this file to discover all available pages before exploring further. # Evaluation workflow > The complete search benchmark program: smoke tests, controlled fixtures, ecosystem evidence, human calibration, and release gates. Search evaluation is a separate evidence system from protocol conformance. A relevant result does not prove a payment is valid, and a successful settlement does not prove ranking quality. The repository keeps several cohorts because they answer different questions and must not be merged into one convenient score. ## Evaluation tracks | Track | Corpus | Purpose | Current status | | ---------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | | Golden smoke | 12 resources, 15 queries, hand-graded 0-3 judgments | Fast regression test for ranking mechanics and obvious violations. | Checked in and runnable. | | Controlled release benchmark | 300 Stellar-shaped fixtures, 100 queries, 30,000 query-resource pairs | Reproducible release comparison across lexical, semantic, hybrid, and reranked pipelines. | Requires final calibration and release artifacts. | | Development v2 report | 500 records, 100 queries, 1,442 checked-in judgments | Development model bakeoff and production-profile comparison. | Measured development evidence; release gate blocked. | | Ecosystem cohort | Source-diverse CDP/Agent Tools snapshots and probes | Measure behavior on broader, changing metadata without confusing it with controlled ground truth. | Versioned separately with provenance and licensing limits. | The development v2 numbers on [Search evaluation](/concepts/evaluation) are not substitutes for a completed release run. The controlled and ecosystem cohorts also have separate manifests, snapshots, and judgment provenance. ## Controlled corpus contract The release fixture set contains: * 150 HTTP records derived only for shape from a CDP snapshot; * 60 MCP tool records; * 45 schema-valid adversarial records; and * 45 sparse/cold-start records. Fixtures use repository-owned `.example` URLs, deterministic Stellar-shaped addresses, seven-decimal atomic amounts, exact-only payment options for this benchmark version, and a sidecar for category, liveness, provenance, and fixed USD authoring snapshots. The sidecar is not exposed as Bazaar metadata. The 300 fixtures are compiled with the published seller SDK and checked with both upstream Bazaar validators. The wire object contains only official x402 fields. Foreign CDP data is kept outside the repository's Apache-2.0 claim; committed transformed fixtures do not copy source prose, tags, examples, or schemas. ## Query and qrel model The release query set is frozen: 50 release queries and 50 development queries. Queries cover normal developer tasks, semantic paraphrases, network/scheme and price constraints, MCP tools, no-result cases, and adversarial wording. Query wording generated by a model can assist development only; release questions are curated and are never replaced by model output. Each query-resource pair receives a grade: ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} 3 = ideal match 2 = relevant match 1 = marginally related 0 = must not rank / irrelevant / hard-constraint violation ``` Hard constraints—resource type, network, scheme, asset, recipient, extension presence, and evaluation-only price/category conditions—are calculated deterministically before semantic judging. The judge cannot turn an `$8` resource into a valid result for a `max_price_usd: 5` query. ## Human calibration and model judging OpenRouter is optional and isolated to the relevance-labeling workflow. The default judge is `deepseek/deepseek-v4-flash-0731`, temperature zero, strict JSON output, bounded retries, rate-limited concurrency, and resumable cache checkpoints. The API key is read from `OPENROUTER_API_KEY` only. Before release judging: 1. deterministically select a stratified 400-pair calibration set; 2. obtain genuine human grades in a separate import file; 3. compare model and human grades; 4. review disagreements and severe disagreements; 5. publish model ID, revision, prompt hash, timestamp, response hash, and calibration results. Release requires at least 300 reviewed pairs, quadratic weighted kappa of at least `0.70`, and severe disagreement at or below `5%`. Model output is never called human review. Pending and placeholder grade-0 rows remain explicitly marked until independent review completes. No-result queries are catalog-absence assertions. They are evaluated by no-result accuracy, not allowed to inflate recall/nDCG by pretending that an empty result has a semantic relevance label. ## Handwritten v2 design notes The handwritten benchmark is agent-authored and owner-reviewed, not falsely described as fully human-authored ground truth. Fresh-context authoring agents receive isolated task packs and write only to their assigned staging shard; a separate grading context must not see the retrieval system, rank, score, or authoring context. The project owner accepts or rejects merged artifacts. The labeled core is 100 slots across 20 capability families, with five slots per family. Its target mix is 85 HTTP and 15 MCP resources; 70 exact-only, 22 exact-plus-upto, and 8 upto-only payment sets; about 15 adversarial records; and about 10 sparse records. Four hundred individually authored distractors make the MVP corpus 500 records. The exact family tables and axis assignments live in `handwritten-evals/spec/families.md` and `axes.md`. The authoring axes are deliberately anti-correlated: capability family, resource type, input shape, output shape, coverage, freshness, price tier, network, scheme set, and attestation must not become shortcuts for relevance. MCP has additional tuple, schema, transport, and tool-count axes. `is_sparse`, `is_distractor`, and `adversarial_kind` are orthogonal tags, not one overloaded `source_class` enum. Qrels distinguish an inspected grade-0 pair from an unjudged pair. Exact dense and exact hybrid methods build the candidate pool; production lexical, semantic, and hybrid profiles are scored against that pool rather than defining their own ground truth. The current MVP scope explicitly defers the hardened hash-chain release ledger and the critic workflow; the reports must say so rather than implying those controls ran. The six maintenance invariants planned for CI are equally important as aggregate scores: stable known-item retrieval, bounded paraphrase churn, filter monotonicity, threshold compliance, duplicate resistance, and deletion isolation. A benchmark that improves MRR while violating one of these invariants is not an acceptable search release. ## Search profiles Every profile runs through the same production `SearchService`: 1. lexical PostgreSQL FTS; 2. semantic embedding-only; 3. weighted hybrid RRF; and 4. hybrid plus a real configured reranker. Fake providers are allowed for mechanics tests but cannot enter release mode. A missing reranker is an unavailable profile, not a zero-quality reranker run. The production path records model generation, dimensions, provider health, fallback state, and latency. The standard release benchmark keeps the reranked profile in its comparison matrix. The separate handwritten-v2 MVP plan temporarily scores lexical, semantic, and hybrid plus an exact-pool BM25 baseline; it defers a reranked release profile until a real configured reranker is available. These are different release scopes, not contradictory model results. ## Metrics The report includes: * Precision\@5 with denominator five; * Recall\@20; * MRR; * nDCG\@5 and nDCG\@10; * bpref and judged\@k; * hard-filter and evaluation-only constraint violations; * no-result accuracy; * grade-0 rate; * adversarial top-1 and resistance rates; * provider concentration; * embedding completeness and failures; * warm/cold result share; * reranker lift; and * p50/p95/p99 latency. Aggregate and per-class quality metrics use deterministic 1,000-sample bootstrap 95% intervals. Queries with no grade-2-or-higher result are excluded from relevance aggregate denominators; no-result accuracy handles them separately. ## Isolation and reproduction Release mode creates a random PostgreSQL schema, sets `search_path`, runs migrations, proves table ownership, seeds the fixture catalog, runs the same HTTP search service used in production, and drops the schema after the run. It never truncates a configured catalog. If isolation cannot be proven, it aborts before seeding. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} cd facilitator npm run benchmark:generate npm run benchmark:validate OPENROUTER_API_KEY=... npm run benchmark:judge npm run benchmark:import-human -- /path/to/human-review.jsonl npm run benchmark:calibration npm run benchmark:gates TEST_DATABASE_URL=postgresql://... npm run benchmark:evaluate ``` For the ecosystem track: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} npm run benchmark:fetch-ecosystem npm run benchmark:probe-ecosystem npm run benchmark:seed-ecosystem-queries npm run benchmark:validate-ecosystem npm run benchmark:evaluate-ecosystem ``` Every report records dataset hashes, host/runtime details, PostgreSQL version, model IDs and revisions, unavailable profiles, and limitations. The fetcher records page and aggregate hashes, response provenance, and sampling seed; it does not assume the first page is representative. ## Release gate The gate opens only when all of these are true: * zero production hard-filter violations; * zero invalid indexed records; * complete current pool, run, and holdout artifacts; * genuine human calibration passes; * hybrid improves lexical on the frozen release split; * reranking matches or improves hybrid when enabled; * perfect no-result accuracy; * adversarial fixtures do not dominate top results; * no embedding/provider fallback in the release run; and * configured p95 targets are met. The checked-in release-gates report is currently blocked because the final pool snapshot, holdout ledger, calibration/agreement evidence, and final report inputs are incomplete. That status is documented rather than hidden behind the stronger development benchmark numbers. See [Search evaluation](/concepts/evaluation) for measured model values and [CLI](/reference/cli) for the complete command surface.