Skip to main content
Search evaluation is a separate evidence system from protocol conformance. A relevant result does not prove a payment is valid, and a successful settlement does not prove ranking quality. The repository keeps several cohorts because they answer different questions and must not be merged into one convenient score.

​
Evaluation tracks

The development v2 numbers on Search evaluation are not substitutes for a completed release run. The controlled and ecosystem cohorts also have separate manifests, snapshots, and judgment provenance.

​
Controlled corpus contract

The release fixture set contains:
  • 150 HTTP records derived only for shape from a CDP snapshot;
  • 60 MCP tool records;
  • 45 schema-valid adversarial records; and
  • 45 sparse/cold-start records.
Fixtures use repository-owned .example URLs, deterministic Stellar-shaped addresses, seven-decimal atomic amounts, exact-only payment options for this benchmark version, and a sidecar for category, liveness, provenance, and fixed USD authoring snapshots. The sidecar is not exposed as Bazaar metadata. The 300 fixtures are compiled with the published seller SDK and checked with both upstream Bazaar validators. The wire object contains only official x402 fields. Foreign CDP data is kept outside the repository’s Apache-2.0 claim; committed transformed fixtures do not copy source prose, tags, examples, or schemas.

​
Query and qrel model

The release query set is frozen: 50 release queries and 50 development queries. Queries cover normal developer tasks, semantic paraphrases, network/scheme and price constraints, MCP tools, no-result cases, and adversarial wording. Query wording generated by a model can assist development only; release questions are curated and are never replaced by model output. Each query-resource pair receives a grade:
Hard constraints—resource type, network, scheme, asset, recipient, extension presence, and evaluation-only price/category conditions—are calculated deterministically before semantic judging. The judge cannot turn an $8 resource into a valid result for a max_price_usd: 5 query.

​
Human calibration and model judging

OpenRouter is optional and isolated to the relevance-labeling workflow. The default judge is deepseek/deepseek-v4-flash-0731, temperature zero, strict JSON output, bounded retries, rate-limited concurrency, and resumable cache checkpoints. The API key is read from OPENROUTER_API_KEY only. Before release judging:
  1. deterministically select a stratified 400-pair calibration set;
  2. obtain genuine human grades in a separate import file;
  3. compare model and human grades;
  4. review disagreements and severe disagreements;
  5. publish model ID, revision, prompt hash, timestamp, response hash, and calibration results.
Release requires at least 300 reviewed pairs, quadratic weighted kappa of at least 0.70, and severe disagreement at or below 5%. Model output is never called human review. Pending and placeholder grade-0 rows remain explicitly marked until independent review completes. No-result queries are catalog-absence assertions. They are evaluated by no-result accuracy, not allowed to inflate recall/nDCG by pretending that an empty result has a semantic relevance label.

​
Handwritten v2 design notes

The handwritten benchmark is agent-authored and owner-reviewed, not falsely described as fully human-authored ground truth. Fresh-context authoring agents receive isolated task packs and write only to their assigned staging shard; a separate grading context must not see the retrieval system, rank, score, or authoring context. The project owner accepts or rejects merged artifacts. The labeled core is 100 slots across 20 capability families, with five slots per family. Its target mix is 85 HTTP and 15 MCP resources; 70 exact-only, 22 exact-plus-upto, and 8 upto-only payment sets; about 15 adversarial records; and about 10 sparse records. Four hundred individually authored distractors make the MVP corpus 500 records. The exact family tables and axis assignments live in handwritten-evals/spec/families.md and axes.md. The authoring axes are deliberately anti-correlated: capability family, resource type, input shape, output shape, coverage, freshness, price tier, network, scheme set, and attestation must not become shortcuts for relevance. MCP has additional tuple, schema, transport, and tool-count axes. is_sparse, is_distractor, and adversarial_kind are orthogonal tags, not one overloaded source_class enum. Qrels distinguish an inspected grade-0 pair from an unjudged pair. Exact dense and exact hybrid methods build the candidate pool; production lexical, semantic, and hybrid profiles are scored against that pool rather than defining their own ground truth. The current MVP scope explicitly defers the hardened hash-chain release ledger and the critic workflow; the reports must say so rather than implying those controls ran. The six maintenance invariants planned for CI are equally important as aggregate scores: stable known-item retrieval, bounded paraphrase churn, filter monotonicity, threshold compliance, duplicate resistance, and deletion isolation. A benchmark that improves MRR while violating one of these invariants is not an acceptable search release.

​
Search profiles

Every profile runs through the same production SearchService:
  1. lexical PostgreSQL FTS;
  2. semantic embedding-only;
  3. weighted hybrid RRF; and
  4. hybrid plus a real configured reranker.
Fake providers are allowed for mechanics tests but cannot enter release mode. A missing reranker is an unavailable profile, not a zero-quality reranker run. The production path records model generation, dimensions, provider health, fallback state, and latency. The standard release benchmark keeps the reranked profile in its comparison matrix. The separate handwritten-v2 MVP plan temporarily scores lexical, semantic, and hybrid plus an exact-pool BM25 baseline; it defers a reranked release profile until a real configured reranker is available. These are different release scopes, not contradictory model results.

​
Metrics

The report includes:
  • Precision@5 with denominator five;
  • Recall@20;
  • MRR;
  • nDCG@5 and nDCG@10;
  • bpref and judged@k;
  • hard-filter and evaluation-only constraint violations;
  • no-result accuracy;
  • grade-0 rate;
  • adversarial top-1 and resistance rates;
  • provider concentration;
  • embedding completeness and failures;
  • warm/cold result share;
  • reranker lift; and
  • p50/p95/p99 latency.
Aggregate and per-class quality metrics use deterministic 1,000-sample bootstrap 95% intervals. Queries with no grade-2-or-higher result are excluded from relevance aggregate denominators; no-result accuracy handles them separately.

​
Isolation and reproduction

Release mode creates a random PostgreSQL schema, sets search_path, runs migrations, proves table ownership, seeds the fixture catalog, runs the same HTTP search service used in production, and drops the schema after the run. It never truncates a configured catalog. If isolation cannot be proven, it aborts before seeding.
For the ecosystem track:
Every report records dataset hashes, host/runtime details, PostgreSQL version, model IDs and revisions, unavailable profiles, and limitations. The fetcher records page and aggregate hashes, response provenance, and sampling seed; it does not assume the first page is representative.

​
Release gate

The gate opens only when all of these are true:
  • zero production hard-filter violations;
  • zero invalid indexed records;
  • complete current pool, run, and holdout artifacts;
  • genuine human calibration passes;
  • hybrid improves lexical on the frozen release split;
  • reranking matches or improves hybrid when enabled;
  • perfect no-result accuracy;
  • adversarial fixtures do not dominate top results;
  • no embedding/provider fallback in the release run; and
  • configured p95 targets are met.
The checked-in release-gates report is currently blocked because the final pool snapshot, holdout ledger, calibration/agreement evidence, and final report inputs are incomplete. That status is documented rather than hidden behind the stronger development benchmark numbers. See Search evaluation for measured model values and CLI for the complete command surface.
Last modified on August 10, 2026