Skip to main content
The repository includes a development evaluation for the Bazaar search stack. It is separate from protocol conformance: a search score cannot prove that a payment was valid, and a successful payment cannot prove that ranking is good.
The current release-gates report is blocked. The results below are measured development evidence, not a claim that the search release gate has passed.

​
Dataset and method

The current v2 development corpus contains 500 catalog records, 100 queries, and 1,442 relevance judgments in the checked-in report. The model bakeoff uses the development split and 20 measured latency samples per model. The corpus is synthetic and human-reviewed; it is not a live catalog. Judgments use grades 0-3. Hard constraints are evaluated deterministically before semantic grading. The full evaluation records dataset hashes, model identity, dimensions, run files, and limitations. OpenRouter DeepSeek is used only as an optional relevance judge; it does not generate indexed catalog text.

​
Embedding quality

These values come from facilitator/handwritten-evals/reports/embedding-bakeoff-v2.json (generated 2026-08-09). Metrics are development-split aggregates. Higher is better.
Bar chart comparing MRR, nDCG at 10, Recall at 20, and bpref for four embedding models.

Embedding model retrieval quality on the Stellar Bazaar v2 development split.

​
Embedding latency

The same report measured provider latency on 20 samples. Lower is better.
Log-scale bar chart comparing p50 and p95 embedding latency for four models.

Single-query embedding latency from 20 timed samples per model.

These are the two primary views of the same tradeoff: quality and latency. The additional plots below are retained as exploratory diagnostics and are clearly separated from the release-gate evidence.

​
Production search path

The real facilitator search run used PostgreSQL plus the pinned local BGE-M3 embedding path on an isolated database seeded with the benchmark catalog. The development report measured these configurations: The development report measured four production-search profiles. Real lexical scored MRR 0.6121, nDCG@10 0.5334, Recall@20 0.6828, and bpref 0.5068. Real BGE-M3 semantic scored 0.8325, 0.7160, 0.8422, and 0.6837. The shipped hybrid default (0.7/0.3, rrf_k=20) scored 0.6740, 0.6114, 0.7859, and 0.5189; the tuned profile (0.2/0.8, rrf_k=6) scored 0.8441, 0.7241, 0.8795, and 0.6907, respectively. The tuned profile is evidence for a follow-up configuration decision, not an automatic production change. The report explicitly notes that the development corpus is synthetic, judgments are incomplete, and no click or payment conversion validation exists yet.
Horizontal bar chart comparing lexical, dense, and hybrid nDCG at 10.

Development fusion comparison using the benchmark's pool-build stand-in.

Scatter plot comparing nDCG at 10 with p95 latency for four embedding models.

Quality versus p95 embedding latency for the development model bakeoff.

These plots are exploratory development artifacts. The fusion chart uses a pool-build stand-in and must not be read as a production BM25 claim. The quality-latency plot describes the same 20-sample provider measurement shown above; neither chart opens the blocked release gate.

​
Metrics and release gates

The evaluator reports MRR, Recall@20, nDCG@5/nDCG@10, bpref, judged@k, hard-filter violations, no-result accuracy, adversarial top-1 rate, provider concentration, embedding failures, reranker lift, and p50/p95/p99 latency. The release gate requires, among other checks:
  • zero hard-filter violations;
  • zero invalid indexed records;
  • complete current run and pool artifacts;
  • human calibration and adjudication evidence;
  • hybrid improvement over lexical on the frozen release split;
  • reranker improvement or parity when enabled;
  • perfect no-result accuracy;
  • adversarial listings not dominating the top result; and
  • configured latency targets.
The checked-in release-gates-v2.json is currently blocked because pool snapshot, final holdout ledger, calibration/agreement, and final report inputs are incomplete. Do not present the development table as a passed release gate.

​
Reproduce the evaluation

From facilitator/:
For the formal release workflow, follow facilitator/docs/SEARCH-BENCHMARK.md. Keep API keys in the environment. Never commit OpenRouter responses containing credentials or claim generated labels as human labels.
Last modified on August 10, 2026