---

name: core-indexing-retrieval description: "Build a BM25S sparse index and retrieve ranked document IDs, scores, or position-aligned application records with the low-level BM25 API." disable-model-invocation: true metadata: disco-role: operating license: MIT

Core indexing and retrieval

Use this route when the downstream task needs direct control of bm25s.BM25: construct an eager sparse BM25 index, score tokenized queries, select top-k results, or map result positions back to metadata. This route owns the low-level index/retrieval contract and the scoring variants. Route text splitting, stopwords, stemming, and tokenizer-vocabulary persistence to tokenization-and-stopwords; route index save/load, mmap, and JSONL corpus files to persistence-and-corpus-io. Optional Numba retrieval and JAX top-k selection are acceleration choices, not prerequisites for the CPU workflow.

Minimal safe recipe

import bm25s

texts = ["red cat likes purr", "blue dog likes play", "fish swim in water"]
corpus_tokens = bm25s.tokenize(texts, stopwords=None, show_progress=False)
retriever = bm25s.BM25()  # k1=1.5, b=0.75, delta=0.5, method="lucene"
retriever.index(corpus_tokens, show_progress=False)
query_tokens = bm25s.tokenize(["cat purr"], stopwords=None, show_progress=False)
result = retriever.retrieve(query_tokens, k=2, show_progress=False)
print(result.documents.shape, result.scores.shape)  # (1, 2), (1, 2)

result is a bm25s.Results named tuple with .documents and .scores. Without a display corpus, documents are integer positions. For a ready-to-run metadata fixture, use scripts/metadata_retrieval.py.

Choose the index input

• Token strings: pass list[list[str]] when you already have a stable token representation. index builds a vocabulary from these tokens.
• Token IDs: pass list[list[int]] for a numeric corpus. The index creates a mapping from the supplied external IDs to dense internal columns; retrieve integer queries from that same ID space.
• Vocabulary-carrying IDs: pass a bm25s.tokenization.Tokenized object, a (ids, vocab_dict) pair, or an object exposing .ids and .vocab. The first member is the per-document ID lists and the mapping must be the corpus/query vocabulary.
• Empty documents: keep create_empty_token=True unless the application deliberately handles documents or queries with no surviving tokens. With numeric corpora the sentinel is incorporated before scoring; without a usable sentinel, an all-unknown integer query raises.

Do not tokenize corpus and query independently into integer IDs and assume the IDs match. Either pass string tokens to both retrieval calls or reuse the tokenizer state/vocabulary described by the sibling route.

Retrieval contract

retrieve accepts batches (list[list[str]], list[list[int]], or Tokenized) and returns two-dimensional arrays of shape (number_of_queries, k). Use return_as="tuple" (default) for Results, or return_as="documents" for only the document array. corpus= is display data, not the indexed text: position i must describe indexed document i; if omitted, the retriever's BM25(corpus=...) value is used. See references/api-reference.md for exact signatures, shapes, and parameter semantics.

Start with k <= number_of_indexed_documents, a nonnegative integer, and sorted=True. n_threads=0 is the deterministic simple CPU path; n_threads=-1 uses all reported CPUs, and positive values use a thread pool. chunksize matters only to the threaded NumPy path. backend_selection="auto" may use JAX when available; select "numpy" for the explicit minimal dependency path.

A weight_mask must be a one-dimensional NumPy array with exactly one value per indexed document. It multiplies document scores by position; zeroed documents can still appear when ties or insufficient positive candidates make that unavoidable. It is not a replacement for a filter that guarantees exclusion.

Output and reproducibility

• Keep sorted=True when a ranked list is part of the user-facing contract; use sorted=False only when order does not matter.
• Treat document IDs as row positions, not stable external IDs, unless corpus supplies those external records.
• Record the model method, IDF method, k1, b, delta, dtypes, and selected backends with benchmark results.
• For smoke checks, use a tiny local fixture, show_progress=False, and the explicit NumPy/CPU choices.
• Compare scores as well as IDs when validating a variant or a changed vocabulary.

Scoring decisions

The constructor defaults to Lucene-style BM25. method selects the term-frequency variant: "robertson", "lucene", "atire", "bm25l", or "bm25+". idf_method=None follows method, and can be set independently to one of those supported scoring methods. delta affects BM25L/BM25+; it is ignored by the first three variants. Keep k1 and b explicit when reproducing a benchmark. Invalid methods fail during index construction, so validate them before expensive indexing. Formula and variant details are in references/api-reference.md.

Index state and low-level hooks

Indexing is eager: the model computes document-token contributions into CSC arrays. Confirm retriever.scores["num_docs"] after index and keep retriever.vocab_dict with the query producer. get_tokens_ids and get_scores are useful for a one-query diagnostic; get_scores_from_ids is an unchecked lower-level primitive. build_index_from_tokens and build_index_from_ids are extension hooks for specialized pipelines, not substitutes for a normal index call. Do not mutate scores, vocab_dict, or unique_token_ids_set between indexing and retrieval unless rebuilding the complete compatible state.

Boundary cases to handle explicitly

• retrieve([[]], k=...) is supported and returns top-k zero scores; ties are not meaningful rankings. Prefer this over calling the lower-level get_scores([]) directly.
• Unknown string tokens are omitted from a query and can produce all-zero scores. Unknown integer IDs are filtered by retrieve; if all are unknown and no empty-token sentinel is available, it raises a ValueError.
• get_scores_from_ids is lower-level and does not filter; a too-large ID raises a ValueError about the index vocabulary. Use nonnegative IDs from the same vocabulary.
• An invalid return_as raises a ValueError; a negative k reaches top-k validation and fails, while k=0 produces empty result columns. k > num_docs is rejected early.
• The current retrieval implementation does not validate a supplied display corpus length. Validate len(corpus) == retriever.scores["num_docs"] yourself; a short corpus can raise IndexError, and a long or reordered corpus can silently return misleading records.
• A mask must be an actual numpy.ndarray, one-dimensional, and length num_docs; wrong type, rank, or length raises a ValueError.

See references/troubleshooting.md for install/import, optional-backend, data alignment, method, and workflow failures. Use references/workflows.md for metadata, numeric-ID, variant comparison, masking, and arbitrary-cwd recipes.

Handoff checklist

At minimum, a downstream handoff should include the query count, result shape, and whether returned documents are IDs or application records. This makes an empty result row and a positional metadata mismatch distinguishable.

1. Record the token representation and the exact corpus/query vocabulary strategy.
2. Record num_docs, k, return_as, sorting, and whether display metadata is attached or supplied at retrieval.
3. Validate display corpus alignment and masks before calling retrieve.
4. Keep the required backend CPU/NumPy unless an optional accelerator was deliberately selected and checked.
5. If an index must be written or reopened, hand off to persistence-and-corpus-io rather than reproducing its file contract here.