---

name: data-preparation description: "Prepare and validate Tevatron JSON/JSONL datasets, rankings, qrels, hard-negative inputs, rerank inputs, and ranking conversion outputs." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Tevatron Data Preparation

Use this sub-skill when the task is to prepare, inspect, validate, repair, or convert files before Tevatron training, encoding, retrieval evaluation, or reranking. It covers local JSON/JSONL loading, Hugging Face dataset loading, train/corpus/query schemas, ID-list data with separate corpora, qrels/rankings, hard-negative prerequisites, rerank input bridges, and tiny local ranking conversions.

Fast Path

1. Identify the downstream consumer:
   • Retriever training accepts either embedded positive_passages/negative_passages, compact positive_document_ids/negative_document_ids plus a corpus, or pre-tokenized rows.
   • Retriever encoding accepts query rows with query_id plus query_text or query, or corpus rows with docid plus text/media fields.
   • Reranker inference accepts one query-document pair per JSONL row with query_id, query, docid, title, and text.
   • Retrieval evaluation and hard-negative mining consume ranking/qrels files whose delimiters must match the target tool.
2. Decide loader mode:
   • Local JSONL: use --dataset_name json --dataset_path <file.jsonl>.
   • Local ID-list training: add --corpus_name json --corpus_path <corpus.jsonl> so document IDs can be resolved.
   • Hosted Tevatron/Hugging Face data: use --dataset_name <namespace/name> with optional --dataset_config, --dataset_split, and cache flags.
3. Validate local files from this sub-skill directory before expensive runs:
   • python scripts/validate_tevatron_jsonl.py --kind train --input train.jsonl --corpus corpus.jsonl
   • python scripts/validate_tevatron_jsonl.py --kind corpus --input corpus.jsonl
   • python scripts/validate_tevatron_jsonl.py --kind query --input queries.jsonl
   • python scripts/validate_tevatron_jsonl.py --kind ranking --input rank.tsv --corpus corpus.jsonl
   • python scripts/validate_tevatron_jsonl.py --kind qrels --input qrels.txt --corpus corpus.jsonl
   • python scripts/validate_tevatron_jsonl.py --kind rerank --input rerank.jsonl
4. Convert retrieval rankings when evaluation or submission tooling needs another format:
   • python scripts/convert_result_to_trec.py --input rank.tsv --output rank.trec
   • python scripts/convert_result_to_marco.py --input rank.tsv --output rank.marco.tsv
5. Route after validation:
   • Training command assembly: ../training/.
   • Embedding/search execution and ranking generation: ../encoding-retrieval/.
   • Pairwise reranker scoring: ../reranking/.
   • Media-model dependencies, Qwen/ColPali/DSE, and optional multimodal execution: ../multimodal-llm/.

Key References

• references/data-formats.md: accepted record schemas, loader flags, qrels/ranking formats, local-vs-hosted dataset decisions, and multimodal-ish fields.
• references/hard-negatives.md: hard-negative mining inputs, top-k/min-count requirements, ID consistency checks, and rerank input bridge requirements.
• references/troubleshooting.md: fixes for malformed JSONL, missing IDs/text, corpus pairing mistakes, delimiter mismatches, hard-negative shortages, and missing media assets.

Bundled Scripts

• scripts/validate_tevatron_jsonl.py: static validator for train, corpus, query, rerank, ranking, and qrels files; it performs no Tevatron imports, model loads, or network calls.
• scripts/convert_result_to_trec.py: standalone converter from Tevatron-style rankings to six-column TREC run format.
• scripts/convert_result_to_marco.py: standalone converter from Tevatron-style rankings to MS MARCO qid<TAB>docid<TAB>rank format.

Safe Defaults

• Keep IDs as strings across train rows, corpus rows, rankings, qrels, and rerank inputs.
• Keep ranking files grouped by query in desired rank order before conversion; converters recompute rank per contiguous query block.
• For default train_group_size=8, plan at least seven usable negatives per query after removing positives and duplicate/invalid docids.
• Treat hosted datasets as network/cache-dependent unless the working environment already has the dataset cached.
• Check media path fields with --assets-root before multimodal execution; a schema-valid row can still fail later if assets are missing.

Boundaries

• Do not assemble or launch model training commands here; route to ../training/.
• Do not encode embeddings, run FAISS search, merge shards, or evaluate retrieval here; route to ../encoding-retrieval/.
• Do not score reranker pairs or select reranker model arguments here; route to ../reranking/.
• Do not execute multimodal or LLM retrievers here; only validate fields/assets and route execution to ../multimodal-llm/.