---

name: retrieval-representation description: "Use Simple Transformers representation, dense retrieval, DPR, BEIR/MSMARCO, hard-negative, and retrieval-evaluation workflows." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Simple Transformers Retrieval and Representation Sub-skill

Use this sub-skill when a task asks for sentence/word embeddings, RepresentationModel.encode_sentences, dense retrieval with query/context encoders, DPR-style training, hard negatives, BEIR/MSMARCO/TREC data, FAISS indexes, or retrieval metrics.

Owns

• RepresentationModel for hidden-state/token/word/sentence representations.
• RetrievalModel / RetrievalArgs for dense retrieval train/eval/predict.
• PretrainRetrievalModel as an advanced pretraining route.
• Retrieval data schemas: query_text, gold_passage, optional title, prediction queries, BEIR/MSMARCO/TREC conversion.
• Optional faiss, pytrec_eval, and beir branches.

Route elsewhere

• Cross-encoder sentence-pair classification reranking: classification.
• T5/monoT5 text-to-text reranking: generative-workflows.
• Extractive QA after passage retrieval: token-and-qa.

Read first

1. API reference for constructors and method map.
2. Data formats before preparing retrieval or representation inputs.
3. Workflows for encode, train/eval/predict, and optional evaluation routes.
4. Troubleshooting for optional dependencies, cache/index files, hard-negative shapes, and dependency compatibility.

Validation helper

python scripts/validate_retrieval_data.py --task retrieval-csv --input train.csv
python scripts/validate_retrieval_data.py --task query-lines --input queries.txt
python scripts/validate_retrieval_data.py --task beir-corpus-jsonl --input corpus.jsonl
python scripts/validate_retrieval_data.py --task beir-queries-jsonl --input queries.jsonl
python scripts/validate_retrieval_data.py --task tsv --input train.tsv

The helper checks file shape only. It does not import FAISS, build indexes, download datasets, train encoders, or run BEIR evaluation.

Key decisions

• Use RepresentationModel when the output is embeddings or token vectors.
• Use RetrievalModel when the task includes query/passage training, hard negatives, or retrieval prediction.
• Decide whether evaluation needs built-in metrics only, pytrec_eval, or full BEIR evaluation.
• Install FAISS only for workflows that actually build or load FAISS indexes.
• Keep CPU for small smoke checks; CUDA is performance acceleration, not required for schema validation.

Verification status

Constructor signatures were inspected with compatibility warnings. Native representation/retrieval examples require checkpoint/dataset downloads and optional dependencies, so default verification uses validators and static/API evidence unless the user approves larger checks.