---

name: dense-encoding description: "Use Pyserini dense encoders, Faiss search, Lucene dense/HNSW routing, hybrid retrieval, and dense backend troubleshooting." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Dense Encoding

Use this sub-skill when a task involves encoding corpora or queries, choosing dense encoder classes, searching Faiss indexes, searching Lucene dense/HNSW indexes, combining dense and sparse retrieval, selecting CPU/GPU backends, using OpenAI embeddings, or diagnosing missing faiss and model-backend failures.

Route First

• Send Python, Java, package installation, optional dependency installation, and broad runtime checks to ../install-and-runtime/SKILL.md.
• Send Lucene sparse/impact indexing, local Lucene HNSW index construction, analyzers, query builders, index readers, and document fetching details to ../index-search-fetch/SKILL.md.
• Send qrels, run evaluation, run conversion, interpolation, and fusion-result scoring to ../evaluation-and-fusion/SKILL.md.
• Send REST/MCP dense index aliases and server configuration to ../serving-and-agent-tools/SKILL.md.

Default Workflow

1. Validate dense input files with scripts/validate_dense_jsonl.py before any model command.
2. Choose an encoder/backend from references/model-and-backend-reference.md; prefer --device cpu unless GPU availability is confirmed.
3. Build commands with scripts/dense_cli_builder.py and review the printed command before running it.
4. For corpus encoding, use python -m pyserini.encode with input, output, and encoder subcommands.
5. For Faiss retrieval, either encode directly with output --to-faiss or build a Faiss index from encoded JSONL, then search with python -m pyserini.search.faiss.
6. For Lucene dense retrieval, search with python -m pyserini.search.lucene --dense --hnsw or --flat; route index-building details to index-search-fetch.
7. For hybrid retrieval, pair a FaissSearcher with a sparse Lucene searcher and tune fusion parameters; route final run evaluation to evaluation-and-fusion.

References

• references/encoding-and-dense-search.md covers JSONL schemas, encoding, Faiss indexing/search, Lucene dense search, and hybrid retrieval.
• references/model-and-backend-reference.md maps encoder classes, devices, OpenAI, multimodal, UniIR, and encoded-query choices.
• references/troubleshooting.md covers missing faiss, Torch CPU/CUDA mismatch, model downloads, OpenAI keys, JSONL field errors, dimension mismatches, and hybrid alignment failures.

Safety Defaults

• Do not run model commands until the user has confirmed model downloads, local model paths, cached weights, or OpenAI credentials as appropriate.
• Do not assume Faiss indexes store raw document text; use a companion Lucene index for content fetch workflows.
• Do not assume Faiss GPU support from faiss-cpu; use CPU defaults unless a compatible GPU build is available.