---

name: inference description: "Load and smoke-check FlagEmbedding embedders, M3 models, and rerankers for retrieval inference." disable-model-invocation: true metadata: disco-role: operating license: MIT

Inference

Use this sub-skill when the task is to load a FlagEmbedding model for inference, encode queries or corpus passages, compute embedding similarity, compute BGE-M3 dense/sparse/ColBERT scores, rerank query-passage pairs, or smoke-check the inference API before a retrieval run.

Do not use this sub-skill for fine-tuning data formats, fine-tuning launchers, training hyperparameters, or checkpoint production. Route those tasks to the sibling fine-tuning sub-skill. Do not use it for benchmark suites, leaderboard evaluation, or metric scripts. Route those tasks to the sibling evaluation sub-skill. For package installation, import failures, and global optional dependency repair, route to the root troubleshooting reference at ../../references/troubleshooting.md.

Start Here

1. Decide whether the model is an embedder, an M3 embedder, or a reranker.
2. Prefer FlagAutoModel.from_finetuned(...) for embedders and FlagAutoReranker.from_finetuned(...) for rerankers when the checkpoint name is in the built-in mapping.
3. For a custom local checkpoint or an unmapped remote checkpoint, pass model_class explicitly and choose the concrete class behavior from references/api-reference.md.
4. For retrieval, use encode_queries(...) for queries and encode_corpus(...) for passages. Query instructions are applied by encode_queries; passages normally should not receive the query instruction.
5. For reranking, pass a pair as ("query", "passage") or a batch as a list of pairs to compute_score(...); validate the returned list or nested list shape before sorting results.
6. For a safe local check, run scripts/smoke_inference_api.py --help first. Running the script with no --model-name performs import and signature probing only. Loading or encoding happens only when --model-name is passed.

Public Entry Points

Embedder auto loader and concrete classes:

• FlagAutoModel
• FlagModel
• BGEM3FlagModel
• FlagLLMModel
• FlagICLModel
• FlagPseudoMoEModel

Reranker auto loader and concrete classes:

• FlagAutoReranker
• FlagReranker
• FlagLLMReranker
• LayerWiseFlagLLMReranker
• LightWeightFlagLLMReranker

Embedder model_class ids:

• encoder-only-base: encoder-only dense embedder, normally FlagModel.
• encoder-only-m3: BGE-M3 dense/sparse/ColBERT embedder, BGEM3FlagModel.
• decoder-only-base: decoder-only LLM embedder, FlagLLMModel.
• decoder-only-icl: few-shot ICL LLM embedder, FlagICLModel.
• decoder-only-pseudo_moe: pseudo-MoE LLM embedder, FlagPseudoMoEModel.

Reranker model_class ids:

• encoder-only-base: encoder cross-encoder reranker, FlagReranker.
• decoder-only-base: decoder-only LLM reranker, FlagLLMReranker.
• decoder-only-layerwise: layer-selectable LLM reranker, LayerWiseFlagLLMReranker.
• decoder-only-lightweight: token-compression LLM reranker, LightWeightFlagLLMReranker.

Loader Choice

Use auto loaders when the checkpoint's effective model name is in the built-in mapping. The effective name is the basename of model_name_or_path; if the basename starts with checkpoint-, the parent directory basename is used. This means a fine-tuned checkpoint directory such as .../my-bge/checkpoint-1000 is mapped as my-bge, not checkpoint-1000.

If auto mapping raises a model-not-found error, do not keep retrying with the same call. Pick the correct model_class, and for embedders also set pooling_method, query_instruction_format, and trust_remote_code rather than relying on mapping defaults. Custom decoder-only and remote-code checkpoints are especially likely to need explicit trust_remote_code=True after code review.

Use concrete classes directly when the architecture is already known or when you need class-specific options such as M3 return modes, ICL examples, pseudo-MoE domains, layerwise cutoff layers, or lightweight compression.

Core Parameters

Use references/api-reference.md for verified signatures and parameter defaults. The parameters most likely to affect inference behavior are:

• normalize_embeddings: normalize dense embeddings before similarity.
• use_fp16 and use_bf16: inference dtype. Use CPU with full precision for conservative smoke checks.
• query_instruction_for_retrieval and query_instruction_format: query-side retrieval instructions for embedders.
• devices: None auto-selects available accelerators then CPU; a string such as "cpu" or "cuda:0" pins one device; a list enables multi-process inference across target devices.
• pooling_method: use cls or mean for compatible encoder-only models; decoder-only embedders require last_token.
• trust_remote_code: keep False unless the checkpoint requires custom model code and that code has been reviewed.
• truncate_dim: truncate output vectors for Matryoshka-style embeddings.
• batch_size, query_max_length, and passage_max_length: embedder throughput and truncation controls.
• Reranker query_max_length, max_length, and normalize: pair scoring truncation and optional sigmoid normalization.
• Layerwise/lightweight reranker cutoff_layers, compress_ratio, and compress_layers: score layer selection and token compression controls.

Expected Outputs

Base embedders return a NumPy array by default. A single input returns a 1-D vector; a list returns a 2-D array shaped like (num_texts, embedding_dim). With convert_to_numpy=False, expect a torch tensor instead.

BGEM3FlagModel returns a dictionary with dense_vecs, lexical_weights, and colbert_vecs. Keys whose return flag is false are present with None values. Use dense_vecs @ dense_vecs.T for dense similarity, use compute_lexical_matching_score(...) for sparse lexical weights, and use compute_score(...) for per-pair M3 mode scores.

Standard rerankers return a list of float scores for a batch. Layerwise and lightweight rerankers return one score list when a single cutoff layer is used, or a list of score lists when multiple cutoff layers are requested. Do not sort documents until this shape is validated.

Workflows

Use references/workflows.md for these recipes:

• Auto and concrete embedder loading.
• Query/corpus encoding and similarity computation.
• M3 dense, sparse, ColBERT, and compute_score workflows.
• Decoder-only, ICL, and pseudo-MoE embedder options.
• Encoder, decoder, layerwise, and lightweight reranker workflows.
• Batch sizing, CPU fallback, and multi-device inference.

Use references/troubleshooting.md when:

• Auto mapping cannot find a model.
• Remote-code or local custom-code loading fails.
• Network/cache behavior is surprising.
• CPU/GPU fallback, dtype, or OOM behavior changes results or fails.
• Output shapes or score modes are misused.

Smoke Checks

Run the bundled helper before writing a larger pipeline:

python sub-skills/inference/scripts/smoke_inference_api.py

The default command imports public inference symbols and prints signatures only. It does not instantiate a model and should not trigger downloads. From the flag-embedding skill directory, load and encode only when an explicit local path or model id is supplied:

python sub-skills/inference/scripts/smoke_inference_api.py \
  --model-kind embedder \
  --model-name MODEL_NAME_OR_PATH \
  --devices cpu \
  --batch-size 2

For a custom local checkpoint, add --model-class, --pooling-method, and --query-instruction-format as needed. For a reranker, use --model-kind reranker; for BGE-M3-specific probes, use --model-kind m3.