---

name: evaluation description: "Routes PyTorch Metric Learning questions about AccuracyCalculator, testers, and nearest-neighbor inference." disable-model-invocation: true metadata: disco-role: operating license: MIT

Evaluation and inference

Use this sub-skill when the user wants to score an embedding space, compare query/reference splits, or perform nearest-neighbor lookup after training.

Typical triggers

• "How do I compute precision@1, mAP, R-precision, NMI, or AMI?"
• "How do I use GlobalEmbeddingSpaceTester or the other testers?"
• "How do I do nearest-neighbor search on embeddings?"
• "How do I save or reload the faiss index?"
• "How do I compare query and reference splits or use a custom label-comparison function?"

In scope

• Accuracy: AccuracyCalculator and its metric family (precision_at_1, r_precision, mean_average_precision, mean_average_precision_at_r, mean_reciprocal_rank, NMI, AMI).
• Testers: GlobalEmbeddingSpaceTester, WithSameParentLabelTester, GlobalTwoStreamEmbeddingSpaceTester, and BaseTester.
• Inference: InferenceModel, MatchFinder, FaissKNN, FaissKMeans, CustomKNN.
• Tiny in-memory datasets for evaluation smoke checks, including EmbeddingDataset.

Out of scope

• Choosing the loss/miner stack belongs in components.
• Training loop, logging, and checkpointing belong in training.
• Dataset download and sampler construction belong in data unless the question is purely about evaluation input shape.

How to use this sub-skill

1. Read references/evaluation-and-inference.md for the metric map, tester usage, and inference search patterns.
2. Run scripts/smoke_evaluation.py when you want a tiny confirmation that AccuracyCalculator, a tester, and InferenceModel still work together.
3. Read references/troubleshooting.md when the failure mentions faiss, k, custom label comparison, or an untrained index.
4. If the user is still deciding which loss/miner stack to train, route to components before finalizing the evaluation settings.

Common routing decisions

• If the user only needs a metric definition or retrieval score, stay here.
• If the user asks how to choose a validation hook or checkpoint policy, route to training.
• If the user asks for the right dataset split or sampler input before evaluation, route to data.

Useful public facts

• AccuracyCalculator can include or exclude metrics, and k can be None, a positive integer, or "max_bin_count".
• ref_includes_query=True is the right choice when the query set is part of the reference set.
• GlobalEmbeddingSpaceTester is the usual default tester for embedding retrieval tasks.
• WithSameParentLabelTester expects hierarchical labels.
• GlobalTwoStreamEmbeddingSpaceTester expects datasets that yield (anchor, positive, label).
• InferenceModel defaults to a faiss-backed k-NN index if you do not override knn_func.

Read next

• references/evaluation-and-inference.md for the metric, tester, and inference reference.
• references/troubleshooting.md for faiss, k, and label-comparison failures.
• scripts/smoke_evaluation.py for a tiny evaluation and inference smoke check.