---

name: evaluation description: "Compute and troubleshoot Microsoft Recommenders offline metrics for rating prediction, top-k ranking, diversity, novelty, serendipity, and optional Spark evaluation." disable-model-invocation: true metadata: disco-role: operating license: MIT

Evaluation

Use this sub-skill when a user has true interactions and model predictions and needs Recommenders metrics, metric selection guidance, dataframe column checks, or ranking/top-k troubleshooting.

Backend truth for this skill

• Verified in the base CPU scope: Python evaluation module import and metric signatures for rating, ranking, diversity, novelty, serendipity, and top-k helpers.
• Optional and not verified in this CPU scope: Spark evaluation classes. They require [spark], Java/JDK, and a working Spark session.
• Synthetic or docs-only checks cannot prove Spark metrics; keep Spark runtime evidence separate.

Start here

• For metric signatures and required columns, read api-reference.md.
• For rating, ranking, diversity, novelty, and Spark metric recipes, read workflows.md.
• For common metric errors and fixes, read troubleshooting.md.
• To run a tiny no-network metric smoke check, execute:

python sub-skills/evaluation/scripts/metrics_tiny_smoke.py

Run the command from the generated skill root, or adapt the script path to the installed skill location.

Route elsewhere

• Loading, splitting, validating, and filtering interaction data belongs in data-preparation.
• Fitting models and generating predictions or recommendations belongs in modeling.
• Benchmark loops, hyperparameter sweeps, and cloud logging belong in operations-and-tuning.

Working rules

1. Decide whether the task is rating prediction, top-k ranking, classification, or beyond-accuracy evaluation.
2. Ensure true and predicted dataframes share compatible userID and itemID dtypes before calling metrics.
3. For rating metrics, predictions should align with true pairs and include prediction.
4. For ranking metrics, predictions should represent candidate item scores per user; remove seen training items upstream when evaluating recommendation quality.
5. Keep k, threshold, relevancy_method, score_type, and custom column names explicit in reports.
6. Use Spark metrics only when the user is already in a verified Spark environment.