---

name: evaluation-and-search description: "Evaluate Surprise predictions, cross-validate algorithms, split data, tune hyperparameters, inspect cv_results, and run the Surprise CLI." disable-model-invocation: true metadata: disco-role: operating license: BSD 3-Clause

Evaluation and search

Use this sub-skill when a future agent must score Surprise predictions, build train/test or cross-validation splits, run cross_validate, tune algorithms with GridSearchCV or RandomizedSearchCV, inspect cv_results, or evaluate a dataset from the surprise command line.

Boundaries

In scope:

• surprise.accuracy.rmse, mse, mae, and fcp on Prediction lists.
• train_test_split, KFold, ShuffleSplit, RepeatedKFold, LeaveOneOut, and PredefinedKFold.
• cross_validate and the minimal fit_and_score semantics needed to reason about returned measures and timings.
• GridSearchCV, RandomizedSearchCV, best_score, best_params, best_estimator, best_index, and cv_results.
• A safe unbiased tuning/evaluation split workflow.
• The surprise CLI for evaluation on built-in, custom-file, or predefined-fold datasets.

Out of scope:

• Algorithm internals, baseline formulas, and similarity formulas. For algorithm choice or options, use the sibling prediction-algorithms sub-skill.
• Dataset parsing depth, raw/inner id conversion, and built-in dataset cache management. For those, use the sibling data-loading sub-skill.
• Top-N recommendation, precision/recall@k, dump/load, and exported prediction analysis. Use the sibling recommendation-and-analysis sub-skill.
• Benchmark table generation, documentation table generation, notebooks, and maintainer “run all examples” harnesses; they are not runtime evaluation paths.

Read path

1. Start with references/evaluation.md for API workflows and code templates.
2. Read references/cli-reference.md for surprise command-line evaluation and safe quoting.
3. Read references/troubleshooting.md when a split, metric, search, or CLI call fails.
4. Use the bundled scripts as tiny, deterministic smoke checks when validating an environment or demonstrating a workflow:
   • python scripts/cross_validate_smoke.py
   • python scripts/cv_iterators_smoke.py
   • python scripts/grid_search_smoke.py
   • python scripts/unbiased_split_smoke.py
   • python scripts/cli_eval_smoke.py

The scripts create temporary local rating files and do not download built-in datasets.

Task routing quick map

User task	Use	Key reference
“Compute RMSE/MAE/MSE/FCP for predictions”	accuracy.<metric>(predictions, verbose=False)	Metrics
“Split once into train/test”	train_test_split(data, test_size=..., random_state=...)	One train/test split
“Run k-fold CV”	cross_validate(algo, data, cv=KFold(...), measures=[...])	Cross-validation
“Manually control folds”	Iterate for trainset, testset in cv.split(data)	CV iterators
“Tune hyperparameters”	GridSearchCV or RandomizedSearchCV	Search
“Inspect all grid-search results”	gs.cv_results, optionally pandas.DataFrame.from_dict	cv_results
“Tune, then report unbiased performance”	Hold out raw ratings before search; fit on tuning set; test on untouched holdout	Unbiased split
“Evaluate from shell”	surprise -algo ... -load-custom ... -reader ... -n-folds ...	CLI reference

Safety notes

• Prefer local files, dataframes, or predefined folds for examples and tests. Dataset.load_builtin() may prompt for or download data if the cache is missing.
• Keep n_jobs=1 while debugging custom algorithms, tiny examples, or failures; increase parallelism only after a serial run is correct.
• Use explicit random_state or CLI -seed for reproducible folds and algorithm initialization.
• The CLI evaluates Python expressions supplied to -params and -reader. Only pass trusted strings.
• Search objects expose test() and predict() only when refit is enabled; without refit, manually fit best_estimator[measure] before using it.