---

name: evaluation-and-validation description: "Use mlxtend.evaluate metrics, validation splitters, resampling procedures, model-comparison tests, permutation importance, and counterfactual helpers." disable-model-invocation: true metadata: disco-role: operating license: NOASSERTION

evaluation-and-validation

Use this sub-skill when the task is about evaluating predictions, choosing validation splits, comparing fitted estimators statistically, estimating uncertainty or feature importance, or creating counterfactual examples with mlxtend.evaluate.

Read first

• references/api-reference.md for verified mlxtend.evaluate signatures, return shapes, scorer contracts, and interpretation notes.
• references/workflows.md for decision recipes: metrics/scoring, holdout splitters, bootstrap/OOB/.632, permutation tests, paired model tests, F/McNemar/Cochran tests, permutation feature importance, and counterfactuals.
• references/troubleshooting.md for common shape, scoring, predict_proba, method-name, time-series split, and resampling-cost failures.
• scripts/evaluation_smoke.py to smoke-test the installed API on deterministic tiny CPU examples.

Route here for

• accuracy_score, scoring, confusion_matrix, lift_score, and proportion_difference for prediction metrics and simple proportion tests.
• bootstrap, BootstrapOutOfBag, bootstrap_point632_score, bias_variance_decomp, and permutation_test for uncertainty, OOB/.632 validation, bias-variance checks, and nonparametric tests.
• paired_ttest_resampled, paired_ttest_kfold_cv, paired_ttest_5x2cv, combined_ftest_5x2cv, ftest, mcnemar_table(s), mcnemar, and cochrans_q for model comparison.
• RandomHoldoutSplit, PredefinedHoldoutSplit, and GroupTimeSeriesSplit for sklearn-compatible validation splitters.
• feature_importance_permutation and create_counterfactual for evaluation-time interpretability helpers.

Boundaries and sibling routes

• Do not construct or tune estimators here except as minimal evaluation examples. For mlxtend classifiers, regressors, stacking, voting, and Kmeans, read ../estimators-and-ensembles/SKILL.md.
• Do not cover feature selection or extraction algorithms here. For selectors, PCA/LDA/kernel PCA, preprocessing, and transaction encoding, read ../feature-workflows/SKILL.md.
• Do not use this sub-skill for plotting confusion matrices, split diagrams, learning curves, decision regions, or feature-importance charts. For visualization and utility helpers, read ../plotting-and-utilities/SKILL.md.

Operating reminders

• Most estimator-based routines repeatedly call fit and predict or a sklearn scorer. Use fresh estimator instances when preserving fitted state matters.
• Set random_seed or seed for deterministic validation and keep num_rounds, n_splits, and exact permutation sizes small during exploration.
• Interpret p values against a predeclared significance level; a low p-value rejects the null hypothesis for the chosen test but does not rank models by practical effect size.