---

name: evaluation-and-export description: "Evaluate MedMNIST predictions and export 2D or 3D datasets safely, including metric calculation, standard result filenames, montages, CSV/image output, and the Fire CLI." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

MedMNIST evaluation and export

Use this skill when the task concerns Evaluator, getAUC, getACC, score shapes, metric interpretation, standard result CSV names, parse_and_evaluate, MedMNIST2D/3D.save, montage, PNG/GIF/CSV export, or the public save and evaluate CLI commands. This skill is for evaluation and output handling, not for loading arbitrary datasets or training models.

Route and safety boundary

1. First identify the flag, split, task, score shape, and root. The split must be train, val, or test; y_score.shape[0] must equal the label count. Use INFO[flag]["task"] to choose binary, multilabel, multiclass, or ordinal handling. For registry and NPZ-loading questions, route to the sibling dataset-loading skill instead.
2. Use an explicit, already-created root for programmatic evaluation. The constructor never creates a missing custom root and never downloads unless the dataset class is constructed with download=True.
3. Keep result files and exports in a new, isolated output directory. The evaluator does not create save_folder; create it first. Do not use bulk download, default-root cleanup, broad development test(), or network downloads as part of a smoke check.
4. Scores are probabilities/scores, not class labels. Binary and multilabel scores are thresholded only for ACC; multiclass and ordinal predictions use argmax. Preserve row order and do not add a CSV header to an evaluator input file.

For a compact API contract, see references/api-reference.md. For CLI syntax and network/destructive boundaries, see references/cli-reference.md. For reproducible evaluation and export procedures, see references/export-recipes.md. For failures and recovery, see references/troubleshooting.md. Run the safe local check with scripts/medmnist_smoke.py.

Evaluation procedure

Direct Evaluator use

Use the installed API, with the dataset NPZ already present in root:

import numpy as np
from medmnist import Evaluator

evaluator = Evaluator("pneumoniamnist", "test", size=64, root="./data")
y_score = np.asarray(...)                 # N or N x 2 for binary
assert y_score.shape[0] == evaluator.labels.shape[0]
metrics = evaluator.evaluate(y_score, save_folder="./results", run="run1")
print(metrics.AUC, metrics.ACC)

Evaluator(flag, split, size=None, root="~/.medmnist") maps None and 28 to the unsuffixed NPZ. A non-28 size is written as flag_size.npz; the available 2D sizes are 28/64/128/224 and 3D sizes are 28/64. The root must exist and contain the exact file. evaluate computes a Metrics(AUC, ACC) namedtuple and, when save_folder is supplied, writes the score matrix with pandas using no header and an index column. Pass a stable run value when the filename must be reproducible.

Shape and metric decision table

INFO[flag]["task"]	Accepted score shape	AUC	ACC
binary-class	(N,) or (N, 2)	ROC one-vs-rest on the positive score; for 2-D input the last column is used	threshold the selected score at 0.5
multi-label, binary-class	(N, L)	compute ROC AUC independently for each label and average	threshold each label at 0.5, compute per-label accuracy, then average
multi-class	(N, C)	one-vs-rest AUC for each class column and average	argmax over columns
ordinal-regression	(N, C)	same one-vs-rest loop as multiclass	argmax over columns

Labels should be class indices 0..C-1 for multiclass/ordinal and binary 0/1 for binary. Every one-vs-rest target must contain both positive and negative examples; otherwise scikit-learn cannot define ROC AUC. For multilabel, retain the 2-D (N, L) shape even when inspecting one label.

getAUC(y_true, y_score, task) and getACC(y_true, y_score, task, threshold=0.5) are the low-level functions. They call squeeze(), so an unexpected singleton axis can change dimensionality; assert shapes before calling them. Do not interpret ACC as a probability metric, and do not pass already-argmaxed class labels in place of scores.

get_dummy_prediction() is useful for discovering a rough output shape, but it is random and is not a reproducible test fixture. In the inspected version, its multiclass branch derives the column count from labels.max(); construct explicit (N, C) scores when all classes must be represented.

Standard result filenames and round trips

With metrics = Metrics(AUC, ACC), the standard name is:

{flag}{size_flag}_{split}_[AUC]{auc:.3f}_[ACC]{acc:.3f}@{run}.csv

size_flag is empty for 28/default and _64, _128, or _224 for the corresponding 2-D size (or _64 for a 3-D size). For example: pneumoniamnist_64_test_[AUC]1.000_[ACC]1.000@run1.csv. run=None uses the current time, so use a string or number for stable output.

Evaluator.parse_and_evaluate(path, run=None) parses the flag, optional size, and split from the underscore-separated filename, reads the CSV with index_col=0, header=None, sorts by that index, evaluates the resulting score matrix, prints Metrics(...), and writes a standardized evaluated CSV beside it. If run is omitted, the filename must contain @. The parser constructs an evaluator with the default root; direct evaluation with an explicit root is safer for isolated fixtures. Put parsed inputs in an isolated output directory because a matching standard name can be rewritten.

Export and montage procedure

2-D

from medmnist import PathMNIST

dataset = PathMNIST(split="test", root="./data", size=28)
dataset.save("./export", postfix="png", write_csv=True)
montage = dataset.montage(length=2, replace=True, save_folder="./export")

MedMNIST2D.save calls save2d, creates export/{flag}{size_flag}/, and writes one image per sample plus {flag}{size_flag}.csv when write_csv=True. The default postfix is png; the implementation can pass another image extension accepted by Pillow. Image names are {SPLIT}{index}_{labels}.{postfix} where SPLIT is TRAIN, VALIDATION, or TEST. The CSV is append-mode, has no generated header, and contains split, filename, and label values; re-running against the same CSV can append duplicates.

montage(length=20, replace=False, save_folder=None) randomly selects length * length samples. For a tiny fixture, either use replace=True or ensure the dataset has at least that many samples. A saved 2-D montage is {flag}{size_flag}_{split}_montage.jpg; the method returns a PIL image. Random selection is not deterministic unless the caller seeds NumPy before calling it.

3-D

from medmnist import OrganMNIST3D

dataset = OrganMNIST3D(split="test", root="./data", size=28)
dataset.save("./export", postfix="gif", write_csv=True)
frames = dataset.montage(length=2, replace=True, save_folder="./export")

MedMNIST3D.save requires postfix="gif"; it emits one GIF per volume and a CSV under the same naming layout. A 3-D montage returns a list of frames and, when saved, writes {flag}{size_flag}_{split}_montage.gif. The current 3-D montage path requires a single channel. GIF output requires a nonempty volume and compatible PIL frames.

CLI procedure

The package exposes Fire commands. With data already local, use:

python -m medmnist available
python -m medmnist info --flag=pneumoniamnist
python -m medmnist save --flag=pneumoniamnist --folder=./export --postfix=png --download=False --size=28 --root=./data
python -m medmnist save --flag=organmnist3d --folder=./export --postfix=gif --download=False --size=28 --root=./data
python -m medmnist evaluate --path=./results/pneumoniamnist_test_[AUC]1.000_[ACC]1.000@run1.csv

save loops over all three splits, so all three corresponding NPZ arrays are required. --download=True is a network operation and is intentionally not part of a safe smoke test. evaluate has only --path; its parser resolves the dataset through the default root rather than exposing a custom-root flag. For a custom root, use the direct Evaluator API (or arrange the data in the expected default root only with explicit user approval). available and info are read-only metadata commands. Do not use clean in this skill: it deletes downloaded *mnist*.npz files.

Verification and return handoff

Run the bundled helper before relying on a new environment:

python scripts/medmnist_smoke.py --help
python scripts/medmnist_smoke.py

A successful run reports deterministic binary, multilabel, and multiclass AUC=1.000 ACC=1.000 checks, a _64/@run CSV parse round trip in a temporary root, and 2-D/3-D export checks without network access. For a concise handoff, report: files used or emitted; evidence (task, split, root, score shapes); checks and metric values; difficult cases (singleton axes, missing ROC classes, _64/@run, tiny montage sampling, 3-D GIF); gaps such as default-root-only parsing or untested optional backends; and intentional omissions (training, bulk downloads, default-root clean, and broad test()).