---

name: evaluation-optimization description: "Configure YiVal evaluators, human ratings, AHP selection, and prompt/combination enhancers safely and accurately." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

YiVal evaluation and optimization

Use this sub-skill when the user needs to choose or debug evaluators, configure AHP selection, add human ratings, run prompt enhancers, compare model outputs, or understand metric names used in YiVal aggregated results.

Read first

• Evaluators reference: evaluator ids, config fields, evaluator types, and safety notes.
• Selection and metrics: AHP criteria names, aggregation, maximization, and weights.
• Enhancers reference: openai_prompt_based_combination_enhancer, optimize_by_prompt_enhancer, pe2_enhancer.
• Evaluation troubleshooting: metric and provider failures.

Useful helper:

• python sub-skills/evaluation-optimization/scripts/evaluate_expected_result_smoke.py runs the string expected-result evaluator and AHP selection without network calls.

Evaluator types

Type	YAML value	When it runs
Individual	evaluator_type: individual	Per ExperimentResult; output is attached to each result and can be averaged.
Comparison	evaluator_type: comparison	Per input group, comparing multiple outputs for the same input.
All-results	evaluator_type: all	After aggregation, with access to the whole experiment list.

Common evaluator choices

Goal	Evaluator id	Notes
Check expected answer text, exact/include/fuzzy/JSON validity	string_expected_result	Fast and offline. Reads InputData.expected_result.
Execute generated Python and score success	python_validation_evaluator	Unsafe unless sandboxed; uses exec.
Semantic similarity with BERTScore	bertscore_evaluator	Can download/use models; tests mock the scorer.
ROUGE for summarization-like text	rouge_evaluator	Offline once dependency is installed.
Judge one output against a prompt and choices	openai_prompt_based_evaluator	OpenAI/network/billing.
Global Elo-style preference judging	openai_elo_evaluator	OpenAI/network/billing.
AlpacaEval comparison	alpaca_eval_evaluator	External annotator/model setup required.

AHP selection pattern

selection_strategy:
  ahp_selection:
    criteria:
      - "string_expected_result: matching"
      - average_token_usage
      - average_latency
    criteria_weights:
      "string_expected_result: matching": 0.8
      average_token_usage: 0.1
      average_latency: 0.1
    criteria_maximization:
      "string_expected_result: matching": true
      average_token_usage: false
      average_latency: false
    normalize_func: null

Criteria can name built-in aggregate fields (average_token_usage, average_latency), evaluator names, or aggregated metric keys like openai_prompt_based_evaluator: clarity.

Enhancer pattern

enhancer:
  name: optimize_by_prompt_enhancer
  model_name: gpt-4
  max_iterations: 2
  enhance_var:
    - task

Enhancers normally call LLM providers and should be run only after a tiny evaluator/selector workflow is proven.

Workflow guidance

1. Start with string_expected_result or another offline metric when possible.
2. Add OpenAI/Alpaca/provider-backed judge metrics only after credentials, costs, and prompts are approved.
3. Make display_name unique when multiple evaluators share the same name so metric keys are distinguishable.
4. Keep AHP weights explicit and ensure maximization direction matches business intent.
5. Treat python_validation_evaluator as unsafe code execution; sandbox it or avoid it.
6. Inspect Experiment.combination_aggregated_metrics before finalizing selector criteria.

Route elsewhere

• For data and prompt-generation setup, read prompt-automation.
• For running and reading output pickles, read run.
• For custom evaluator/selector/enhancer classes, read custom-components.