---

name: yi-val description: "Operate YiVal prompt/model evaluation workflows, including YAML configs, CLI runs, data readers, prompt generators, evaluators, AHP selection, enhancers, and custom components." disable-model-invocation: true metadata: disco-role: operating repo: YiVal/YiVal package: yival version-baseline: "0.0.0 at 22a1fa0" license: Apache 2.0

YiVal operating skill

Use this skill when the task involves YiVal or the Python package yival: building or validating experiment YAML, running prompt/model comparisons, evaluating LLM outputs, selecting the best variation, enhancing prompts, or extending YiVal with custom readers/wrappers/evaluators/generators.

YiVal's core flow is:

1. produce input examples from a dataset, user input, or a data generator;
2. create combinations from wrapper variations and variation generators;
3. call a user custom_function(..., state=ExperimentState);
4. evaluate outputs with individual, comparison, or all-results evaluators;
5. optionally select a best combination, enhance prompts, display results, or open the bot/UI.

First checks

• Install or activate a Python environment with yival importable. The package supports Python >=3.10,<3.13.
• If using a checkout, install with python -m pip install -e .; for trainer workflows add python -m pip install -e '.[trainers]' only when local fine-tuning is explicitly needed.
• If root CLI import fails with ModuleNotFoundError: pkg_resources, install a compatible setuptools that still exposes the import: python -m pip install 'setuptools<81'.
• Run the bundled smoke check from this skill root: python scripts/check_install.py --check-cli.
• For a no-network mini experiment fixture, run: python scripts/write_minimal_experiment.py --run.

Route map

User intent	Read
Install YiVal, inspect CLI commands, initialize or validate YAML, choose dataset source fields	setup
Run yival run, yival demo, Dash/interactive/bot workflows, ExperimentRunner, LiteExperimentRunner, outputs	run
Generate synthetic examples or prompt variations with OpenAI/document/chain-of-density/self-exemplar generators	prompt-automation
Configure evaluators, AHP selection, human ratings, OpenAI/Alpaca/BERTScore/ROUGE/Python validation, prompt enhancers	evaluation-optimization
Implement custom readers, wrappers, evaluators, data generators, variation generators, enhancers, selection strategies, or output parsers	custom-components

Shared repo references:

• Workflow map: YiVal architecture, execution stages, and cross-sub-skill responsibilities.
• Registry overview: built-in component ids, registration behavior, and config keys.
• Troubleshooting: install, CLI, config, provider, UI, and safety failures.
• Repo provenance: source baseline and coverage limits.

Common fast paths

Create and validate a config

1. Read setup.
2. Generate a template with either yival init ... or python sub-skills/setup/scripts/build_config_template.py ....
3. Fill in dataset, custom_function, variations, and evaluators.
4. Validate with yival validate config.yml or python - <<'PY' using load_and_validate_config.

Run a dataset-backed prompt comparison

1. Read run for the execution model.
2. Use dataset.source_type: dataset, reader: csv_reader, and a CSV with columns matching the custom function arguments.
3. Wrap tunable prompt strings in StringWrapper(name=<variation name>, state=state).
4. Run yival run config.yml --output_path results.pkl and inspect the generated <stem>_0.pkl or UI.

Build automated prompt-generation and evaluation

1. Read prompt-automation for openai_prompt_data_generator, document_data_generator, and prompt variation generators.
2. Read evaluation-optimization for evaluators, AHP criteria names, and enhancers.
3. Keep provider-token requirements explicit; built-in OpenAI/Replicate/Alpaca workflows can bill or call network services.

Extend YiVal with a custom component

1. Read custom-components.
2. Implement the correct base class and config dataclass.
3. Add the corresponding custom_* block to YAML.
4. Run the root smoke script and a tiny local experiment before running provider-backed experiments.

Scope and safety boundaries

• This skill covers core YiVal experiment setup, runtime, generation, evaluation, selection, enhancement, and custom-component workflows.
• Fine-tuning modules are present in YiVal but are optional and heavier. Treat local SFT, OpenAI fine-tuning jobs, and Replicate fine-tuning as out-of-scope unless the user explicitly requests training and approves required extras, credentials, cost, model downloads, and GPU/runtime needs.
• python_validation_evaluator executes model-produced Python with exec. Use only with sandboxed, trusted, tiny code snippets; never run untrusted code in a production or credentialed environment.
• OpenAI, Replicate, AlpacaEval, Hugging Face dataset URLs, Google Drive document loading, Streamlit/Dash/ngrok, and some demos require network access, credentials, or public services. Prefer the offline smoke fixture before provider-backed runs.

Expected artifacts and outputs

• CLI runs can display Dash results, write pickled experiment objects as <output_stem>_<config_index>.pkl, or open interactive/bot modes.
• Programmatic runs use ExperimentRunner(config_path).run(...) for YAML-driven workflows or LiteExperimentRunner(...).run_experiment(enable_selector=True/False) when the caller already holds data, evaluator, token logger, and config objects.
• Data rows become InputData(content={...}, expected_result=...); custom functions should return MultimodalOutput(text_output=..., image_output=..., video_output=...).

Refresh triggers

Refresh this skill when YiVal changes CLI subcommands, config schema names, registry ids, evaluator/enhancer contracts, OpenAI SDK support, optional trainer dependencies, or result object serialization.