---

name: evals-and-graph description: "Guides agents building Pydantic Evals datasets, evaluators, reports, Logfire evals, and pydantic-graph GraphBuilder state-machine workflows." disable-model-invocation: true metadata: disco-role: operating license: MIT

Evals and Graph

Use this sub-skill when a task involves pydantic_evals datasets, cases, evaluators, reports, experiment runs, Logfire-backed eval analysis, or pydantic_graph state machines built with GraphBuilder.

Read First

• Read references/evals-workflows.md when creating Dataset/Case objects, custom evaluators, report evaluators, serialized YAML/JSON datasets, concurrency/retry settings, lifecycle hooks, or agent-eval integrations.
• Read references/graph-workflows.md when designing a graph with GraphBuilder, StepContext, typed state/deps, BaseNode interop, decisions, joins, maps, broadcasts, or Mermaid diagrams.
• Read references/troubleshooting.md when eval outputs, evaluator return types, optional Logfire tracing, graph typing, legacy Graph, joins, or rendered edges behave unexpectedly.
• Run scripts/evals_graph_smoke.py after installing pydantic-evals and pydantic-graph to verify a no-network eval plus a builder graph run.

Routing

• Use evals for repeatable system-quality measurement over cases; keep ordinary unit tests for deterministic API contracts, regressions, and fast local assertions.
• Start eval work with typed Dataset[InputsT, OutputT, MetadataT], named Case(...) rows, deterministic code evaluators, and evaluate_sync(..., progress=False, max_concurrency=1) while debugging.
• Add LLMJudge only when deterministic checks cannot express the quality target; route judge model/provider selection to ../models-and-providers/SKILL.md.
• Evaluate Pydantic AI agents by wrapping agent.run_sync() or agent.run() as the dataset task; route agent construction, TestModel, FunctionModel, and Agent.override(...) details to ../agent-core/SKILL.md.
• Use GraphBuilder for new graph workflows; import the legacy Graph runner only to maintain existing code and expect a deprecation warning when imported from pydantic_graph.

Boundaries

• For core Agent construction, deterministic TestModel tests, run methods, dependencies, streaming, and message history, read ../agent-core/SKILL.md.
• For LLM judge provider strings, optional provider extras, model settings, fallback, and auth diagnostics, read ../models-and-providers/SKILL.md.
• For function tools, toolsets, approvals, deferred calls, and ModelRetry, read ../tools-and-toolsets/SKILL.md.
• For structured output/message serialization used by evaluated agents, read ../outputs-and-messages/SKILL.md.
• For maintainer VCR/cassette, native repo test selection, and CI workflow mechanics, read ../repo-development/SKILL.md when that sub-skill exists.

Non-Negotiables

• Keep bundled eval examples deterministic and no-network unless the user explicitly asks for live provider judging or Logfire upload.
• Give every eval case a useful name; include expected_output or metadata whenever evaluators need ground truth.
• Return evaluator scalars, EvaluationReason, or mappings of those values; do not return arbitrary models or nested dicts from Evaluator.evaluate().
• Use builder-first graph APIs: GraphBuilder(...), @g.step, g.edge_from(...).to(...), g.decision(), g.join(...), .map(), and graph.run(...).
• Keep graph state and deps types consistent across GraphBuilder, StepContext, BaseNode[...], and GraphRunContext[...] annotations.