---

name: evaluation-and-observability description: "Evaluate Haystack retrieval, answers, and pipelines, and instrument applications with tracing, logging, telemetry control, and diagnostic outputs." disable-model-invocation: true metadata: disco-role: operating license: NOASSERTION

Evaluation and Observability

Use this sub-skill when the task is about measuring Haystack output quality or diagnosing runtime behavior: retrieval and answer evaluators, evaluation reports, trace spans, logging output, pipeline debug output, and telemetry opt-out.

Route here

• Score retrieved Document lists with recall, MRR, MAP, or NDCG and interpret aggregate versus per-query metrics.
• Score answer strings with exact match, SAS, faithfulness, context relevance, or a custom LLMEvaluator.
• Build an evaluation pipeline or wrap evaluator outputs in EvaluationRunResult reports.
• Add LoggingTracer, OpenTelemetry, Datadog, or a custom tracer to inspect component and pipeline execution.
• Enable local pipeline debug outputs, structured logging, or content tracing while avoiding sensitive-data leakage.
• Disable Haystack telemetry or auto tracing for privacy-sensitive applications.

Reroute

• Build RAG pipelines, document stores, retrievers, rankers, readers, or prompt-to-answer flow: use ../retrieval-and-rag/SKILL.md.
• Configure provider generators, model credentials, prompt builders, or chat/message APIs: use ../generation-and-model-components/SKILL.md.
• Create custom components, connect pipeline sockets, serialize pipelines, or use Pipeline/AsyncPipeline mechanics: use ../pipelines-and-components/SKILL.md.
• Run repository tests, Hatch commands, release notes, or contributor workflows: use the repo-development route if it is present in the generated skill.

Start fast

1. Choose metric family: retrieval metrics compare expected versus retrieved Document objects; answer metrics compare expected versus predicted answers; LLM evaluators judge faithfulness, context relevance, or custom rubric outputs.
2. Keep every evaluator input list aligned by sample count; most evaluator failures are length mismatches or nested-list shape mistakes.
3. Use statistical evaluators first for deterministic checks, then add LLM evaluators only when credentials and JSON-mode generator behavior are available.
4. Wrap evaluator outputs with EvaluationRunResult(run_name=..., inputs=..., results=...) to produce aggregated_report() and detailed_report() dictionaries, CSV files, or dataframes.
5. For diagnostics, start with LoggingTracer and logging.getLogger("haystack").setLevel(logging.DEBUG) before adding network tracing backends.
6. Treat content tracing as sensitive: enable tracing.tracer.is_content_tracing_enabled = True only for local debugging or sanitized data.

Run the bundled smoke check from this sub-skill directory to validate deterministic evaluator and logging-tracer behavior without network credentials:

python scripts/evaluation_smoke_check.py

If working in the Haystack repository checkout, follow the repository’s Hatch policy for execution, for example:

hatch -e test run python skills/disco/haystack/sub-skills/evaluation-and-observability/scripts/evaluation_smoke_check.py

References

• references/api-reference.md lists evaluator imports, inputs, outputs, report APIs, tracing APIs, logging controls, and telemetry switches.
• references/workflows.md gives copyable workflows for retrieval evaluation, answer evaluation, LLM evaluation, reporting, tracing, pipeline debug output, and privacy-safe observability.
• references/troubleshooting.md maps install/import, optional dependency, credential/backend, API misuse, data/config, and workflow failures to checks and fixes.