---

name: cli-batch-processing description: "Use Trafilatura's command line for stdin, URL/list, and local directory batch extraction with safe outputs, configs, backups, parallelism, and validation." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

CLI Batch Processing

Use this sub-skill when the user wants Trafilatura from a shell or automation script rather than Python APIs: stdin extraction, -u/--URL, URL-list batches, local --input-dir batches, output directories, backups, --keep-dirs, output format flags, --parallel, --config-file, metadata filters, deduplication, precision/recall presets, and XML-TEI validation.

Do not use this sub-skill for direct Python function calls; route those requests to python-extraction. For feed/sitemap/crawl algorithms beyond the CLI flag invocation, route to discovery-downloads. For conceptual deduplication, corpus formats, and TEI quality details, route to corpus-quality.

Start here

1. Confirm the CLI is reachable:

   trafilatura --version
   trafilatura --help
   # PATH fallback when the console script is unavailable:
   python -m trafilatura.cli --help
   

2. Choose exactly one primary input mode:
   • no input flag: read HTML bytes from stdin;
   • -u/--URL: download and extract one URL;
   • -i/--input-file: process a text file containing one URL per line;
   • --input-dir: walk local downloaded HTML files.
3. Choose output behavior:
   • no -o/--output-dir: print to stdout;
   • -o DIR: write result files, creating directories as needed;
   • --backup-dir DIR: keep gzipped raw HTML for URL downloads;
   • --keep-dirs: preserve local input paths, but only with -o/--output-dir.
4. Add extraction and format options deliberately: --with-metadata, --only-with-metadata, --deduplicate, --precision, --recall, --xmltei --validate-tei, --json, --markdown, etc.
5. For a no-network sanity check, run the bundled helper:

   python scripts/cli_fixture_runner.py
   python scripts/cli_fixture_runner.py --json
   

References

• references/cli-reference.md — flag groups, input/output rules, format interactions, safe command construction, exit statuses.
• references/batch-workflows.md — ready-to-adapt stdin, URL-list, directory, backup, config, listing, and TEI recipes.
• references/troubleshooting.md — CLI-specific failure symptoms and recoveries.
• scripts/cli_fixture_runner.py — no-network smoke helper for installed CLI behavior.

Safety checklist for generated commands

• Quote URLs and paths in shell examples; in Python, prefer subprocess.run([...]) argument lists over shell strings.
• Keep network batches polite: start with --parallel 1, increase only when allowed, and use SLEEP_TIME in a config file for domain-level pacing.
• Separate discovery/listing from extraction: --list prints URLs only and ignores extraction/output-format expectations.
• Use a small fixture or sample directory before a large batch; verify output content and file names before scaling.
• Prefer relative --input-dir values when using --keep-dirs, so mirrored output paths stay inside the chosen output directory.

Provenance

This sub-skill distills Trafilatura 2.2.0 CLI behavior from public package docs, CLI source, settings source, and selected CLI/download tests. Evidence labels include docs/usage-cli.rst, docs/quickstart.rst, docs/installation.rst, docs/settings.rst, docs/troubleshooting.rst, trafilatura/cli.py, trafilatura/cli_utils.py, trafilatura/settings.py, trafilatura/settings.cfg, tests/cli_tests.py, and selected tests/downloads_tests.py queue cases.