---

name: data-preparation description: "Clean extracted text, stage Unstructured elements for annotation and labeling tools, use lightweight NLP helpers, and prepare local outputs for downstream systems. Route raw partitioning to partitioning, generic element JSON/schema work to elements-and-metadata, provider embeddings to embeddings, and chunk composition to chunking." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Data Preparation

Use this sub-skill when a task starts from text or already-partitioned Unstructured elements and needs local cleanup, annotation export, lightweight tokenization, or preparation for downstream systems. Do not use it to partition raw source files, design embeddings providers, or inspect the full element JSON schema.

Start Here

• For OCR or PDF-extracted text cleanup, use unstructured.cleaners.core first; these helpers are local and deterministic.
• For metadata extraction from email-like text or HTML snippets, use unstructured.cleaners.extract regular-expression helpers.
• For annotation tools, stage existing text elements with unstructured.staging.prodigy, label_studio, label_box, datasaur, baseplate, or huggingface after confirming each target tool's schema.
• For tokenization and sentence splitting, use unstructured.nlp.tokenize only when the environment can load the spaCy model; first use may install or load model data.
• Treat translation, Hugging Face tokenizers/models, annotation SaaS APIs, and hosted storage URLs as third-party/service boundaries; require explicit dependencies, network access, and credentials where applicable.

Local Text Cleaning

Use the combined clean() function for simple normalization:

from unstructured.cleaners.core import clean

normalized = clean(
    raw_text,
    extra_whitespace=True,
    dashes=True,
    bullets=True,
    trailing_punctuation=True,
)

Use focused cleaners when order matters or when a single transform is safer:

• clean_extra_whitespace() collapses newlines, non-breaking spaces, and repeated spaces.
• clean_bullets() removes a leading Unicode bullet only, preserving bullets elsewhere.
• clean_ordered_bullets() removes numeric/alphanumeric outline prefixes such as 1.1 when they look like section markers.
• clean_ligatures() maps common ligatures such as fi and fl to ASCII-style text.
• replace_unicode_quotes() fixes common mojibake quote/dash sequences; replace_mime_encodings() decodes quoted-printable MIME fragments.
• clean_prefix() and clean_postfix() remove anchored regex patterns from text edges.

For a safe command-line preview, use the bundled script:

python sub-skills/data-preparation/scripts/clean_text_preview.py --text "●  RISK\u00a0FACTORS---"
python sub-skills/data-preparation/scripts/clean_text_preview.py input.txt --json-output preview.json
python sub-skills/data-preparation/scripts/clean_text_preview.py elements.json --elements-json --max-items 5

The script reads local text or Unstructured element JSON, applies selected local cleaners, and writes a preview. It does not call partition APIs, translation models, hosted services, or annotation APIs.

Extraction Helpers

Use unstructured.cleaners.extract for lightweight pattern extraction:

from unstructured.cleaners.extract import extract_email_address, extract_text_after

emails = extract_email_address(header_text)
body = extract_text_after(raw_email, r"\n\n", index=0)

Common helpers include extract_email_address(), extract_ip_address(), extract_ip_address_name(), extract_mapi_id(), extract_datetimetz(), extract_us_phone_number(), extract_ordered_bullets(), and extract_image_urls_from_html(). extract_text_before() and extract_text_after() raise ValueError if the requested match index does not exist; catch that when patterns are optional.

Annotation Staging Routes

Work from Text-like elements (Title, NarrativeText, Text, etc.) whenever an annotation target expects plain text examples:

from unstructured.staging.prodigy import stage_for_prodigy

records = stage_for_prodigy(elements, metadata=[{"source": "batch-a"} for _ in elements])

• Prodigy: stage_for_prodigy() emits JSON-like records with text and meta; stage_csv_for_prodigy() emits CSV. Do not pass an id key in metadata because element ids are injected into meta.id.
• Label Studio: stage_for_label_studio() emits task dictionaries under data; optional annotations/predictions must match element count. LabelStudioResult.type must be one of the module's supported label types, and prediction score must be between 0 and 1. This function is deprecated in the package, so prefer validating against current Label Studio tooling before production use.
• Labelbox: stage_for_label_box() writes one text file per element and returns import config with data URLs. It requires an output directory and a URL prefix where those files will be hosted; attachment type must be IMAGE, VIDEO, RAW_TEXT, TEXT_URL, or HTML.
• Datasaur: stage_for_datasaur() emits text plus entities; each entity requires text, type, start_idx, and end_idx, and the entity list length must match the elements length.
• Baseplate: stage_for_baseplate() flattens element dictionaries into rows with data and metadata fields suitable for API upload payloads.
• Hugging Face/transformers: stage_for_transformers() and chunk_by_attention_window() split long text to fit a provided tokenizer's attention window. This requires a tokenizer object and the transformers dependency.

See references/cleaning-and-staging.md for schema examples and references/troubleshooting.md for common validation failures.

NLP and Translation Boundaries

unstructured.nlp.tokenize.sent_tokenize(), word_tokenize(), and pos_tag() use spaCy under the hood with caching. They may load or install a pinned en_core_web_sm model on first use, so do not assume they are zero-dependency helpers in locked-down or offline environments.

unstructured.cleaners.translate.translate_text() uses langdetect, transformers, Marian tokenizer/model loading, and Hugging Face model names such as Helsinki-NLP/opus-mt-de-en. It returns unchanged text when the source and target language are the same or when text is blank, but actual translation can require large model downloads, network access, and supported source/target language pairs. Ask before enabling it in offline, credential-sensitive, or latency-sensitive workflows.

Boundaries

• Raw document partitioning and document-specific parser options belong to partitioning.
• Element schema inspection, generic JSON/NDJSON/Markdown/Text conversion, metadata coordinates, and table fidelity belong to elements-and-metadata.
• Chunk construction with chunk_elements() or chunk_by_title() belongs to chunking; this sub-skill only covers Hugging Face attention-window staging.
• Provider embeddings, embedding interfaces, and vector output preparation belong to embeddings.
• Review/test artifacts do not belong in this runtime subtree.

Evidence

This sub-skill is grounded in the package cleaner, staging, and NLP helper modules plus their behavior tests.