---

name: vision-document-ai description: "Use UniLM vision, vision-language, OCR, webpage, and document-AI workflows across BEiT, DiT, LayoutLM, MarkupLM, XDoc, TrOCR, VLMo, and VL-BEiT." disable-model-invocation: true metadata: disco-role: operating license: MIT

Vision Document AI

Use this sub-skill when the task is about UniLM vision or document-AI models: image classification/pretraining, semantic segmentation, document layout/text detection, OCR, form understanding, token classification, relation extraction, visual question answering, captioning, retrieval, NLVR2, webpage extraction, or cross-format document understanding.

Route Here

• Build or adapt command plans for BEiT, BEiT v2, BEiT-3, DiT, LayoutLM/LayoutLMv2/LayoutLMv3, LayoutXLM, MarkupLM, XDoc, TrOCR, VLMo, or VL-BEiT.
• Validate dataset assumptions for ImageFolder, ADE20K-style segmentation, FUNSD, CORD, XFUND, PubLayNet, ICDAR cTDaR, WebSRC, SWDE, SQuAD, COCO, Flickr30k, VQAv2, NLVR2, or pyarrow vision-language data.
• Diagnose vision/document dependency failures involving legacy PyTorch, CUDA wheels, Detectron2, MMCV/mmsegmentation, Apex, fairseq, Deepspeed, or Hugging Face Transformers drift.
• Choose among OCR, document layout, form extraction, webpage markup, image-only, and vision-language routes before constructing a command.

Route Elsewhere

• Use embeddings-retrieval for E5, SimLM, dense retrieval, embedding indexing, or text-only retriever training.
• Use multimodal-generation for TextDiffuser, Kosmos, audio, LatentLM, image/text generation, or open-ended multimodal generation systems.
• Use architectures-training for architecture-only distributed training, generic TorchScale/Magneto/DeepNet wiring, optimizer schedules, or Deepspeed mechanics not tied to a vision/document task.

Read Or Run

• Read references/model-task-map.md to pick the correct model family, task owner, native entry point class, dependency stack, and sibling route.
• Read references/workflows.md before building training, evaluation, data-conversion, OCR, or vision-language commands.
• Read references/troubleshooting.md when a dependency stack, dataset schema, checkpoint, tokenizer, bounding-box, OCR, or Hugging Face version error blocks progress.
• Run scripts/build_vision_doc_command.py to print safe command templates for BEiT classification, BEiT-3 tasks, LayoutLMv3 FUNSD/CORD/XFUND, DiT detection, TrOCR inference, or VLMo runs without training, downloading, or requiring model/data files.

Safe Default Workflow

1. Classify the request by modality: image-only, document OCR/layout, form/entity extraction, webpage markup, cross-format document QA, or vision-language.
2. Use the model-task map to select the family and confirm whether the original workflow expects legacy PyTorch, Detectron2/MMCV, fairseq, Deepspeed, or Transformers.
3. Use the command builder for a template, then replace placeholders only after confirming data layout, checkpoint availability, GPU count, and backend versions.
4. Treat original UniLM training and inference programs as heavy reference entry points; do not run them until the user confirms environment, data, checkpoint, and hardware readiness.