---

name: multimodal-understanding description: "Guides Janus-family image question answering, prompt formatting, image loading, and multimodal input preparation." disable-model-invocation: true metadata: disco-role: operating license: NOASSERTION

Multimodal Understanding

Use this sub-skill when the task is about image+question prompting, OCR/formula conversion, visual question answering, image placeholder debugging, or adapting the Janus understanding examples.

Read first

• references/workflows.md for end-to-end understanding recipes.
• references/api-reference.md for verified processor/model signatures and output fields.
• references/troubleshooting.md for image, template, dtype, and backend failures.
• scripts/janus_understanding.py for a safe dry-run helper and an optional real model run.

For installation and model-family selection, use the root ../../references/installation-and-models.md.

When to use this route

Choose this route when the user asks to:

• Answer a question about an image with Janus, Janus-Pro, or JanusFlow.
• Convert a formula or document image into text or LaTeX.
• Build a conversation containing <image_placeholder>.
• Debug processor outputs, image token masks, or prepare_inputs_embeds.
• Validate image paths/base64 inputs before a full model download.

Core workflow

1. Choose the model family and model id.
2. Build a conversation with one user turn containing <image_placeholder> and one empty assistant turn.
3. Load images as RGB PIL images.
4. Call VLChatProcessor(..., force_batchify=True).
5. Move the batched processor output to the model device and dtype.
6. Call vl_gpt.prepare_inputs_embeds(**prepare_inputs).
7. Generate from vl_gpt.language_model.generate(...) with the attention mask and tokenizer special ids.
8. Decode the first output and inspect the formatted prompt if the answer looks wrong.

Dry-run before model execution

When the task is only to validate prompt/image layout, use:

python sub-skills/multimodal-understanding/scripts/janus_understanding.py \
  --family janus-pro \
  --model-id deepseek-ai/Janus-Pro-1B \
  --image ./sample.png \
  --question "Convert the formula into LaTeX."

The default mode prints a plan and does not download model weights. Add --run-model only when the environment, model access, and GPU/CPU trade-off are acceptable.

Route elsewhere

• Use ../image-generation/SKILL.md for Janus or Janus-Pro text-to-image generation.
• Use ../janusflow-workflows/SKILL.md for JanusFlow rectified-flow image generation.
• Use ../demos-and-serving/SKILL.md for Gradio or FastAPI serving.