---

name: multi-modality description: "Guides InternVideo2 multi-modality Stage2 and CLIP video-text/audio training, retrieval evaluation, demo setup, Python configs, launchers, and troubleshooting." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

InternVideo2 Multi-Modality

Use this sub-skill for InternVideo2/multi_modality: Stage2 video-language alignment, CLIP post-pretraining, zero-shot/fine-tuned retrieval, audio-text/audio-visual tasks, demo retrieval, config files, and distributed launch adaptation.

Read first

• Workflows explains Stage2, CLIP, evaluation, demo, preprocessing, and launcher layers.
• Configuration and data records config system, path variables, model families, annotation/preprocess expectations, and checkpoint naming.
• Troubleshooting covers PYTHONPATH, FlashAttention/Apex/DeepSpeed, checkpoint, tokenizer, and dataset failures.

Safe helper

Use the bundled launch builder to print command skeletons without submitting jobs:

python scripts/build_multimodal_launch.py --task pretrain --branch stage2 --config scripts/pretraining/stage2/1B/config.py --nodes 8 --gpus-per-node 8
python scripts/build_multimodal_launch.py --task evaluate --branch stage2 --config scripts/evaluation/stage2/zero_shot/1B/config_msrvtt.py --pretrained ${INTERNVIDEO2_MODEL_PATH}/1B_stage2_pt.pth
python scripts/build_multimodal_launch.py --task retrieval --branch clip --config scripts/evaluation/clip/zero_shot/1B/config_msrvtt.py --pretrained ${INTERNVIDEO2_MODEL_PATH}/InternVideo2_CLIP_1B.pth --no-slurm

The helper prints a tools/run.py dry-run concept for Stage2 plus a direct srun or torchrun skeleton. It never calls the source launcher.

Route by task

Task	Route
Stage2 pretraining or finetuning	tasks/pretrain.py plus a Python config under scripts/pretraining or scripts/finetuning; config owns text encoder, dataset roots, DeepSpeed, and evaluation flags.
CLIP post-pretraining	tasks_clip/pretrain.py; use matching vision/text checkpoint paths and remember CLIP variants may be InternVL/LLM-backed or MobileCLIP-backed.
Zero-shot retrieval/action evaluation	Stage2 uses tasks/pretrain.py with evaluate True; CLIP retrieval uses tasks_clip/retrieval.py; both rely on dataset-specific config files.
Demo video-text retrieval	Use the demo config and demo/utils.py with the multi-modality folder on PYTHONPATH; Stage2 demo needs tokenizer and checkpoint paths before inference.
Preprocess/annotation conversion	Use preprocess/create_sqlite_db.py as schema evidence for JSON-to-SQLite shape, but keep destructive conversions out of the helper script.
Job submission abstraction	Treat tools/run.py as a launcher template that copies code to VL_EXP_DIR, resolves SLURM/local mode, and forwards tasks through torchrun; use the bundled helper for dry runs.
Audio-video workflows	Stage2 configs can switch to BEATs audio encoders and audio-visual losses; verify audio packages, media roots, and checkpoint paths first.

Required decisions before execution

1. Branch: Stage2 or CLIP.
2. Mode: pretrain, evaluate/retrieval, demo, or preprocess.
3. Config file and overrides: dataset paths, output dir, pretrained_path, checkpoint paths, DeepSpeed setting, and optional extra config args.
4. Text/vision/audio checkpoint family: BERT tokenizer folder, InternVL/LLM text encoder path, MobileCLIP/LoRA path, vision backbone checkpoint, BEATs audio weights, or distilled extra checkpoint.
5. Runtime: SLURM or local torchrun, GPU count, FlashAttention/Apex/DeepSpeed readiness, and whether PYTHONPATH must be rooted at the multi-modality folder.

Boundaries

• Route video-only action recognition finetuning to ../single-modality/SKILL.md.
• Route MLLM long-video chat to ../video-mllm/SKILL.md.
• Route annotation validation and InternVid dataset questions to ../datasets/SKILL.md.