---

name: finetuning description: "Prepare and run Qwen-VL supervised finetuning, LoRA, and Q-LoRA workflows." metadata: disco-role: operating disable-model-invocation: true license: NOASSERTION

Qwen-VL Finetuning

Use this sub-skill when the user wants to adapt Qwen-VL or Qwen-VL-Chat for a downstream instruction-tuning task, a domain-specific adapter, or a memory-constrained Q-LoRA run. It covers the official training script, launch templates, data layout, checkpoint handling, and the most common configuration and dependency failures.

Read first

• Data format for the conversation JSON schema, multimodal markup, and adapter-related constraints.
• Launch reference for the official full-parameter, LoRA, and Q-LoRA launch templates.
• Troubleshooting for peft/transformers compatibility, DeepSpeed, CUDA, memory, and special-token issues.
• Bundled finetune validator for a quick schema check before launching a training job.
• Bundled training script and the shell templates in the same directory are copied/adapted runtime entrypoints from the source repo.

Choose this sub-skill for

• Building a training command from a JSON conversation dataset.
• Choosing between full-parameter finetuning, LoRA, and Q-LoRA.
• Preparing a DeepSpeed launch, single-GPU launch, or multi-GPU launch.
• Validating that the data has the right conversation structure, image markup, and box format.
• Loading a LoRA adapter for inference after training or merging a LoRA adapter into a standalone checkpoint.
• Diagnosing OOM, missing CUDA tooling, wrong model type, or adapter/token compatibility failures.

Route elsewhere

• One-off multimodal chat, grounding, quantization-aware inference, or box rendering belong in ../inference/SKILL.md.
• Serving a trained checkpoint behind a Gradio or FastAPI listener belongs in ../serving/SKILL.md.
• Benchmark scoring and submission formatting belong in ../evaluation/SKILL.md.

Operating rules

1. Use Qwen/Qwen-VL-Chat for instruction-style finetuning unless the user explicitly wants base-model behavior; the chat model already carries the chat-token conventions expected by the examples.
2. Keep the data in the documented list-of-conversations JSON form. Each sample needs an id and a conversations array with alternating user and assistant messages.
3. For multimodal messages, keep the Picture n: <img>...</img> prefix and the <ref>/<box> markup exactly as documented. The box coordinates are normalized values in [0, 1000).
4. Full finetuning, LoRA, and Q-LoRA are separate choices. Do not mix the launch flags casually:
   • full finetuning uses the base training script with --use_lora off;
   • LoRA uses --use_lora and usually --bf16;
   • Q-LoRA adds --q_lora and uses --fp16 rather than --bf16.
5. For Q-LoRA, prefer the Int4 chat model. The repo docs warn against using BF16 for that path.
6. The prepared inspection environment verified a compatible combination of transformers==4.32.0, peft==0.5.0, deepspeed, and CUDA torch wheels. If a different environment is used, re-check those versions before trusting the launch templates.
7. CUDA_HOME is not required to read the help text, but it matters if you want to compile or use optional DeepSpeed CUDA extensions. The troubleshooting reference explains the warning.

Typical workflow

1. Validate the dataset with python scripts/validate_finetune_data.py --data DATA.json.
2. Pick the launch template that matches the task size and adapter strategy.
3. Replace the placeholder MODEL, DATA, and OUTPUT_DIR values.
4. Run the shell template explicitly; do not infer defaults from the source repo.
5. After training, load the adapter with the inference sub-skill or merge it only when the docs say the adapter type supports merging.

Quick launch map

• Full finetuning: scripts/finetune_full_ds.sh
• LoRA single GPU: scripts/finetune_lora_single_gpu.sh
• LoRA DeepSpeed: scripts/finetune_lora_ds.sh
• Q-LoRA single GPU: scripts/finetune_qlora_single_gpu.sh
• Q-LoRA DeepSpeed: scripts/finetune_qlora_ds.sh

What this sub-skill does not do

• It does not run a long training job on your behalf.
• It does not download data or checkpoints automatically.
• It does not merge Q-LoRA adapters, because the source docs say that merge path is not supported.

If the user already has a trained adapter and wants to use it for multimodal chat, route to the inference sub-skill for adapter-loading guidance.