---

name: inference-workflows description: "Use when running RobustVideoMatting inference, loading weights, using convert_video or TorchHub, converting image/video inputs, or reasoning about exported model runtimes." disable-model-invocation: true metadata: disco-role: operating license: GPL 3.0

RobustVideoMatting Inference Workflows

Use this sub-skill for user-facing RVM inference: loading models, converting videos or image sequences, configuring outputs, and adapting the documented runtime formats.

Read this when

• The user asks to run RVM on a video or image sequence.
• The task mentions convert_video, python inference.py, output alpha, foreground, or composition files.
• You need to load official PyTorch weights, TorchHub models, TorchScript, ONNX, TensorFlow, TensorFlow.js, or CoreML RVM artifacts.
• The user is tuning downsample_ratio, seq_chunk, input_resize, video bitrate, or image-sequence output.

Route other tasks elsewhere:

• MattingNetwork constructor, forward tensors, and recurrent-state internals: model-api.
• Dataset layouts and training stages: training-data.
• Metric evaluation or speed benchmarking: evaluation-tools.

Inference workflow

1. Confirm the runtime surface.

   • For source-checkout PyTorch inference, import MattingNetwork and convert_video from the local RVM source modules.
   • For TorchHub, expect network access unless weights are already cached.
   • For exported formats, follow the tensor I/O contracts in references/model-loading.md.

2. Load a model.

   import torch
   from model import MattingNetwork
   
   model = MattingNetwork("mobilenetv3").eval().to("cuda")
   model.load_state_dict(torch.load("rvm_mobilenetv3.pth", map_location="cuda"))
   

3. Convert a video file or sorted image-sequence directory with the converter API. Always request at least one output.

   from inference import convert_video
   
   convert_video(
       model,
       input_source="frames_or_input.mp4",
       output_type="png_sequence",
       output_composition="composition",
       output_alpha="alpha",
       downsample_ratio=0.25,
       seq_chunk=4,
   )
   

4. For safe PNG image-sequence conversion from arbitrary working directories, use the bundled wrapper:

   python scripts/rvm_convert_image_sequence.py \
     --repo-root /path/to/RobustVideoMatting \
     --variant mobilenetv3 \
     --checkpoint rvm_mobilenetv3.pth \
     --input-dir frames \
     --output-dir rvm_outputs \
     --device cpu \
     --alpha --composition
   

5. Validate outputs. PNG sequence mode writes numbered images under the output directories you request. Video mode uses PyAV/H.264 and can fail for media dependency or codec reasons unrelated to model quality.

Bundled references and script

• Read references/converter-reference.md for the verified convert_video signature, CLI flags, output modes, and downsample/sequence-chunk behavior.
• Read references/model-loading.md for PyTorch, TorchHub, TorchScript, ONNX, TensorFlow, TensorFlow.js, and CoreML loading and tensor I/O contracts.
• Read references/troubleshooting.md for checkpoint, media dependency, output selection, device/dtype, recurrent-state, and exported-runtime failures.
• Run scripts/rvm_convert_image_sequence.py when the user has a local checkpoint plus frame directory and wants safe PNG outputs without invoking the original repo script directly.

Key decisions

• Prefer mobilenetv3 unless the user explicitly asks for the larger ResNet50 variant.
• Use output_type="png_sequence" for debuggable alpha/foreground/composition artifacts; use output_type="video" only when video encoding dependencies are available and a video container is required.
• Leave downsample_ratio=None for auto max-side-512 behavior, or set it based on resolution/content. For 1080p portrait video, 0.25 is a common starting point.
• Increase seq_chunk to process multiple sequential frames at once when memory permits. The converter still recycles recurrent states across chunks.
• Do not automate pretrained weight downloads in generated scripts; ask users to provide explicit checkpoint paths or use TorchHub with clear network/cache expectations.

Acceptance check for inference answers

A good answer names the selected runtime, exact converter arguments or CLI flags, required input/output paths, checkpoint/device handling, validation steps, and likely failure modes. It should not require future agents to open the original repository docs or scripts.