---

name: training description: "Configure and safely operate PointLLM's two-stage training workflow, flags, checkpoints, and resource-sensitive validation without launching distributed training." disable-model-invocation: true metadata: disco-role: operating license: CC BY-NC-SA 4.0

PointLLM training

Use this sub-skill when a Researcher needs to configure, inspect, resume, or troubleshoot PointLLM v1 training. It covers the repository's two stages and its custom dataclass flags. It does not download data or weights, change an environment, or launch torchrun.

Route first

1. Establish local paths for an initial Hugging Face model directory, the point-cloud directory, its annotation JSON, and a writable output directory.

2. Choose exactly one stage:

   • Stage 1 aligns the language model with PointBERT features. The shipped profile fixes the LLM and point backbone and tunes the point projector (new point-token input embeddings can also become trainable). It loads a separate point-backbone checkpoint.
   • Stage 2 starts from the complete Stage-1 output, enables LLM tuning, uses the complex-instruction conversations, and keeps the point backbone in no-gradient/eval execution with the shipped fix_pointnet=True.

3. Run the bundled validator before editing a command or starting a job:

   python scripts/validate_training_config.py \
     --stage 1 --config training.json
   

   Add --allow-missing only for a path-planning dry run. The validator never invokes torchrun, imports the training entry point, or loads a checkpoint.

4. Read workflows.md for inert command templates and checkpoint/resume behavior. Read configuration.md before changing a flag. Use troubleshooting.md when a check fails.

Required stage invariants

• Stage 1: stage_2=False, model_name_or_path is the initial LLM/PointLLM directory, and point_backbone_ckpt points to the compatible PointBERT file.
• Stage 2: stage_2=True, model_name_or_path is the Stage-1 final output; do not point it at the initial directory unless intentionally restarting.
• Both stages require an annotation JSON and point-cloud directory. With use_color=True, each selected file is expected to be (N, 6) with XYZ in the first three columns and RGB in the final three columns.
• Keep pointnum=8192 unless the model's PointBERT configuration and every {object_id}_{pointnum}.npy file have been changed together.
• Do not infer successful training from a created output directory. Confirm logs, loss progress, saved model state, and (when applicable) point_proj.bin.

Resource and safety gates

• The reference profile uses one node and eight processes. This is a launch template, not an assertion that the current machine can run it.
• bf16=True requires a compatible accelerator and matching PyTorch/CUDA runtime. The inspection baseline supplied for this skill is Python 3.10, torch 2.0.1+cu117, Transformers 4.28.0.dev0 at commit cae78c46, tokenizers 0.12.1, and an A100 40GB (compute capability 8.0). Re-check the actual runtime rather than copying this baseline into a new environment.
• train_mem.py applies the repository's FlashAttention monkey patch before importing the model. A compatible compiled flash-attn is therefore a hard precondition for that entry point; train.py avoids that explicit patch but is not a performance or memory-equivalent fallback.
• Stage 2's full_shard auto_wrap and LlamaDecoderLayer settings use the source's experimental partially-frozen/FSDP path. Treat FSDP warnings, missing use_orig_params support, or backend/version mismatch as stop conditions; do not silently remove FSDP or unfreeze the point backbone.
• Stop before launch if the validator reports a stage contradiction, missing required input, incompatible point-backbone configuration, or unavailable required compiled backend. Stop after launch if the first batch cannot load, point-token counts disagree, or no trainable parameters receive loss.

Outputs and handoff

The training code auto-resumes when output_dir already contains any checkpoint-* directory. The templates set save_strategy="no", so ordinary runs do not create periodic checkpoints. PointLLMTrainer writes adapter state under point_proj/ for checkpoint saves and writes a final point_proj.bin when adapter tuning is enabled; the final save also uses the Hugging Face model serialization path. Preserve the output directory and trainer state together when handing Stage 1 to Stage 2.

This sub-skill links only to its bundled references and validator. It records source-derived behavior; it does not claim that a training run succeeded.