---

name: pointllm description: "Guide PointLLM point-cloud language-model installation, data preparation, CUDA inference, two-stage training, demos, and benchmark evaluation." disable-model-invocation: true metadata: disco-role: operating license: CC BY-NC-SA 4.0

PointLLM

Use this repo skill when a task involves PointLLM, colored point-cloud language models, PointBERT point tokens, Objaverse 8192-point inputs, ModelNet40 zero-shot classification, PointLLM chat, the Gradio demo, two-stage training, or the repository's benchmark evaluators.

This is an operating router, not a copy of the implementation. Read the smallest route that owns the task and keep the original result files and credentials auditable.

Choose a route

• Data and schemas — read data-preparation for Objaverse NPY files, instruction JSON, ModelNet .dat data, normalization, sampling, dataset classes, and the bundled local validator.
• Chat, batch generation, or demo — read inference-serving for model/tokenizer registration, point-token prompts, CUDA/dtype budgets, file or object-ID inputs, batch-generation flags, output JSON, and Gradio operations.
• Training — read training for Stage 1 projector alignment, Stage 2 instruction tuning, dataclass flags, checkpoint handoff, FSDP/FlashAttention constraints, and preflight validation. It never launches distributed training automatically.
• Scoring and metrics — read evaluation for benchmark selection, result validation, OpenAI judging, resume files, cost accounting, traditional caption metrics, and invalid-response handling.

For a task spanning routes, start here, then follow the explicit sibling links. Data preparation normally precedes inference or training; inference precedes benchmark scoring.

Installation and compatibility gate

PointLLM is distribution pointllm version 0.1.2, imported as pointllm. The repository metadata requires Python >=3.8, but the documented and inspected baseline is Python 3.10 with torch 2.0.1 + CUDA 11.7, Transformers 4.28.0.dev0 from commit cae78c46, and tokenizers 0.12.1. Use an isolated environment; do not install this legacy stack into an unrelated application environment.

From a PointLLM checkout, install the package and its declared dependencies:

pip install -e .

For the documented GPU baseline, install a torch/torchvision pair compatible with the host driver before the editable package. Training's train_mem path also requires a FlashAttention build compatible with that torch/Python/CUDA combination; this is a compiled optional acceleration with no safe generic wheel assumption. Deepspeed/FSDP are used by the Stage 2 profile. Keep numpy<2 with the older torch/Open3D stack, and use the legacy OpenAI client (openai==0.28.1) for the source evaluator API. Do not use a modern OpenAI client without adapting the evaluator.

Verify the environment before loading weights or data:

python -c "import torch, pointllm; print(torch.__version__, torch.cuda.is_available())"
python -m pip check
python scripts/check_env.py

Run that command from this generated skill directory; the bundled check is read-only and reports package/backend facts. It does not load a checkpoint, download data, contact OpenAI, or start a server. Read references/troubleshooting.md when imports, legacy dependency pins, CUDA, or optional services fail.

Runtime invariants

• Inference launchers call .cuda() and require a compatible CUDA runtime; this skill does not claim CPU inference.
• The normal colored input is a finite (N, 6) array: XYZ followed by RGB in [0, 1], usually N=8192. XYZ is centered and scaled to unit maximum radius.
• Point tokens are checkpoint-configured; use the model's point_token_len and configured start/end tokens rather than hard-coding a prompt layout.
• The model context is 2048 total tokens. Point tokens consume context before the user's question and generated answer.
• Full checkpoints, Objaverse files, ModelNet files, and GPT judging are external resources. Validate paths and schemas before spending GPU, network, or API budget.

Provenance and refresh

Read references/repo-provenance.md before assuming this route matches a changed checkout. The structured routing record is in references/repo-routing-metadata.json. The shared installed-package launcher is scripts/run_installed_cli.py; use it from this generated skill directory for the historical chat/evaluation files. If the source commit, package metadata, or public entry points differ, use a refresh workflow instead of silently relying on this snapshot.

Scope limits

This skill does not embed model weights, datasets, API keys, external services, large benchmark outputs, or private environment paths. It does not promise that an expensive training run, full benchmark, FlashAttention compilation, Objaverse download, Gradio exposure, or OpenAI evaluation has succeeded merely because the package imports.