---

name: prompted-segmentation description: "Use SAM's SamPredictor for prompt-based masks from points, labels, boxes, iterative low-res mask refinement, image embeddings, and CPU/GPU checkpoint setup." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

Prompted Segmentation

Use this sub-skill when the user wants Segment Anything (SAM) masks for a specific object or region using prompts such as foreground/background points, boxes, prior masks, or repeated prompts on the same image.

When to Use

• Segment an object from a point prompt, box prompt, or combined box plus foreground/background points.
• Reuse one image embedding for multiple prompt attempts with SamPredictor.set_image(...) followed by repeated predict(...) calls.
• Refine a mask interactively by passing the best previous low_res_masks slice back as mask_input.
• Debug prompt shape, image format, checkpoint, model type, or device mismatch failures.
• Extract image embeddings for an ONNX/browser handoff with get_image_embedding(); route ONNX export and browser runtime work to ../onnx-and-browser/.

For full-image proposal generation without prompts, use ../automatic-mask-generation/ instead.

Core Pattern

1. Choose a model type from default, vit_h, vit_l, or vit_b, and pass the matching user-provided checkpoint path to sam_model_registry[model_type](checkpoint=...).
2. Move the SAM model to the selected device before creating or using the predictor: sam.to(device=device).
3. Create predictor = SamPredictor(sam) and call predictor.set_image(image, image_format="RGB") once per image.
4. Pass prompts in original image pixel coordinates: points as Nx2 (x, y), point labels as length N with 1 foreground and 0 background, and boxes as XYXY.
5. Use multimask_output=True for ambiguous single-click prompts; use False for boxes or multiple prompts when one mask is desired.
6. Read outputs as (masks, scores, low_res_masks), where masks is CxHxW, scores is length C, and low_res_masks is Cx256x256 logits for refinement.

Required Details

• API signatures, return shapes, and direct Sam batch notes: references/api-reference.md.
• Prompting workflows, embedding reuse, and iterative refinement: references/workflows.md.
• Diagnosis for common runtime failures and shape errors: references/troubleshooting.md.
• Safe command-line helper template: scripts/predictor_prompt_template.py.

Minimal Prompt Example

import numpy as np
import torch
from segment_anything import SamPredictor, sam_model_registry

model_type = "vit_b"
device = "cuda" if torch.cuda.is_available() else "cpu"
sam = sam_model_registry[model_type](checkpoint="/path/to/sam_vit_b.pth")
sam.to(device=device)
predictor = SamPredictor(sam)
predictor.set_image(image_rgb_uint8_hwc, image_format="RGB")

point_coords = np.array([[500, 375]], dtype=np.float32)
point_labels = np.array([1], dtype=np.int32)
masks, scores, low_res_masks = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=True,
)
best_mask = masks[int(np.argmax(scores))]