---

name: visual-pushing-grasping description: "Route Visual Pushing and Grasping workflows for RGB-D robotic manipulation, including VPG model training/testing, heightmap geometry, CoppeliaSim simulation, session evaluation, and guarded UR5/RealSense operation." disable-model-invocation: true metadata: disco-role: operating license: BSD 2-Clause

Visual Pushing and Grasping

Use this operating skill when a task involves the historical andyzeng/visual-pushing-grasping implementation: complementary pushing and grasping from RGB-D observations, pixel-wise affordance/Q maps, V-REP or CoppeliaSim scenes, VPG transition logs, or a UR5 plus RealSense camera.

Route the request

• Train, resume, or test a reactive/reinforcement VPG policy: read training. It owns CLI flags, model behavior, snapshots, logging, exploration, replay, and CPU/CUDA decisions.
• Project RGB-D data into robot-frame heightmaps or validate calibration: read perception-geometry. It owns array shapes, units, workspace bounds, depth scale, and rigid transforms.
• Run or author a simulated manipulation scenario: read simulation. It owns CoppeliaSim/V-REP startup, remote API prerequisites, object meshes, preset files, and safe preflight validation.
• Summarize a completed session or plot training curves: read evaluation. It owns transition-log schemas, reactive/reinforcement metric thresholds, and headless plots.
• Connect a RealSense stream, calibrate, or operate an UR5: read real-robot. It is safety-first and never starts physical motion implicitly.

For requests spanning routes, begin here, then load only the owning sub-skills: perception-geometry supplies camera/heightmap contracts to training and real-robot; simulation or real-robot supplies observations/actions to training; evaluation consumes the logs produced by training.

Runtime boundary

This is a historical research implementation, not a packaged distribution. The source artifact has no pyproject.toml, setup.py, or console entry point. Use an isolated environment with the public numerical requirements (NumPy, SciPy, OpenCV-Python, Matplotlib, PyTorch, and TorchVision) before any operator-supplied application run. Current-Python checks are bounded compatibility diagnostics only; the full loop may require an older compatible Torch/Python combination, especially for the historical pretrained snapshot.

Let <skill-root> mean the directory containing this root SKILL.md. The self-contained environment check is the only root runtime prerequisite probe:

python <skill-root>/scripts/check_environment.py

The historical source-module import check was construction evidence only; it is not a runtime instruction and must not be used after this graph is deployed. Use --cpu for a deliberate CPU run. CUDA is optional for correctness but practically useful for training; verify it with the bundled helper rather than assuming a visible device is usable. Read troubleshooting.md before changing Torch, NumPy, snapshot, or simulator versions.

Safety and verification gates

Do not start the main loop merely to inspect the CLI: construction opens a camera/robot adapter and can connect to external services. First run parser help, validate paths and preset schemas with the bundled helpers, and confirm that the requested external service or hardware is present. Never run calibration, touch/debug, grasp, push, or restart procedures without a human approved workspace and an explicit stop plan.

The generated graph deliberately does not bundle a V-REP/CoppeliaSim main loop, scene, native remote-API binary, RealSense SDK libraries, UR5 services, model weights, or large presentation assets. Those are operator-supplied external prerequisites and are described with their failure signals in the focused routes. The root helper is linked here: check_environment.py. Read repo-provenance.md before deciding whether this graph is stale for another checkout.

Shared source facts

• Observations are RGB-D images reprojected into a top-down heightmap; the default heightmap resolution is 0.002 metres per pixel.
• The model evaluates 16 rotations and separate push/grasp branches. Reactive mode emits class-like affordances; reinforcement mode emits Q values.
• 0 is the push action and 1 is the grasp action in transition logs.
• Session evaluation interprets reactive grasp success as reward 0, and reinforcement grasp success as reward at least 0.5; pushes never count as successful grasps.
• Simulation uses the implemented V-REP remote API endpoint on port 19997. The physical camera streamer defaults to TCP port 50000; UR5 command and real-time channels default to 30002 and 30003.

All detailed commands, data contracts, and recovery steps live in the linked sub-skills and references rather than in this router.