---

name: "detection-inference" description: "Run and troubleshoot the repository's TensorRT YOLOv3/YOLOv4 and UFF SSD object-detection demos across files, streams, cameras, asynchronous pipelines, visualization, and MJPEG output." disable-model-invocation: true metadata: disco-role: operating license: MIT

TensorRT detection inference

Use this skill when the task is to run, adapt, or diagnose the repository's YOLOv3/YOLOv4 or UFF SSD inference examples. It covers the inference and presentation path only; engine conversion, model downloads, training, and mAP evaluation are outside this skill. Keep experiment logs and review artifacts outside this runtime skill directory.

Applicability and hard gates

• YOLO: TrtYOLO consumes a serialized engine named by the model under the runtime's yolo/ artifact directory and loads the yolo_layer TensorRT plugin before constructing CUDA buffers. The model string describes a YOLOv3/YOLOv4 family and input size, for example yolov4-416 or yolov3-tiny-288; custom names are possible if their engine and class count match. The plugin is not bundled here and must be built/provided by the runtime owner.
• SSD: TrtSSD consumes a serialized UFF-derived engine under the runtime's ssd/ artifact directory. The demo's supported model names are ssd_mobilenet_v1_coco, ssd_mobilenet_v1_egohands, ssd_mobilenet_v2_coco, ssd_mobilenet_v2_egohands, ssd_inception_v2_coco, and ssdlite_mobilenet_v2_coco.
• Both paths require a working TensorRT Python API, CUDA/PyCUDA context, and OpenCV. An input parser or this skill's argument checker cannot prove that an engine, plugin, camera, decoder, or CUDA device is usable.
• The evidence is historical repository behavior. The README reports YOLO inference with the custom plugin from TensorRT 6+ and UFF SSD with the TensorRT Python API from TensorRT 5.x+; it also reports tested Jetson Nano, TX2, Xavier NX, and selected x86 NVIDIA systems. Do not infer modern-stack compatibility from those claims.
• Stop and preserve the exact error when a modern TensorRT/PyCUDA/OpenCV stack cannot deserialize a legacy engine or load its plugin. Do not copy a binary, installer, engine, or shared library into this skill.

Fast routing

1. Identify the detector and its engine artifact before choosing an input.

2. Validate exactly one input mode with the bundled dependency-free checker:

   python3 scripts/validate-input-mode.py --detector yolo \
     --model yolov4-416 --image /path/to/image.jpg
   

   The checker validates only command arguments; it does not read the image, open a device, import TensorRT, or check an engine/plugin.

3. Run the matching native demo from its normal demo runtime environment. The native entrypoints import CUDA/TensorRT before argparse help can execute, so --help is a dependency smoke check rather than a pure parser check.

4. If the display is not needed, use a video-writer or MJPEG variant rather than removing inference code. Record detector, model, input mode, input dimensions, confidence threshold, letterbox mode, and TensorRT version.

See references/workflows.md for complete recipes, references/api-reference.md for contracts, and references/troubleshooting.md for failure classification.

Native entrypoints and detector contracts

YOLOv3/YOLOv4 — TrtYOLO

• The interactive entrypoint accepts --model (required), --category_num (default 80), --conf_thresh (default 0.3), and --letter_box.
• TrtYOLO.detect(img, conf_th=0.3, letter_box=None) accepts a BGR uint8 image shaped (height, width, 3) and returns (boxes, scores, classes). Boxes are integer (x1, y1, x2, y2) corners clipped to the original image; scores are confidence products; classes are numeric class IDs.
• Preprocessing resizes to the engine input, converts BGR to RGB, transposes to CHW, and scales to [0, 1]. Letterbox mode preserves aspect ratio with gray padding and reverses the offset during postprocessing; use it only when the engine/task expects that behavior.
• Postprocessing concatenates the plugin outputs, filters on box_confidence * class_probability >= conf_th, applies per-class NMS at IoU 0.5, maps coordinates back to the source image, and clips them. A non-positive category count is rejected by the interactive entrypoint.
• The code selects execute_async for TensorRT versions before 7 and execute_async_v2 for TensorRT 7+, while buffer allocation distinguishes implicit and explicit batch dimensions. This is a compatibility clue, not a guarantee for current TensorRT APIs.

UFF SSD — TrtSSD

• trt_ssd.py selects one of the six model names above and uses a fixed (300, 300) input. Its interactive loop uses confidence 0.3; the class API exposes detect(img, conf_th=0.3) for another threshold.
• SSD preprocessing resizes BGR to 300x300, converts to RGB, changes HWC to CHW, scales to [-1, 1], and runs the serialized UFF/TensorRT engine.
• SSD postprocessing reads seven-value detection records, ignores records below conf_th, scales normalized (xmin, ymin, xmax, ymax) to the original image, and returns corner boxes, confidences, and integer classes.
• The loader registers TensorRT plugins and, for TensorRT versions before 7, loads the legacy FlattenConcat shared library. A warning such as Could not register plugin creator: FlattenConcat_TRT is a known historical TensorRT 6-era observation in the README; treat it as a compatibility signal and verify actual deserialization/inference rather than suppressing it.

Input modes and frame ownership

The shared camera layer supports these modes. Pass one, not several; its implementation resolves them in this order: image, video, RTSP, USB, custom GStreamer, onboard camera.

Mode	Native arguments	Behavior and important options
Image	--image FILE	Reads one image and repeats a copy on every read(). --do_resize resizes it to --width/--height.
Video	--video FILE	Reads sequentially with OpenCV. --video_looping reopens it at EOF; --do_resize resizes frames.
RTSP	--rtsp URI	Builds an H.264 GStreamer pipeline, with --rtsp_latency in milliseconds (default 200). Requires an available omxh264dec or avdec_h264.
USB	--usb N	Opens /dev/videoN; the historical default uses a GStreamer v4l2src pipeline at the requested --width/--height.
GStreamer	--gstr STRING	Opens a caller-supplied GStreamer string and formats {width} and {height} placeholders.
Onboard	--onboard N	Opens the Jetson onboard camera via the available legacy nvcamerasrc or nvarguscamerasrc pipeline; the selector value is not used to choose a sensor in this implementation.

--width and --height default to 640x480. For RTSP, USB, GStreamer, and onboard sources they become pipeline dimensions; for image/video they affect resizing only when --do_resize is set. Live sources use a background grab thread. Camera.read() returns a shared live frame by default, so detection code that draws in place can feed its overlay back into the next inference. Use --copy_frame for live sources when inference/display can outpace frame arrival or when the frame is modified in place. Image reads are already copied.

Async pipeline and CUDA context

trt_ssd_async.py is a producer/consumer pipeline, not a generic async API:

1. The main thread opens Camera, initializes the PyCUDA driver, and creates a condition variable.
2. A child TrtThread creates its CUDA context inside the child thread, constructs TrtSSD, reads frames, calls detect, stores the latest image and result tuple in shared variables, and notifies the main thread.
3. The main thread waits up to 20 seconds, draws boxes/FPS, displays the frame, and stops/join the child on exit.

Preserve this context ownership rule when adapting the pipeline. A timeout is an actionable capture/inference/CUDA-thread failure, not an empty detection. The shared-slot design favors the newest result and can drop intermediate frames; it does not provide ordered delivery or backpressure. The YOLO interactive and MJPEG examples are synchronous loops. trt_yolo_cv.py is a synchronous file-to-file writer and uses a fixed 0.3 confidence threshold.

Visualization and network output

BBoxVisualization.draw_bboxes(img, boxes, confs, clss) draws each corner box in a deterministic per-class color and labels it with the class name and confidence to two decimal places. show_fps overlays an exponentially smoothed FPS value in the interactive loops. open_window creates a resizable OpenCV window; ESC exits and F toggles fullscreen. Drawing mutates the image in place, which is why --copy_frame matters for live camera sources.

trt_yolo_mjpeg.py uses the same YOLO detector and camera arguments, draws boxes/FPS, and sends each frame to MjpegServer (default port 8080). The server publishes multipart JPEG at /mjpg or /, keeps a queue of at most two frames, and drops a frame when the queue is full. It is an unauthenticated HTTP stream: bind it only on a trusted interface/network, choose a permitted port, and shut it down in finally. A disconnected client can produce a broken-pipe path that the handler intentionally ignores.

Evidence basis

This operating contract is distilled from README.md, the trt_yolo.py, trt_yolo_cv.py, trt_yolo_mjpeg.py, trt_ssd.py, and trt_ssd_async.py entrypoints, and the implementation modules utils/camera.py, utils/display.py, utils/visualization.py, utils/yolo_with_plugins.py, utils/ssd.py, and utils/mjpeg.py. The README's Jetson/TensorRT version notes and benchmark values are historical observations; source behavior and limits take precedence over undocumented assumptions. No source checkout path, engine, plugin, model, binary, or installer is part of this skill.

Verification and limits

Use the bundled checker first, then run native help candidates and record whether the failure is parser, dependency, backend, plugin, artifact, or input:

python3 scripts/validate-input-mode.py --help
python3 scripts/validate-input-mode.py --detector yolo --model yolov4-416 \
  --image /path/to/image.jpg
python3 scripts/validate-input-mode.py --detector ssd \
  --model ssd_mobilenet_v1_coco --video /path/to/video.mp4

The planned native candidates are trt_yolo.py --help and trt_ssd.py --help. On a host without PyCUDA, they fail before argparse with ModuleNotFoundError; that is an environment gate, not evidence that the flags are invalid. On a host with imports but without the legacy plugin, YOLO can fail while importing yolo_with_plugins.py. The observed repository limits include TensorRT 6+ for the YOLO plugin, TensorRT 5.x+ and legacy UFF/TensorFlow-era tooling for SSD, and a README-reported TensorRT 6 FlattenConcat plugin registration issue. Modern TensorRT, CUDA, Python, GStreamer, or OpenCV behavior must be verified on the target rather than extrapolated from historical JetPack results.