---

name: triton description: >- Serves trained machine-learning models over HTTP and gRPC with NVIDIA Triton Inference Server: one server for ONNX, TensorRT, PyTorch, TensorFlow, OpenVINO and Python models, with dynamic batching, model ensembles, versioning and Prometheus metrics, on GPU or CPU. Use when a user asks to deploy a model for production inference, write a config.pbtxt, batch requests for GPU throughput, chain preprocessing and a model, or call Triton from Python. license: Apache-2.0 compatibility: "Docker with the NVIDIA Container Toolkit for GPU serving (the server also runs on CPU only), Linux. Images come from NVIDIA NGC (nvcr.io) and are several gigabytes." metadata: author: terminal-skills version: "1.1.0" category: data-ai tags: - inference-server - model-serving - nvidia - dynamic-batching - model-ensembles repository: https://github.com/triton-inference-server/server

NVIDIA Triton Inference Server

Overview

Triton loads models from a directory called the model repository and serves each of them over HTTP (port 8000), gRPC (8001) and Prometheus metrics (8002), using the KServe v2 inference protocol. Each model is run by a backend (onnxruntime, tensorrt, pytorch, tensorflow, openvino, python, fil, vllm and others), so one server can host models from different frameworks. The server is released together with an NGC container of the same year.month: release 2.73.0 is container 26.09 (checked October 2026). Use the newest container your NVIDIA driver supports; the release notes list the minimum driver for each tag.

Instructions

Step 1: Lay out the model repository

model_repository/
├── text_classifier/
│   ├── config.pbtxt
│   ├── 1/model.onnx          # version 1
│   └── 2/model.onnx          # version 2 (latest is served by default)
├── image_model/
│   ├── config.pbtxt
│   └── 1/model.plan          # TensorRT engine
├── preprocess/
│   ├── config.pbtxt
│   └── 1/model.py            # Python backend
└── ensemble_pipeline/
    ├── config.pbtxt
    └── 1/                    # empty directory for an ensemble

Default file names are fixed per backend: model.onnx, model.plan, model.pt (TorchScript), model.savedmodel/, model.py. For ONNX, TensorRT and OpenVINO models Triton can generate the minimal configuration (inputs, outputs, max_batch_size) itself, so config.pbtxt may be left out until you need batching or instance settings. Models of other types, such as TorchScript, need a config file. To see what Triton generated, run curl localhost:8000/v2/models/text_classifier/config.

Step 2: Start the server

docker run --gpus all --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 \
    -v "$(pwd)/model_repository:/models" \
    nvcr.io/nvidia/tritonserver:26.09-py3 \
    tritonserver --model-repository=/models

Drop --gpus all to run on CPU only (use KIND_CPU in instance_group). The log ends with a table of models; each should show READY, otherwise the table gives the load error. curl -s -o /dev/null -w "%{http_code}" localhost:8000/v2/health/ready returns 200 when the server is ready. With --model-control-mode=explicit models load only on request (--load-model=text_classifier or the repository API).

Step 3: Write config.pbtxt

# model_repository/text_classifier/config.pbtxt
name: "text_classifier"
backend: "onnxruntime"          # older configs use: platform: "onnxruntime_onnx"
max_batch_size: 64

input [
  { name: "input_ids"      data_type: TYPE_INT64 dims: [ 128 ] },
  { name: "attention_mask" data_type: TYPE_INT64 dims: [ 128 ] }
]
output [
  { name: "logits" data_type: TYPE_FP32 dims: [ 2 ] }
]

dynamic_batching {
  max_queue_delay_microseconds: 100
}

instance_group [
  { count: 2 kind: KIND_GPU gpus: [ 0 ] }
]

With max_batch_size above 0, the batch dimension is implicit: dims describes one sample and requests carry shape [batch, 128]. Dynamic batching merges concurrent requests up to that size; the queue delay is how long Triton waits to fill a batch. NVIDIA advises leaving out preferred_batch_size for most models, except TensorRT models with several optimization profiles. instance_group runs two copies of the model on GPU 0 so requests overlap. Measure with Perf Analyzer (in the -py3-sdk container) before tuning.

Step 4: Call the server

# http_client.py — plain HTTP, no Triton library needed
import numpy as np
import requests

TRITON_URL = "http://localhost:8000"
assert requests.get(f"{TRITON_URL}/v2/health/ready").status_code == 200

ids = np.random.randint(0, 30000, (1, 128))
payload = {"inputs": [
    {"name": "input_ids", "shape": [1, 128], "datatype": "INT64", "data": ids.flatten().tolist()},
    {"name": "attention_mask", "shape": [1, 128], "datatype": "INT64", "data": [1] * 128},
]}
reply = requests.post(f"{TRITON_URL}/v2/models/text_classifier/infer", json=payload, timeout=30)
reply.raise_for_status()
print(reply.json()["outputs"][0]["data"])      # two logits for the one sample

# grpc_client.py — pip install "tritonclient[grpc]"
import numpy as np
import tritonclient.grpc as grpcclient

client = grpcclient.InferenceServerClient(url="localhost:8001")
assert client.is_model_ready("text_classifier")

input_ids = np.random.randint(0, 30000, (4, 128)).astype(np.int64)    # batch of 4
attention_mask = np.ones((4, 128), dtype=np.int64)
inputs = [grpcclient.InferInput("input_ids", input_ids.shape, "INT64"),
          grpcclient.InferInput("attention_mask", attention_mask.shape, "INT64")]
inputs[0].set_data_from_numpy(input_ids)
inputs[1].set_data_from_numpy(attention_mask)

result = client.infer("text_classifier", inputs, outputs=[grpcclient.InferRequestedOutput("logits")])
print(result.as_numpy("logits").shape)         # (4, 2)

The tritonclient version follows the server release (2.73.0 today). Extras are http, grpc, cuda, perf-analyzer and all.

Step 5: Python backend and ensembles

A Python model is a class named TritonPythonModel in 1/model.py, with a config that sets backend: "python":

# model_repository/preprocess/config.pbtxt
name: "preprocess"
backend: "python"
max_batch_size: 64
input  [ { name: "raw_text"       data_type: TYPE_STRING dims: [ 1 ] } ]
output [ { name: "input_ids"      data_type: TYPE_INT64  dims: [ 128 ] },
         { name: "attention_mask" data_type: TYPE_INT64  dims: [ 128 ] } ]

# model_repository/preprocess/1/model.py
import numpy as np
import triton_python_backend_utils as pb_utils

class TritonPythonModel:
    def execute(self, requests):
        responses = []
        for request in requests:
            texts = pb_utils.get_input_tensor_by_name(request, "raw_text").as_numpy()  # shape (batch, 1), bytes
            ids = np.zeros((len(texts), 128), dtype=np.int64)
            mask = np.zeros((len(texts), 128), dtype=np.int64)
            for row, item in enumerate(texts):
                tokens = [ord(c) for c in item[0].decode("utf-8")[:128]]   # stand-in for a real tokenizer
                ids[row, :len(tokens)] = tokens
                mask[row, :len(tokens)] = 1
            responses.append(pb_utils.InferenceResponse(output_tensors=[
                pb_utils.Tensor("input_ids", ids), pb_utils.Tensor("attention_mask", mask)]))
        return responses     # one response per request, in order

An ensemble wires the steps together without client round trips; the keys in input_map and output_map are each model's own tensor names, the values are names inside the pipeline:

# model_repository/ensemble_pipeline/config.pbtxt
name: "ensemble_pipeline"
platform: "ensemble"
max_batch_size: 64
input  [ { name: "raw_text" data_type: TYPE_STRING dims: [ 1 ] } ]
output [ { name: "logits"   data_type: TYPE_FP32   dims: [ 2 ] } ]
ensemble_scheduling {
  step [
    { model_name: "preprocess" model_version: -1
      input_map  { key: "raw_text"       value: "raw_text" }
      output_map { key: "input_ids"      value: "ids" }
      output_map { key: "attention_mask" value: "mask" } },
    { model_name: "text_classifier" model_version: -1
      input_map  { key: "input_ids"      value: "ids" }
      input_map  { key: "attention_mask" value: "mask" }
      output_map { key: "logits"         value: "logits" } }
  ]
}

model_version: -1 means the latest version. For branching or loops, use Business Logic Scripting in the Python backend instead of an ensemble.

Examples

Example 1: Serve an exported ONNX model on a single GPU

Request: "I exported sentiment.onnx from PyTorch; serve it so the app can POST to it."

mkdir -p model_repository/sentiment/1 && cp sentiment.onnx model_repository/sentiment/1/model.onnx
docker run --gpus all --rm -p 8000:8000 -v "$(pwd)/model_repository:/models" \
    nvcr.io/nvidia/tritonserver:26.09-py3 tritonserver --model-repository=/models
curl -s localhost:8000/v2/models/sentiment/config      # the configuration Triton generated

With no config.pbtxt, the startup log lists sentiment | 1 | READY, and the config endpoint shows the inputs, outputs and batch size taken from the file. Copy those into a config.pbtxt and add dynamic_batching { } once you need batching.

Example 2: A model that will not load

Request: "Triton says my model is UNAVAILABLE."

Read the status table at startup; the reason is printed in the last column, for example a name mismatch such as unexpected inference input 'ids', allowed inputs are: input_ids, attention_mask. Fix name and dims in config.pbtxt to match the model (curl localhost:8000/v2/models/<name>/config shows the generated values for ONNX), then restart or reload with the model-control API. A wrong file name inside the version directory (for example model.onnx expected, classifier.onnx found) gives a similar failure.

Guidelines

• Pin the container tag; do not use latest. A GPU image needs a recent enough NVIDIA driver, and the SDK container (-py3-sdk) matches the server release.
• Never expose ports 8000 to 8002 to the internet. Triton has no authentication by default; put it behind a gateway, and keep the model-control and repository endpoints private.
• dims never includes the batch dimension when max_batch_size is above 0; with max_batch_size: 0 it must include it.
• Do not guess dynamic-batching numbers: benchmark with Perf Analyzer, and raise max_queue_delay_microseconds only if latency targets allow it.
• Triton metrics may not work when another DCGM agent runs on the same host; the release notes list this known issue.
• Container memory growth after unloading models is often allocator behavior; LD_PRELOAD with tcmalloc or jemalloc (both shipped in the image) is the documented mitigation.
• For serving a single large language model, a dedicated engine (vLLM, TensorRT-LLM; Triton has backends for both) may be simpler than a generic repository.