---

name: serving-and-clients description: "Serve BentoML services locally over HTTP or gRPC and call endpoints with Python clients, curl/OpenAPI, streaming, websocket clients, and operational serve flags." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

BentoML Serving And Clients

Use this sub-skill when the user needs to run a BentoML service locally, choose HTTP versus gRPC serving, call service endpoints from Python or curl, inspect endpoint schemas, or debug client/serve failures. For defining @bentoml.service and @bentoml.api, use ../service-authoring/SKILL.md. For bentoml build, containerization, and Bento packaging, use ../packaging-and-containerization/SKILL.md. For metrics, logs, tracing, scaling, and runtime resource configuration, use ../observability-and-operations/SKILL.md. For BentoCloud deployments and cloud endpoints, use ../cli-and-cloud/SKILL.md.

Local Serving

• Start HTTP serving with bentoml serve SERVICE_TARGET, where SERVICE_TARGET is an import target such as service.py:Summarization, a local Bento tag such as fraud_detector:latest, or a Bento directory.
• Prefer explicit source serving commands during development: bentoml serve service.py:ServiceClass --working-dir . --development --reload --port 3000.
• Production mode is the CLI default; use --development for local iteration because it uses one API worker and, with --reload, restarts on code/model-store changes.
• Set --working-dir whenever the service module is not in the current shell directory; this prevents import target failures caused by the wrong sys.path.
• Use --host, --port, --api-workers, --backlog, --timeout, and SSL flags for local server behavior; environment variables BENTOML_HOST, BENTOML_PORT, BENTOML_API_WORKERS, and BENTOML_TIMEOUT can also supply values.
• Use bentoml serve-grpc SERVICE_TARGET only when the caller needs gRPC; it is hidden in the CLI but implemented, supports reflection/channelz flags, and may require installing BentoML with gRPC dependencies.

See references/cli-reference.md and the bundled dry-run helper scripts/serve_command_builder.py for safe command construction.

HTTP Client Calls

• Use bentoml.SyncHTTPClient(url, token=None, timeout=30, server_ready_timeout=None) for blocking Python calls and bentoml.AsyncHTTPClient(...) for async code.
• Prefer context managers so connections close cleanly: with bentoml.SyncHTTPClient("http://localhost:3000") as client: or async with bentoml.AsyncHTTPClient("http://localhost:3000") as client:.
• Client methods are created from the server’s /schema.json; call client.<endpoint>(...) when the endpoint name is a valid Python attribute or client.call("endpoint_name", ...) when it is dynamic.
• Match arguments to the service API signature. Normal structured inputs use keyword arguments such as client.summarize(text="..."); root-input APIs require exactly one positional argument.
• Set server_ready_timeout when connecting while a server is booting, and use client.is_ready() to check /readyz before sending expensive calls.
• For authenticated endpoints, pass token="..."; if omitted, the client reads BENTO_CLOUD_API_KEY and sends Authorization: Bearer <token> when available.

See references/client-reference.md and scripts/inspect_service_client.py for schema inspection without starting a server.

Curl, OpenAPI, And Schema

• The local UI is available from the server root, the client schema is at /schema.json, readiness is at /readyz, and OpenAPI JSON is exposed under the docs route as docs.json.
• For JSON APIs, send POST /<route> with Content-Type: application/json; the hello-world endpoint shape is curl -X POST http://localhost:3000/summarize -H 'Content-Type: application/json' -d '{"text":"..."}'.
• If the API uses root input, the raw request body may be a scalar, bytes/text, or a serialized JSON value rather than an object with named fields.
• For files, the Python client switches to multipart when the schema contains file fields; with curl, mirror the schema’s content type and multipart field names.
• For endpoint-name confusion, inspect /schema.json first; API method names, custom route=... values, and visible HTTP paths can differ.

Streaming And WebSockets

• A BentoML API that returns a Python generator or async generator is exposed as a streaming HTTP response; the Sync client returns an iterator and the Async client returns an async iterator for stream outputs.
• Iterate streaming results instead of forcing a single response value: for chunk in client.generate(prompt="..."): or async for chunk in await client.generate(prompt="..."): depending on the async method’s return.
• WebSocket endpoints are mounted through FastAPI/ASGI apps, usually with @bentoml.asgi_app(app, path="/chat"); the BentoML Python HTTP clients do not support WebSockets.
• Use a WebSocket library such as websockets and connect to the mounted path plus websocket route, for example ws://localhost:3000/chat/ws.
• Keep service-authoring details for ASGI mounting in ../service-authoring/SKILL.md; this sub-skill focuses on calling and debugging the running server.

gRPC Serving

• Use bentoml serve-grpc SERVICE_TARGET --port 3000 --host 0.0.0.0 for local gRPC serving when the environment includes gRPC support.
• gRPC serving exposes BentoML’s generated service protocol and health service; tests use grpc.health.v1.Health/Check and generated BentoML stubs.
• Useful gRPC flags include --enable-reflection, --enable-channelz, --max-concurrent-streams, --protocol-version v1|v1alpha1, --api-workers, --backlog, and SSL cert/key/CA options.
• On Windows, production gRPC serving has limitations around SO_REUSEPORT; use --development for local testing or containerize on Linux for production-like behavior.

Helper Scripts

• scripts/serve_command_builder.py builds shell-safe bentoml serve or bentoml serve-grpc commands and never starts a server unless a user copies the output.
• scripts/inspect_service_client.py requires an explicit URL, checks /readyz and /schema.json, and prints endpoint routes, input/output schemas, streaming flags, and task flags.
• Keep all script examples self-contained and avoid pointing runtime guidance at source-repo docs, tests, or local checkout paths.

Troubleshooting Checklist

• Import target fails: use module.py:ClassName or module:svc, pass --working-dir, and confirm the module imports from that directory.
• Port conflict: change --port, clear the process occupying the port, or use a separate HTTP and gRPC port.
• Readiness timeout: increase server_ready_timeout, check /readyz, and verify expensive model loading or startup hooks are not failing.
• Wrong endpoint name: fetch /schema.json, then call the route’s name with client.call(name, ...) or the generated method.
• Request schema error: compare your kwargs to the endpoint input schema; root inputs take one positional argument and reject keyword arguments.
• Auth token error: pass token= explicitly or set BENTO_CLOUD_API_KEY; verify the endpoint actually expects BentoCloud-style bearer auth.
• Missing gRPC support: install the gRPC extra/dependencies before using serve-grpc or generated gRPC clients.
• Async lifecycle leak: use async with bentoml.AsyncHTTPClient(...) or call await client.close().

For deeper failure mapping, see references/troubleshooting.md.