---

name: cluster-lifecycle description: "Guide TensorFlowOnSpark cluster startup, reservation, shutdown, TensorBoard, TF_CONFIG, GPU/resource selection, and independent TFParallel.run jobs." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

cluster-lifecycle

Use this route when the task is about starting or stopping a TensorFlowOnSpark cluster, reserving executors, debugging reservation waits, interpreting TF_CONFIG or TensorBoard setup, choosing GPUs or Spark resources, or running independent TFParallel.run jobs.

In scope

• TFCluster.run(...) and the returned TFCluster object's train, inference, shutdown, and tensorboard_url methods.
• TFSparkNode.TFNodeContext and the context methods used by map functions.
• reservation.Server and reservation.Client, including the reservation handshake that coordinates executor startup.
• TFParallel.run(...) for independent, one-process-per-executor jobs.
• gpu_info.* and util.single_node_env(...) for GPU discovery, CUDA selection, and host/runtime setup.

Out of scope

• Data-feed queue semantics, EndPartition, and batch/output handling.
• Spark ML TFEstimator / TFModel workflows.
• TFRecord/DataFrame conversion.
• Long example migrations and example-to-example conversions.

Read first

• API reference for signatures, object relationships, and internal flow.
• Cluster workflows for startup, shutdown, TensorBoard, TF_CONFIG, GPU, and TFParallel.run recipes.
• Troubleshooting for reservation waits, duplicate executor ids, Windows/background limits, Spark config, and GPU failures.

Use the bundled scripts

• tfos_cluster_env_check.py to check Python/package, Spark/Java, GPU visibility, and background-mode prerequisites before cluster work.
• request_streaming_stop.py to request a reservation-server stop for a streaming job; it is dry-run by default and only sends the control message when --execute is supplied.

Typical routing cues

• “cluster hangs waiting for reservations”
• “duplicate executor id”
• “TensorBoard URL missing”
• “TF_CONFIG not set”
• “GPU allocation failed”
• “background mode on Windows”
• “spark.python.worker.reuse”
• “run several independent TensorFlow jobs on Spark executors”

Common decisions

1. Decide whether this is a shared TFCluster job or an independent TFParallel.run job.
2. Decide the execution mode: InputMode.TENSORFLOW or InputMode.SPARK.
3. Decide whether you need TensorBoard, driver_ps_nodes, master_node, eval_node, or release_port=False.
4. If your map_fun starts its own TF gRPC server, reserve the port with release_port=False and call ctx.release_port() before binding the server.
5. Decide whether the job should request GPUs, rely on Spark resource assignment, or fall back to CPU.
6. If a streaming job must stop from the driver side, use the bundled stop helper rather than guessing the reservation protocol.

Validation signals

• Reservation logs show every executor registered and TFCluster.run(...) returns a cluster object.
• cluster.tensorboard_url() is non-None only when TensorBoard was enabled and a node advertised a port.
• shutdown() completes without late exceptions and stops the reservation server.
• TFParallel.run(...) returns after all executors complete, with no shared reservation state.

If the task is really about queue semantics, DataFeed batching, or model/dataframe pipelines, route it to the sibling workflow that owns those behaviors instead of expanding this one.