PyTorch 2.15 rewrote the GPU handshake. Test the rollback before the cluster hangs

THE SIGNAL IN ONE SENTENCE

A distributed training job is a group project where every participant must answer at exactly the right time. One GPU pauses. The others wait. The cluster looks busy, the bill keeps moving and the model goes nowhere. PyTorch 2.15 changes the machinery coordinating that conversation. The release makes a rewritten NCCL backend the default for every process group using NCCL, the main communication layer for distributed jobs on NVIDIA GPUs. PyTorch 2.14 offered the rewrite as an opt-in preview called nccl2. Version 2.15 moves it onto the ordinary nccl path. That is the plain-language signal. An infrastructure preview became a default. Treat that as a migration, even if the configuration file did not change. The old implementation is still available in this release through the nccl-legacy backend name or the TORCH_DIST_USE_NCCL2=0 environment variable. The official release notes say both rollback paths are scheduled for removal in the next release. That makes the fallback useful and temporary. The correct response is not to pin the old path forever. It is to use the remaining window to discover what the new default changes in your jobs, prove the rollback works and fix every dependency that assumes legacy behavior. Begin with a canary matrix. Choose one representative job for each topology that matters: single node, multiple nodes, tensor parallelism, data parallelism, pipeline parallelism and any job that places several GPUs inside one process. Run each on the current production stack and on PyTorch 2.15. Keep the model, data slice, seed, driver, NCCL or RCCL version, container, network fabric and launch command fixed. Compare initialization, peak device memory, collective latency, throughput, checkpoint behavior, recovery and shutdown. The last three deserve as much attention as speed. The new backend creates communicators eagerly when a process group is initialized. That can make failures appear sooner and make initialization more predictable. It can also allocate GPU memory for groups that never perform work. Measure the memory before the first training step. A job that fit yesterday may cross its limit before the model touches a batch. The new implementation also binds each process group to one CUDA device. PyTorch chooses a device from the supplied device_id or derives one from the local or global rank. An operation using a tensor on another GPU can now fail with a device mismatch. Pass device_id explicitly. Do not rely on a nearby torch.cuda.set_device call to express the process-group contract. The release notes also say jobs using several GPUs per process need the legacy backend in 2.15. That is not a footnote if your serving or training architecture uses that shape. Inventory those jobs now. Either redesign the ownership boundary, keep them on the temporary fallback with a dated removal plan or hold the framework upgrade until the supported path exists. Point-to-point traffic needs its own test. PyTorch documents a separate nccl-lazy option for jobs that need independent point-to-point communicators. Collectives use one eagerly created communicator while each send and receive peer pair gets a lazy communicator and stream. If a pipeline depends on overlapping communication between different peers, compare nccl and nccl-lazy deliberately. Do not let a performance regression become a mysterious production fact after rollout. Now rehearse the hang. PyTorch 2.15 adds a backend-agnostic flight recorder that writes a post-mortem when a collective hangs. Watchdog health is also exposed to the debug server, allowing a timeout on one rank to trigger recorder dumps from every rank. That is valuable because the first failed rank rarely tells the whole story. One process may be waiting inside an all-reduce. Another may have taken a different branch. A third may be stalled on data loading. The useful evidence is the synchronized picture of what every rank believed it was doing. Create a controlled failure in staging. Pause one rank before a collective, trigger the timeout and verify that every expected recorder artifact arrives. Check its path, ownership, retention, size and timestamp. Confirm that the operations surrounding the stall are visible and that the artifact survives container cleanup. A flight recorder that exists in release notes but disappears with an ephemeral pod is decorative. Wire the dump into the incident path before trusting it. The rollback drill should be just as concrete. Set TORCH_DIST_USE_NCCL2=0 on every rank before process-group creation or launch with nccl-legacy. Confirm the job really uses the old implementation. Run the same smoke test, checkpoint and controlled hang. Then write down exactly who can activate the fallback, where the setting lives, how to verify it and when it expires from your operating plan. The variable is not a forever switch. The project says it goes away in the next release. Treat every use as a migration bug with an owner. Rollback also needs to cover data. PyTorch 2.15 adds tensor strides to distributed-checkpoint metadata. The release notes warn that a checkpoint saved with 2.15 cannot be loaded by PyTorch 2.14 or earlier, and there is no flag to write the old format. That makes a package rollback different from a job rollback. If a 2.15 job writes the only checkpoint after twelve hours of training, returning the environment to 2.14 may leave the checkpoint unreadable. Preserve a known-good older checkpoint until the new stack passes recovery tests. Duplicate the first 2.15 checkpoint before using it as the only recovery point. Test restore into the exact target environment, not only save completion. The same caution applies beyond NCCL. PyTorch 2.15 changes its default wheel to CUDA 13.2 and offers CUDA 13.4 alongside it. It adds ROCm 10.0 wheels, expands Python support through 3.15 and ships platform work across NVIDIA, AMD, Intel and Apple hardware. That is a large compatibility surface. Record the exact wheel and hardware stack in every canary result. A green test on one CUDA wheel does not validate the other. An extension compiled against the previous toolchain may fail before the rewritten backend receives its first message. Keep framework, wheel, driver, fabric library and custom extensions in the same test receipt. Performance claims need their proper boundary too. The release describes faster low-precision decoding on Blackwell GPUs, faster Apple Silicon kernels and new distributed-training controls. Those numbers come from specific operations, shapes and hardware. They are not a promise that an existing cluster job becomes faster after a version bump. Measure end-to-end work completed per dollar and per hour. Include startup, compilation, communication, checkpointing, failed runs and recovery time. A kernel that wins a microbenchmark can still lose inside a job that hangs more often or uses more memory at initialization. Roll out by ring. Start with a replayable benchmark cluster. Move to one internal workload. Then one production team with an active reviewer. Expand only after the new backend survives a full job lifecycle, including a failure. At each ring, record: the PyTorch build and commit, the CUDA or ROCm wheel, the driver and NCCL or RCCL version, the topology and network, the process-group backend, the explicit device binding, initial and peak memory, collective and point-to-point behavior, flight-recorder delivery, checkpoint save and restore, rollback success, and unresolved differences. That record is more useful than a screenshot showing one faster step time. PyTorch 2.15 contains 3,117 commits from 583 contributors. It reaches from distributed communication to low-precision decoding, Apple Silicon, new AMD targets, Intel profiling and Python 3.15. That breadth is impressive. It is also why a production upgrade should not be reduced to pip install and optimism. The rewritten backend may be the better foundation. Defaults become foundations only after real workloads push on them. Run the canary. Force the hang. Read the recorder. Restore the checkpoint. Pull the rollback lever while it still exists. Then move the cluster, one proven ring at a time.