Exact Extract — Huawei Training Material, Pages 57 and 69
"Large scale does not mean strong computing power, and building large computing clusters faces many challenges"
"Cluster availability"
"Frequent failures of core components"
"Difficult cross-domain fault handling"
"Slow recovery of training tasks"
"Cross-domain fault detection and diagnosis for compute, network, and storage"
"High throughput checkpoint access and storage"
"System high availability = as long as possible failure-free running time + as short as possible fault recovery time"
The resumable-training architecture also identifies fault detection, task rescheduling, checkpoint loading, collective communication initialization, and training resumption as essential recovery activities.
Technical Deep Dive:
Large-scale AI training environments contain numerous interconnected components, including Ascend NPUs, Kunpeng CPUs, server memory, network switches, optical transceivers, and distributed storage systems.
Training workloads frequently use synchronous distributed computation, making application performance and availability dependent on the health of multiple devices.
Option A is correct. A single critical hardware failure can interrupt a distributed training job when the affected device participates in required computation or synchronization.
Fault-tolerant training mechanisms can reduce the impact, but uninterrupted execution is not guaranteed.
Option B is correct. Distributed training performance can be constrained by a slow or degraded node.
When synchronization is required, faster nodes may wait for slower participants before continuing to the next training step.
This behavior is commonly known as the straggler effect .
Option C is correct. Trillion-parameter models require large-scale distributed computing infrastructure containing many interdependent hardware and software components.
The resulting complexity creates substantial requirements for fault monitoring, root-cause analysis, and operational coordination.
Option D is incorrect. Failure of a single accelerator card can affect the entire training workload, especially when the device holds a required portion of the model or participates in collective communication.
Huawei addresses these challenges through CCAE cluster management, checkpoint recovery, hardware fault prediction, and automated training resumption.
Therefore, A, B, and C are correct.
Submit