What is tensor parallelism in large model training?

Tensor parallelism splits individual tensors across multiple GPUs.

Instead of splitting layers, tensor parallelism partitions matrix multiplications across devices, enabling training of extremely large layers by distributing computation within a single operation.