How does PyTorch handle sparse gradients?

Sparse gradients only store non-zero gradient entries to save memory.

Used in embedding layers, sparse gradients reduce memory and computation by updating only selected indices. Not all optimizers support sparse gradients, limiting applicability.