How does PyTorch handle memory fragmentation on GPU?

PyTorch uses a caching allocator to reduce fragmentation but fragmentation can still occur over time.

The caching allocator reuses freed memory blocks to avoid repeated allocations. However, varying tensor sizes and lifecycle patterns can lead to fragmentation, causing OOM errors even when memory appears available.