juniorPyTorch

What are tensors in PyTorch, and how are they different from NumPy arrays?

Updated Feb 20, 2026

Short answer

A tensor is PyTorch’s core data structure for storing and computing on multi-dimensional data such as images, text embeddings, and model weights. Tensors are similar to NumPy arrays, but they add features like GPU acceleration and automatic differentiation, which make them suitable for deep learning.

Deep explanation

A tensor is a general-purpose container for numerical data. It can represent a single number, a vector, a matrix, or data with many dimensions. In PyTorch, almost every operation in a neural network involves tensors: inputs, parameters, intermediate activations, and outputs are all tensors.

A tensor is defined by several properties:

  • Shape: The size of each dimension, such as (3, 224, 224) for a color image.
  • Data type: The kind of values stored, such as float32 or int64.
  • Device: Where the data lives, such as the CPU or a GPU.
  • Gradient tracking: Whether PyTorch should record operations to calculate gradients during training.

For example, a batch of images might be represented as a 4-dimensional tensor:

TypeScript
(batch_size, channels, height, width)

A batch containing 32 RGB images of size 224×224 would have shape:

TypeScript
(32, 3, 224, 224)

Why tensors exist in PyTorch

Deep learning requires repeatedly performing large mathematical operations:

  1. Store input data.
  2. Apply operations through neural network layers.
  3. Calculate an error value.
  4. Compute gradients.
  5. Update model parameters.

PyTorch tensors are designed around this workflow.

tensor_example.py
import torch


x = torch.tensor([[1.0, 2.0], [3.0, 4.0]])


weight = torch.tensor([[0.5], [0.5]], requires_grad=True)


output = x @ weight
loss = output.sum()


loss.backward()


print(weight.grad)

Here, requires_grad=True tells PyTorch to track operations involving weight. After backward() runs, PyTorch has calculated the gradients needed to update the model during training.

The relationship looks like this:

Rendering diagram…

Tensors vs NumPy arrays

NumPy arrays and PyTorch tensors are closely related. Both store numerical data efficiently and support vectorized operations. However, tensors include deep-learning-specific capabilities.

FeatureNumPy arrayPyTorch tensor
Main purposeScientific computingMachine learning and deep learning
GPU supportLimited through external toolsBuilt-in device support with cuda
Automatic gradientsNoYes with autograd
Neural network integrationRequires extra librariesWorks directly with torch.nn

The biggest difference is that PyTorch tensors are built to participate in the training process, not just store and manipulate numbers.

GPU acceleration

A major advantage of tensors is that they can move computation from the CPU to a GPU.

move_tensor.py
import torch


x = torch.randn(1000, 1000)


if torch.cuda.is_available():
    x = x.to("cuda")

Once moved to a GPU, supported tensor operations can run much faster because GPUs are optimized for parallel numerical calculations.

A NumPy array usually lives in system memory and does not automatically move to a GPU. Developers often use libraries such as CuPy or specialized frameworks when they need GPU-style array computing.

Relationship between tensors and automatic differentiation

During training, a neural network needs gradients. The gradient tells the optimizer how much each parameter contributed to the error.

PyTorch creates a computation graph while tensor operations are performed:

  • Each operation becomes part of the graph.
  • The graph records how values were produced.
  • backward() traverses the graph in reverse.
  • Gradients are calculated using the chain rule.

A useful interview phrase: tensors are like NumPy arrays with a built-in engine for deep learning computation and gradient tracking.

⚠️ Rule of thumb: If you only need numerical processing, NumPy may be enough. If you are training neural networks, tensors are usually the right abstraction.

Real-world example

Imagine training an image classifier that identifies cats and dogs. The images are loaded as tensors, passed through a neural network, and compared against the correct labels.

A simplified training step looks like this:

training_step.py
import torch
import torch.nn as nn


images = torch.randn(16, 3, 224, 224)
labels = torch.tensor([0, 1, 0, 1])


model = nn.Linear(3 * 224 * 224, 2)


predictions = model(images.view(16, -1))
loss = nn.CrossEntropyLoss()(predictions, labels)


loss.backward()

The input images, model weights, predictions, and loss values are all tensors. Because the tensors track operations, PyTorch can automatically calculate how the model weights should change.

In production machine learning systems, tensors are the bridge between raw data and trainable neural network models.

Common mistakes

  • * **Confusing arrays** - Assuming tensors are completely different from `NumPy` arrays. They share many behaviors, but tensors add machine-learning features.
  • * **Ignoring devices** - Moving a model to a GPU but leaving input tensors on the CPU causes device mismatch errors. Keep related tensors on the same device.
  • * **Skipping gradient settings** - Forgetting when to disable gradients can waste memory during inference. Use `torch.no_grad()` when training is not happening.
  • * **Misunderstanding dimensions** - Treating tensor shapes casually leads to errors in neural networks. Always check shapes before connecting layers.
  • * **Using GPUs automatically** - A tensor does not become faster just because it exists. It must be explicitly moved to a supported GPU device.

Follow-up questions

  • How does PyTorch calculate gradients for tensors?
  • What is the difference between torch.Tensor and numpy.ndarray?
  • How do you move a tensor from CPU memory to GPU memory?
  • What do tensor dimensions represent in a neural network?
  • Why are GPUs faster for tensor operations?

More PyTorch interview questions

View all →