What is tokenization in LLMs?

Tokenization is the process of converting text into smaller units (tokens) that LLMs can process.

LLMs do not process raw text directly. Instead, text is split into tokens such as subwords or characters using tokenizers like BPE. Each token maps to an integer ID. The model processes sequences of token IDs to generate predictions.