What is tokenization in NLP?

Tokenization is the process of breaking text into smaller units like words, subwords, or sentences.

Tokenization is a fundamental preprocessing step in NLP where raw text is split into meaningful units called tokens. These tokens are then used for downstream tasks such as classification, translation, and sentiment analysis. Different tokenization strategies include whitespace-based, rule-based, and subword tokenization (like BPE or WordPiece used in transformers).