What is Tokenization in NLP and why is it a fundamental step in Deep Learning pipelines?

Tokenization is the process of converting raw text into smaller units (tokens) that neural networks can process.

Neural networks cannot directly process raw text; it must first be converted into numerical representations through tokenization.

Core idea:

Types of tokenization:

Methods: