What is hierarchical token merging (ToMe) in Vision Transformers?

Token Merging (ToMe) reduces ViT computation by merging similar tokens during inference or training.

Token Merging (ToMe) identifies similar tokens using similarity metrics (often cosine similarity) and merges them into a single representative token. This reduces sequence length dynamically while preserving information. It significantly reduces the O(N²) cost of attention while maintaining accuracy, especially in redundant regions like backgrounds.