How do transformer models internally represent syntactic structure without explicit grammar rules?

Transformers implicitly learn syntax through attention patterns and learned positional relationships.

Although transformers are not explicitly trained on grammar rules, attention heads specialize in syntactic relations such as subject-verb agreement, dependency parsing, and coreference. Probing studies show that intermediate layers capture syntactic trees implicitly. This emerges due to prediction-based training objectives over large corpora.