What is bilevel optimization in advanced cost function systems?

Bilevel optimization involves optimizing one objective nested inside another.

Bilevel optimization appears in hyperparameter tuning, meta-learning, and RLHF. The inner loop trains a model, while the outer loop optimizes hyperparameters or reward alignment. This creates a nested dependency where gradients must be propagated through an optimization process itself, making computation expensive and unstable.