AdamW

Why embedding layer LR dominates hyperparameter transfer

The paper shows that embedding-layer learning rate is the main reason μP transfers better than standard parameterization.