weight decay

Fixing LLM forgetting in ES fine-tuning

This paper shows LLM fine-tuning with evolution strategies can drift, and anchored weight decay can curb it.