Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

Imagine you're developing a large language model that needs to forget certain sensitive information without losing its overall performance. Currently, when models are trained, they can inadvertently memorize and reproduce undesirable content, which raises privacy and safety concerns. The common solution is to retrain the model, but this can be inefficient and may not effectively erase the unwanted knowledge, especially when the model undergoes changes like quantization. This is what's called the 'forgetting-utility trade-off', where the model's ability to forget conflicts with its performance on tasks it should still handle well.

To address this, a new approach called Forgetting Only What Matters via Unlearning Layers (FOM-UL) is proposed. Instead of applying broad updates across the entire model, FOM-UL selectively targets specific transformer layers based on their significance to the information being forgotten. This means that only the layers that are crucial for the unwanted content are updated, while the rest of the model remains intact. As a result, FOM-UL not only improves the effectiveness of forgetting but also maintains the model's utility, even under conditions like post-training quantization. Compared to existing methods, FOM-UL shows better performance in suppressing memorization while preserving the model's ability to perform on retained tasks.

FOM-UL introduces a targeted layer-level unlearning approach that improves upon existing methods.

The method is evaluated against multiple strong baselines, showing consistent improvements.