Evaluation of Machine Unlearning Through Model Difference

Increasing attention is being paid to machine unlearning, which supports individuals’ “right to be forgotten.” While most studies focus on the efficiency and effectiveness of unlearning algorithms, the evaluation of machine unlearning effectiveness remains underexplored. Offering robust evaluation services for unlearning is critical, not only to uphold privacy legislation but also to assess and improve existing unlearning methods. Lots of existing methods employ backdoor methods to evaluate unlearning effectiveness, which can only verify the unlearning effect of backdoored samples and negatively impact the model utility as they need to embed backdoors into the model first. In this paper, we propose an evaluating machine unlearning (EMU) method, which aims to evaluate the effectiveness of unlearning and verify data removal without the aforementioned adverse effects. Machine unlearning inherently creates a difference on the model before and after unlearning. The model difference contains information about the unlearned samples, which can be extracted through reconstruction models for unlearning effectiveness evaluation. To efficiently generate the model differences as input for evaluation, we simulate the model changes based on the influence function theory. Additionally, we design a multi-task information bottleneck structure to enhance the scalability of EMU and simplify the analysis of different learning tasks. We provide a theoretical analysis of how the similarity between erased and remaining samples, as well as task types, affects the extent of unlearning—factors that have been largely overlooked. Extensive experiments on various model architectures and representative datasets confirm our analysis, demonstrating the effective evaluation for unlearning without any degradation in the service model utility.