Towards regression testing and regression-free update for deep learning systems

Recently, deep neural networks have become prevalent in various systems, such as autonomous driving, privacy leakage detection, and sensitive data protection, and naturally raise wide concerns about their reliability. Current evaluation of the behaviors of DNN models is focused on their overall performance in a statistical way, e.g., measured by accuracy. However, the regression problem on model performance is also an important issue, especially in real-world applications. If a “new and improved” model exhibits errors absent in the old one, frustrated users may abandon the product. Moreover, the reliability of the model could be hard to maintain in the long term. Given its severity and importance, we aimed to detect and fix the regressions on DNN models without affecting the overall performance, and we made a preliminary study on two common situations where regressions occur, i.e., randomness and data evolution. Specifically, we formulated it into a constraint optimization problem by taking the regression-free conditions as constraints and approximating with the combination of two proxies. First, we suppressed the regressions by making the new model mimic the original model and design a quadratic penalty during model training. Second, given the trade-off between similarity to the old model and eligible performance of the new model, we designed a novel biased neuron response variability technique to suppress regression without performance degradation. To evaluate the effectiveness of our technique, we experimented on MINIST, CIFAR-10,and Fashion-MNIST datasets. The results show that our method can reduce the regression rate from 1.53% to 0.69% on average for the random seed change situation, and it was also effective for the data evolution situation.