Security viewpoints on explainable machine learning

In the swiftly evolving research field of machine learning, there is a growing demand for ensuring both the interpretability and security of algorithms. Recently, a variety of approaches to explain how models arrive at their predictions have been developed and constitute a core component towards enabling trustworthy machine learning. In this thesis, we adopt the viewpoints of a developer, an operator and an adversary when investigating explanations and derive research questions that emerge from each perspective. From the perspective of a developer leveraging machine learning for security tasks such as malware and vulnerability detection, we elucidate the question of selecting an optimal explainable learning algorithm. Recognizing the need for a coherent measure of comparison, we introduce a set of multiple evaluation metrics to contrast explanations. These metrics are employed on four security tasks spanning diverse neural network architectures. Furthermore, we present an algorithm to extract meaningful explanations, offering a compact lens into the model and dataset for further investigation. During the analysis of these prototypical explanations, we unveil a pressing concern that many existing models do not necessarily address the issues they were initially trained for. Next, we transition to the viewpoint of an operator of a machine learning service and show how explanations can reveal privacy leaks in machine learning models. Our findings underline the need for strategies for data adjustments without significant disruption to the model parameters, especially in the light of legislature like the General Data Protection Regulation (GDPR). In this endeavor, we introduce the concept of certifiedunlearning for features and labels with a framework based on approximate parameter updates. Conditions that guarantee certified unlearning arederived and tested across a variety of datasets and tasks. Lastly, adopting the vantage point of a potential adversary, we present innovative attack mechanisms exploiting model weaknesses unveiled by explanation techniques. Firstly, we introduce dormant minimal backdoors, a novel approach to embed stealthy backdoors bypassing standard detection paradigms by manipulating hardware accelerators. Secondly, using the similarity of edge detection and explanations for image classifiers, we present the concept of model-independent adversarial examples. These adversarial examples sidestep the need for access to model parameters or black-box queries, confining their operation exclusively to the input presented to the model. Collectively, this work enlightens about chances and challenges when adapting explanations into machine learning based systems. On the one hand, explanations can effectively support practitioners in evaluating and debugging datasets and models. On the other hand, problems like information leakage and attack detection must be addressed and understood further to pave the way to trustworthy and robust learning systems.