Security BSides London, the UK’s biggest community-driven infosec conference is happy to announce its 8th iteration open to all regardless of background, skill level, income or job-title.  
  • Doors to the main event open at 8.30am with talks starting at 9am on 5 June 2019
  • Workshops will be held on 4 June 2018 starting at 10am; Pre-registration is required
Please remember that Security BSides London has a strict code of conduct available here
Back To Schedule
Wednesday, June 5 • 11:30 - 12:30
When the Magic Wears Off: Flaws in ML for Security Evaluations (and What to Do about It)

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Academic research on machine learning-based malware classification appears to leave very little room for improvement, boasting F1 performance figures of up to 0.99. Is the problem solved? In this talk, we argue that there is an endemic issue of inflated results due to two pervasive sources of experimental bias: spatial bias, caused by distributions of training and testing data not representative of a real-world deployment, and temporal bias, caused by incorrect splits of training and testing sets (e.g., in cross-validation) leading to impossible configurations. To overcome this issue, we propose a set of space and time constraints for experiment design. Furthermore, we introduce a new metric that summarizes the performance of a classifier over time, i.e., its expected robustness in a real-world setting. Finally, we present an algorithm to tune the performance of a given classifier. We have implemented our solutions in TESSERACT, an open source evaluation framework that allows a fair comparison of malware classifiers in a realistic setting. We used TESSERACT to evaluate two well-known malware classifiers from the literature on a dataset of 129K applications, demonstrating the distortion of results due to experimental bias and showcasing significant improvements from tuning.

avatar for Lorenzo Cavallaro

Lorenzo Cavallaro

King's College London
Lorenzo grew up on pizza, spaghetti, and Phrack, first. Underground and academic research interests followed shortly thereafter. He holds a PhD in Computer Science from the University of Milan (2008), held Post-Doctoral and Visiting Scholar positions at Vrije Universiteit Amsterdam... Read More →

Wednesday June 5, 2019 11:30 - 12:30 BST
Track 2 ILEC Conference Centre 47 Lillie Road London SW6 1UD