Abstract
As cyber-threats grow in scale and sophistication, intrusion detection systems that incorporate system provenance and deep learning have emerged as a promising direction for detecting advanced persistent threats (APTs). We endeavor to reproduce the experimental results from eight such systems published over the past four years in top-tier research venues. We encountered numerous challenges that obstruct reproducibility, including incomplete or non-functional source code releases, missing documentation, unavailability of datasets or detailed preprocessing steps, and unclear or inconsistent descriptions of experimental procedures. We detail and categorize these challenges to demonstrate the obstacles researchers may encounter when reproducing studies in this domain. Our findings highlight gaps in reaching the ideals of open science in this area of intrusion detection research.