Aug 26, 2026 · 13 min readRe-running a Vulnerability Detection Benchmark, More Carefully
I reproduced Devign (NeurIPS 2019) from scratch on the authors' own data. It lands 11 points below the paper, two-thirds of its test set leaks through shared commits, and a plain CNN beats it.
Read post