When a glitch in a software update from security firm CrowdStrike unintentionally caused digital chaos around the world last month, the first sign was Windows computers displaying blue screens of death. Websites and services went down, and conflicting and inaccurate information abounded as people scrambled to understand what was happening. In the rush to understand the crisis, longtime Mac security researcher Patrick Wardle found there was one place he could turn to for the facts: crash reports from computers affected by the bug.
“I’m not a Windows researcher, but I was intrigued by what was going on, and there was a lack of information,” Wardle tells WIRED. “Everyone was saying it was a Microsoft issue because Windows systems were bluescreening, and there were a lot of wild theories, but it actually had nothing to do with Microsoft. So I looked at the crash reports, which to me are the ultimate truth. And that’s where I was able to pinpoint the root cause long before CrowdStrike announced it.”
Speaking at the Black Hat security conference in Las Vegas on Thursday, Wardle argued that crash reports are an underused tool. Such system snapshots give software developers and maintainers insight into potential problems in their code. And Wardle stressed that crash reports, in particular, can be a fountainhead of information about potentially exploitable vulnerabilities in software for both defenders and attackers.
During his talk, Wardle shared several examples of vulnerabilities he found in software when apps crashed, then combed through the reports for possible causes. Users can easily view their crash reports on Windows, macOS, and Linux, and they’re also available on Android and iOS, though they can be harder to access on mobile operating systems. Wardle noted that gaining insight from crash reports requires a basic understanding of instructions written in low-level machine code known as assembly, but stressed that the payoff is worth it.
During his Black Hat talk, Wardle presented multiple vulnerabilities he discovered simply by examining crash reports from his own devices, including bugs in the YARA analysis tool and in current versions of Apple’s macOS operating system. In fact, when Wardle discovered an iOS bug in 2018 that caused his app to crash every time it displayed the Taiwanese flag emoji, he used crash reports to figure out what was going on.
“We conclusively revealed that Apple had caved to China’s demands to censor the Taiwanese flag, but there was a bug in their censorship code. It’s ridiculous,” he says. “My friend who first discovered this was like, ‘My phone is being hacked by the Chinese. It crashes every time you text me. Or are you hacking me?’ And I said, ‘Excuse me, I don’t hack you. And excuse me, even if I did hack you, I wouldn’t crash your phone.’ So I got the crash report to see what was going on.”
Wardle emphasizes that if so many vulnerabilities can be found simply by looking at crash reports from your own and your friends’ devices, software developers need to look there too. Sophisticated criminals and well-funded nation-state backed hackers have probably already gotten ideas from crash reports. Over the years, news reports have indicated that intelligence agencies, such as the U.S. National Security Agency, mine crash logs. Wardle points out that crash reports are also a valuable source of information for detecting malware, as they can reveal anomalous and potentially suspicious activity. For example, notorious spyware broker NSO Group often builds mechanisms into its malware that delete crash reports as soon as it infects a device. And because malware is buggy and therefore more likely to crash, crash reports are also valuable to attackers in understanding what went wrong in the code.
“The truth is hidden in the accident report,” Wardle said. “No, I think the truth is hidden in there.”
