Including passwords or cryptographic key material in source code is a major security risk for a number of reasons. In the worst case, if the code is public, everyone can read the key. Even if not, access to the code is often easier for an attacker to achieve than direct compromise of the application - the entire development team becomes part of the attack surface. Having obtained the keys, the attacker may no longer need to compromise the application at all, and the breach can go completely undetected since there is nothing in the logs when encrypted data is decrypted offline.
Hardcoding the keys is also a problem for key rollover, and for cryptographic agility. So, we're convinced we need to get rid of them, but how can we check for them at scale across hundreds or thousands of applications?
Limitations of pure SAST
Static analysis security testing tools (SAST) typically include some capability for rooting out such problems. The problem is precision: first, false positives. Source code might contain tests using fixed keys that need to be excluded, or code for verifying the availability of certain crypto algorithms and key-lengths in the system crypto libraries.Second, false negatives: strings might be encoded in various ways in the source code, either to obfuscate the key or just by accident. Values also get passed between variables, and static tools don't track this well.
Combining SAST and IAST results
Our Analyzer tool uses a combined SAST and IAST approach: we first scan the bytecode for possible hardcoded credentials, and then during normal execution, our tracer agent checks all keys and credentials to see if they match a transformation of one of these values. This improves precision considerably, and Analyzer can report both where the credential appears in the code and where it was used.
Comparing Analyzer to SAST
To test this we ran our tool over two applications and compared the results with two static analysis tools, the free and open-source SpotBugs (with the module for hardcoded key detection enabled) and a widely-used commercial static analysis tool that figures regularly in the top recommendations of industry analysts. The first application contains a showcase of crypto mistakes including hard-coded credentials of various different kinds. The other was the Jenkins CI tool, specifically the version we analyzed back in February 2018 (the hardcoded credential issue has been reported and fixed since then).The results are summarized in the table below, giving the number of true positives (TPs) and false positives (FPs) found by each tool. The short story is that our combined approach out-performs both the commercial SAST tool and SpotBugs by some distance. In particular, the commercial tool seems to have only very basic capabilities for detecting these issues, and seems to be triggered by things like a variable name that matches the string "password" rather than a data-flow analysis on keys and strings. SpotBugs found more of the credentials, but also missed important ones like the use of a harcoded string "password" to protect the private TLS key in Jenkins.
ApplicationCryptosense AnalyzerSpotBugsCommercial SASTTPFPTPFPTPFPJenkins500000Showcase908011
We're happy to share privately more details of our tests.