Interactive Application Security Testing (or as we recently argued it should be known, Instrumented Application Security Testing) works by adding hooks into a running application and analyzing its behaviour to look for security flaws. It has many advantages and it's the technique we use to test crypto security in our Analyzer tool.
Modern versions of IAST (like ours) can detect flaws even when the application is executing standard functional tests - there is no need to simulate attacks. This enables these tools to be deployed early in the development lifecycle and integrated into CI toolchains. However, there's one key feature that doesn't figure on most IAST checklists: coverage checking.
Suppose I get a nice green report from my IAST tool saying there are no vulnerabilities in my app - how do I know that all the parts of my code where there might be issues were actually exercised during testing?
This is important in particular for integration into CI: I want my AppSec tool to fail a build when a security-relevant mistake is made, so that it can fixed immediately and not cause a problem later. To be sure that will happen, I need to know that any new code that might affect security that gets added to the application also comes with test cases that exercise it
In conventional functional testing, code coverage is a mainstream concept. Measuring it can be tricky, but there are accepted trade-offs and a host of open-source tools for doing it. Measuring coverage for IAST is a bit different because we specifically want to know about coverage of security-relevant parts of the code. We don't necessarily want to repeat all the tests with the IAST tool enabled.
At Cryptosense we make an IAST tool for cryptography. Our IAST agents record a trace of all the cryptographic operations that the application executes in testing. This trace is then analyzed by our rulebase to spot flaws.
We have now added a static crypto scanning tool to our offer. The tool extracts all of the call sites for crypto from the bytecode, which we've found is already useful information. Once you have a trace from our agent, you can supply that to the scanning tool and find out which calls were actually exercised by the trace.
Sometimes there are crypto call sites in the code that you don't want to test. A good example is calls in the crypto libraries themselves, such as when a high level crypto function (like a PBKDF) calls a lower-level operation (like a hash function). You can exclude packages from the coverage check with a command-line parameter (we exclude the crypto libraries by default).
Code coverage is a complex topic, but being sure that all the ways your application uses crypto are tested at least once gives you good protection against false negatives and more assurance when you want to show that you're doing crypto right.