Some standards come with compliance criteria built in – you can’t say you’ve implemented the standard until your code can pass the tests. With PKCS#11, a 407-page standard specifying the most widely used API in cryptographic hardware, there are no such tests. So how can a would-be PKCS#11 user discriminate between a good implementation of the API and a bad one? And how can a manufacturer find compliance bugs and then demonstrate the quality of their product?
At Cryptosense we’ve been thinking about this problem for some time. It’s easy to see why PKCS#11 does not contain compliance tests: it’s a large general-purpose cryptographic standard, and each implementation will typically support a different subset of commands and mechanisms, and implement them in a different way. Since you don’t know what’s implemented, you can’t just give a list of test cases that have to produce certain outputs. Interoperability is nice, but it doesn’t say anything about security, i.e. what command calls should
Introducing Cryptosense PKCS#11 Compliance Testing
The Cryptosense PKCS#11 Compliance Tester solves this problem. The tool works in two stages. In the first step, we run our smart fuzzing algorithm on the interface. Our algorithm adapts to the functions available, generating test cases on the fly. In the second step, we run the results through our PKCS#11 compliance filter and produce a report (see the screenshots below).
Smart fuzzing in progress
Sample of compliance checking results – an interesting failure case.
You’ll notice that for each compliance criterion, the tool gives the exact reference in the spec where the behaviour is specified, for easy clarification. We’ve extracted over a hundred criteria, many of which apply to multiple functions, giving over 1000 compliance tests. For each test, we show how many calls passed and failed. The tool can also produce a list of the calls which failed the tests, to aid debugging. Finally, just in case the PKCS#11 driver crashes, the tool optionally writes all calls into a log before making them, so you can easily track down bugs.
You now have 15 seconds to comply
Compliance ≠ Security
A compliant PKCS#11 implementation avoids a number of security vulnerabilities such as CVE 2010 3321, but it’s only a first step to a secure deployment. Security depends also on the configuration of the interface, and the calls made by the application.