Re: [Opensc-devel] PKCS#11 Test suite (PIV)

[Jakub J. <jj...@re...>]

On 05/16/2016 07:20 PM, Douglas E Engert wrote:
> On 5/16/2016 10:04 AM, Jakub Jelen wrote:
>> Hello OpenSC devels,
>>
>> I didn't find any test suite or unit tests for OpenSC project. As I
>> noticed, there is a lot of hand-testing work on pull requests for
>> various cards and users. I believe everyone has some use cases to verify
>> basic functionality of their cards.
>> I understand that this fields is very divergent, there is a lot of card
>> variants and it is almost impossible to build automatic test suite that
>> would run in cloud with every build. But would it make sense to have
>> something that devels (or users) can simply run and what would verify
>> basic functionality and possible regressions?
> In-cloud testing would require the cloud test machine to have physical cards.
> (Unless you are suggesting some RDC access to cards.)
Yes, this would be awesome, but probably impossible to achieve. Rather 
having
something that can be simply run by many users with many different cards and
report general success or failure would be achievable.

Theoretically generating logs, collecting them from different cards and 
users and
representing them in readable form could be too useful for future 
codebase stability
among releases. But it is a bit over the initial idea.
> I would say the closest tool we have is: pkcs11-tool -t -l
> It does some basic tests, but as you may have noted if you try and run it
> with a PIV card, it has some problems, especially with the decryption, as it
> says the user is not logged in when trying to use the Sign key. The key usage
> says it should not be used for decryption. With other cards it may have different problems.
Thanks for mentioning pkcs11-tool test mode. I struggled upon it, but 
there were
some problems that prevented me to work with that. I will check what can 
be done there.
>> I went to the directory src/tests/ and fixed the tests that are
>> available now (see pull request [1], broken for 6 years), but they are
>> far away from complete test suite.
>>
>> I also started with the idea from PKCS#11 API and put together basic
>> test suite and inspector for OpenSC, which is currently in my OpenSC
>> fork [2]. It is by no mean complete test suite of all the use cases, but
>> I tried to catch most common cases, represent results in understandable
>> form (currently tested with PIV cards) and add regression test for
>> recent pull request [3].
> Good choice of card :-)
>
> Are you using the the NIST set of 16 demo/test cards?
Yes.

Regards,

-- 
Jakub Jelen
Security Technologies
Red Hat