Re: [Openca-ocspd] Will putting CRLs into LDAP server reduce OCSPD memory usage?

[Dr. P. <ma...@op...>]

Hi Jim,

sorry it took me so much time to reply - I admit I missed your message 
:D For adding a support for DB lookup instead of Memory lookup, the 
effort should be not that much. The way I would architect the project 
would be to add support for the specific DB (in LibPKI Pg and MySQL are 
already supported in the URL interface, however I would not suggest to 
query the database through that interface as it is not optimized for 
speed :D), then I would:

  * Upon CRL loading, I would add a function that takes the single
    entries and put them in the database (and maybe remove the entries
    that are expired - sort of cleanup) and then remove the CRL from
    memory (freeing the memory)
  * Repeat for all the CRLs you want to serve

This might work but you have to be sure you are not updating the DB from 
different instances of the sevice (i.e., there must be a master that 
updates the DB while all the other instances only perform lookup 
operations in the DB).

Possibly, the process that updates the DB could be a separate one (not 
necessarily the OCSPD), while the OCSPD would only perform queries to 
build the responses.

Does this make sense ? Were you thinking about a different type of 
architecture ?

Cheers,
Max


On 7/11/19 9:47 AM, oh...@ya... wrote:
> Hi,
>
> Sorry that it's been awhile since your message below.
>
> We *MAY* be able to provide support for adding the DB option, but, if 
> it is not too much trouble, can you give a high-level description of 
> what would be done?  I am still pushing for it here, but I am not 100% 
> sure yet.
>
> Also, would the modifications be in Java?  Or in C/C++?
>
> Thanks,
> Jim
>
>
>
> On Sunday, May 26, 2019, 11:30:31 PM UTC, Dr. Pala 
> <ma...@op...> wrote:
>
>
> Hi Jim,
>
> interesting issue - something I have not considered (memory usage). 
> Unfortunately, the memory usage will not be different because, in the 
> end, after one of the supported/specific transport protocol is used 
> (i.e., LDAP, HTTP, HTTPS, MySQL, PgSQL, etc.) the CRL is then parsed 
> and entries are created based on the entries in the CRL.
>
> One solution to your problem, maybe, would be to sacrifice the speed 
> of response for the possibility to instantiate OCSPDs in small 
> containters/VMs. My assumption is that disk-space is not an issue.
>
> We could think about modifying the OCSPD to work off a small DB4 or 
> similar... this would allow us to create entries in the DB (filesystem 
> entries instead of memory entries). This might require additional time 
> at CRL reload, but the memory usage would be practically constant with 
> the growing of CRLs.
>
> We have been working recently to make sure LibPKI works well with 
> OpenSSL 1.1.+ branches (and add support for CMS in the same context), 
> and the next update I was thinking was related to two different things:
>
>   * Adding a simple Cache mechanism so that the same answer can be
>     cached for its validity period (or up to 80% of its validity
>     period). This would improve the speed for high-frequency
>     certificates' checks
>   * Adding support for OCSPv2 (and DNS distribution end-points). This
>     is something we have been working for a while and would like to
>     standardize at some point. The main idea is to provide responses
>     for "ranges" of certificates instead of for each of the issued
>     certificates. This would provide smaller overhead when large
>     chunks of the issued certs space are not revoked. For revoked
>     entries, there is no much optimization we can do since you need
>     the full revocation data there.
>
> I guess we can add the low-memory architecture as another enhancement, 
> but I am not sure I have the bandwidth right now to work on that - 
> would you like to work on that together and provide some initial 
> thoughts / code ?
>
> Cheers,
> Max
>
> On 5/24/19 1:00 PM, o haya via Openca-ocspd wrote:
> Hi,
>
> I had asked this question awhile ago, but no one responded, so I would 
> like to ask again.
>
> We currently use OCSPD with multiple CRLs (from different CAs).  Some 
> of these CRLs are quite large, and so the OCSPD memory usage is quite 
> large.
>
> However, we have some scenarios where we have machines with only much 
> smaller memory, but we would still like to use OCSPD and with the same 
> CRLs.
>
> All of our machines include an LDAP server, and from the docs, it 
> seems like OCSPD can work with CRLs in LDAP servers, so we were 
> wondering if we made our own tool to import the CRL contents into the 
> LDAP server, and configured OCSPD to use the CRLs in the LDAP server, 
> do you think/know if the overall memory usage would be reduced, 
> compared to our current configuration?
>
> I understand that the memory usage of the LDAP server, when populated 
> with the CRLs, MIGHT be the same, and if so, we'd end up in a 
> "zero-sum" situation, but I wanted to check what you all thought?  Has 
> anyone used OCSPD with the CRL information in an LDAP?
>
> Thanks,
> Jim
>
>
> _______________________________________________
> Openca-ocspd mailing list
> Ope...@li...  <mailto:Ope...@li...>
> https://lists.sourceforge.net/lists/listinfo/openca-ocspd
> -- 
> Best Regards,
> Massimiliano Pala, Ph.D.
> OpenCA Labs Director
> OpenCA Logo
> _______________________________________________
> Openca-ocspd mailing list
> Ope...@li... 
> <mailto:Ope...@li...>
> https://lists.sourceforge.net/lists/listinfo/openca-ocspd
-- 
Best Regards,
Massimiliano Pala, Ph.D.
OpenCA Labs Director
OpenCA Logo