Re: [Ejbca-develop] EJBCA OCSP Responder does not appear to

[Tomas G. <to...@pr...>]

Hi,

I just finished importing your CRL.
996549 records in approx. 12 hours.

This is on my laptop, starting with 400K records in the database already
(i.e ending with 1408370). There is no slowdown as number of records grow).

It failed in the end storing the actual CRL due to a limitation on
packet size in my MySQL config, but that could be changed easily in
MySQL (MariaDB) config. Actually the CRL itself is not needed in the
database in order to answer OCSP queries, so there should optimally be a
switch to not import the CRL itself, only the revocation records (to
save a bit on database storage).

The import speed can with some rather simple development be made orders
of magnitude faster (you don't want a roundtrip for every record for
example).

Cheers,
Tomas


On 2019-07-19 18:33, oh...@ya... wrote:
> Started testing with the VM on an SSD and now I am getting ~500 - 550
> per minute, or about 30K per hour.  That is faster than before, but
> still way slower than the 100K per hour you mentioned.
> 
> Jim
> 
> On Friday, July 19, 2019, 2:56:11 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
> 
> 
> Hi,
> 
> I don't have any SSDs on the machine that I use as my server... only
> 7200RPM hard drives, except for the OS drive, which is SSD.  I can try
> to add an SSD (I have lots of SSDs, just not on that machine).
> 
> Jim
> 
> On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
> 
> 
> 
> I have now imported 600K of the entries form the CRL. Still running at
> 25 entries per second.
> 
> The logging does not normally affect performance, logging is a normal
> part. Not running an SSD for the database typically hugely affect DB
> performance. It's possible to tune logging a lot, For example disable
> logging to console and only log to the file (if your console is really
> slow). It's also possible to set the database to not synchronize on disk
> for every write, during database restore and certain operations for example.
> 
> I actually have a worse performing log config than you. I have info log
> to the console and DEBUG log to file, so it's logging a lot. I have a
> SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
> top). But it's still a three year old laptop, so not the fastest you can
> get.
> 
> Regards,
> Tomas
> 
> On 2019-07-19 15:36, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> I got the index done and now am running the CRL import, which has been
>> running about 1/2 hour now, but so far, only about 5500 entries
>> imported, so it doesn't look like it hugely faster than before?
>>
>> I was wondering... on the server side, I think outputting messages like
>> the following to stdout for every entry in the CRL:
>>
>> 09:32:37,008 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
>> 09:32:37,025 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,035 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
>> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
>> 'CN=XXXXXX', serialNo=8A296.
>> 09:32:37,079 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,115 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-2) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,133 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-2) 2019-07-19
>>
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>>
>>
>> Is it possible that that logging is slowing the CRL import?
>>
>> How is your logging setup (log level) compared to what mine is?
>>
>> Other that, it just may be that my system is much slower than your
>> laptop :)??
>>
>> Jim
>>
>>
>>
>>
>> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Ah, eap 7.2 corresponds to wildly 14 as base.
>>
>> Yes, run that to convert, then you should be able to create the index.
>>
>> Cheers,
>> Tomas
>>
>>
>> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>    Hi Tomas,
>>
>>    No, I am not using Wildfly, but rather JBOSS... specifically
>>    "jboss-eap-7.2".
>>
>>    So I restored my snapshot from just after I had deleted all the rows
>>    in CertificateData that were created before.
>>
>>    And it looks like all the tables in the "ejbca" database are MyISAM.
>>
>>    So I found a one-liner that will convert them all to InnoDB:
>>
>>    mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
>>    | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}'  |  mysql
>>    -u root -p ejbca
>>
>>    Should I run that and convert all the tables in the "ejbca" to
> InnoDB???
>>
>>
>>    And then try to run the create indexes SQL script?
>>
>>
>>    Thanks for all your help!
>>
>>    Jim
>>
>>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop