Re: [Ejbca-develop] EJBCA OCSP Responder does not appear to

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

 AACK!  Yes, I forgot all about that and just used the vanilla software :(!  Now, if I can remember how to do that patch, I will try it :(...
Thanks,Jim

    On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson <to...@pr...> wrote:  

Did you forget to patch the java file? The top output suggest you did.
Attached the latest patched file that I used for the import.

Regards,
Tomas

On 2019-07-20 17:45, oh...@ya... wrote:
> Hi,
> 
> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
> just started testing with that same CRL, and unfortunately, it is not
> doing very well... actually even slower than on my earlier dev machine
> under VBox :(!!
> 
> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
> CPUs.  I converted the tables to InnoDB and also ran the create index
> SQL script.
> 
> It looks like the import is only processing about 5000 entries per hour.
> 
> Here's the top:
> 
> top - 15:43:29 up 17:25,  7 users,  load average: 1.70, 1.60, 0.98
> Tasks: 127 total,   1 running, 126 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 75.4 us,  0.5 sy,  0.0 ni, 23.2 id,  0.5 wa,  0.0 hi,  0.2 si, 
> 0.2 st
> KiB Mem :  3880256 total,   103696 free,  2799736 used,   976824 buff/cache
> KiB Swap:        0 total,        0 free,        0 used.   788552 avail Mem
> 
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 12115 root      20   0 3529260 1.074g  15368 S 143.0 29.0  18:35.88 java
>  7326 root      20   0 3290332 1.038g  21880 S   7.7 28.1   4:02.67 java
>  1384 mysql     20   0 1580240 231652   8260 S   3.0  6.0   0:41.18 mysqld
>   264 root      20   0       0      0      0 S   0.3  0.0   0:12.29
> xfsaild/xvda2
>     1 root      20   0  125572   3952   2572 S   0.0  0.1   0:03.04 systemd
>     2 root      20   0       0      0      0 S   0.0  0.0   0:00.02 kthreadd
>     3 root      20   0       0      0      0 S   0.0  0.0   0:00.03
> ksoftirqd/0
>     5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00
> kworker/0:0H
> 
> 
> Tomas, you had mentioned that there were some optimizations for MySQL? 
> Could those be the reason that your import is running so much faster
> than when I run the import?
> 
> Thanks,
> Jim
> 
> 
> 
> On Friday, July 19, 2019, 8:18:13 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
> 
> 
> I'm trying to stand up a new EJBCA instance on AWS, to try to see if I
> can replicate the import speed you are seeing, but ran into an error
> (see the new thread I just posted).
> 
> Thanks,
> Jim
> 
> On Friday, July 19, 2019, 6:39:52 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
> 
> 
> Hi,
> 
> I just finished importing your CRL.
> 996549 records in approx. 12 hours.
> 
> This is on my laptop, starting with 400K records in the database already
> (i.e ending with 1408370). There is no slowdown as number of records grow).
> 
> It failed in the end storing the actual CRL due to a limitation on
> packet size in my MySQL config, but that could be changed easily in
> MySQL (MariaDB) config. Actually the CRL itself is not needed in the
> database in order to answer OCSP queries, so there should optimally be a
> switch to not import the CRL itself, only the revocation records (to
> save a bit on database storage).
> 
> The import speed can with some rather simple development be made orders
> of magnitude faster (you don't want a roundtrip for every record for
> example).
> 
> Cheers,
> Tomas
> 
> 
> On 2019-07-19 18:33, oh...@ya... <mailto:oh...@ya...> wrote:
>> Started testing with the VM on an SSD and now I am getting ~500 - 550
>> per minute, or about 30K per hour.  That is faster than before, but
>> still way slower than the 100K per hour you mentioned.
>>
>> Jim
>>
>> On Friday, July 19, 2019, 2:56:11 PM UTC, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> I don't have any SSDs on the machine that I use as my server... only
>> 7200RPM hard drives, except for the OS drive, which is SSD.  I can try
>> to add an SSD (I have lots of SSDs, just not on that machine).
>>
>> Jim
>>
>> On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>>
>> I have now imported 600K of the entries form the CRL. Still running at
>> 25 entries per second.
>>
>> The logging does not normally affect performance, logging is a normal
>> part. Not running an SSD for the database typically hugely affect DB
>> performance. It's possible to tune logging a lot, For example disable
>> logging to console and only log to the file (if your console is really
>> slow). It's also possible to set the database to not synchronize on disk
>> for every write, during database restore and certain operations for
> example.
>>
>> I actually have a worse performing log config than you. I have info log
>> to the console and DEBUG log to file, so it's logging a lot. I have a
>> SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
>> top). But it's still a three year old laptop, so not the fastest you can
>> get.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-19 15:36, oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>> wrote:
>>> Hi,
>>>
>>> I got the index done and now am running the CRL import, which has been
>>> running about 1/2 hour now, but so far, only about 5500 entries
>>> imported, so it doesn't look like it hugely faster than before?
>>>
>>> I was wondering... on the server side, I think outputting messages like
>>> the following to stdout for every entry in the CRL:
>>>
>>> 09:32:37,008 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
>>> 09:32:37,025 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>>> 09:32:37,035 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
>>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
>>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>>> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
>>> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
>>> 'CN=XXXXXX', serialNo=8A296.
>>> 09:32:37,079 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>>> 09:32:37,115 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-2) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>>> 09:32:37,133 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-2) 2019-07-19
>>>
>>
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
>>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
>>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>>> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>>>
>>>
>>> Is it possible that that logging is slowing the CRL import?
>>>
>>> How is your logging setup (log level) compared to what mine is?
>>>
>>> Other that, it just may be that my system is much slower than your
>>> laptop :)??
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
>>> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
>>>
>>>
>>> Ah, eap 7.2 corresponds to wildly 14 as base.
>>>
>>> Yes, run that to convert, then you should be able to create the index.
>>>
>>> Cheers,
>>> Tomas
>>>
>>>
>>> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
>>> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>>>
>>>    Hi Tomas,
>>>
>>>    No, I am not using Wildfly, but rather JBOSS... specifically
>>>    "jboss-eap-7.2".
>>>
>>>    So I restored my snapshot from just after I had deleted all the rows
>>>    in CertificateData that were created before.
>>>
>>>    And it looks like all the tables in the "ejbca" database are MyISAM.
>>>
>>>    So I found a one-liner that will convert them all to InnoDB:
>>>
>>>    mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
>>>    | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}'  |  mysql
>>>    -u root -p ejbca
>>>
>>>    Should I run that and convert all the tables in the "ejbca" to
>> InnoDB???
>>>
>>>
>>>    And then try to run the create indexes SQL script?
>>>
>>>
>>>    Thanks for all your help!
>>>
>>>    Jim
>>>
>>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
> <mailto:Ejb...@li...>
>> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> 
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop