Menu
▾
▴
ejbca-develop — Development discussions
You can subscribe to this list here.
| 2001 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
(3) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2002 |
Jan
(3) |
Feb
(2) |
Mar
(8) |
Apr
(3) |
May
(6) |
Jun
(1) |
Jul
(15) |
Aug
(6) |
Sep
|
Oct
(10) |
Nov
(2) |
Dec
(4) |
| 2003 |
Jan
(1) |
Feb
(7) |
Mar
(3) |
Apr
(6) |
May
(7) |
Jun
(5) |
Jul
(5) |
Aug
(25) |
Sep
(14) |
Oct
(2) |
Nov
|
Dec
(2) |
| 2004 |
Jan
(7) |
Feb
(4) |
Mar
(12) |
Apr
(16) |
May
(43) |
Jun
(56) |
Jul
(43) |
Aug
(40) |
Sep
(66) |
Oct
(12) |
Nov
(26) |
Dec
(10) |
| 2005 |
Jan
(13) |
Feb
(33) |
Mar
(16) |
Apr
(7) |
May
(10) |
Jun
(34) |
Jul
(41) |
Aug
(8) |
Sep
(4) |
Oct
(32) |
Nov
(20) |
Dec
(25) |
| 2006 |
Jan
(30) |
Feb
(101) |
Mar
(5) |
Apr
(75) |
May
(74) |
Jun
(22) |
Jul
(6) |
Aug
(70) |
Sep
(19) |
Oct
(21) |
Nov
(31) |
Dec
(50) |
| 2007 |
Jan
(15) |
Feb
(20) |
Mar
(24) |
Apr
(33) |
May
(13) |
Jun
(18) |
Jul
(13) |
Aug
(7) |
Sep
(63) |
Oct
(68) |
Nov
(29) |
Dec
(68) |
| 2008 |
Jan
(30) |
Feb
(33) |
Mar
(30) |
Apr
(103) |
May
(78) |
Jun
(48) |
Jul
(72) |
Aug
(24) |
Sep
(62) |
Oct
(63) |
Nov
(70) |
Dec
(37) |
| 2009 |
Jan
(34) |
Feb
(35) |
Mar
(64) |
Apr
(34) |
May
(34) |
Jun
(58) |
Jul
(30) |
Aug
(30) |
Sep
(46) |
Oct
(52) |
Nov
(12) |
Dec
(23) |
| 2010 |
Jan
(121) |
Feb
(18) |
Mar
(53) |
Apr
(62) |
May
(62) |
Jun
(20) |
Jul
(33) |
Aug
(20) |
Sep
(36) |
Oct
(35) |
Nov
(44) |
Dec
(63) |
| 2011 |
Jan
(19) |
Feb
(32) |
Mar
(94) |
Apr
(41) |
May
(47) |
Jun
(25) |
Jul
(34) |
Aug
(20) |
Sep
(9) |
Oct
(41) |
Nov
(33) |
Dec
(24) |
| 2012 |
Jan
(12) |
Feb
(36) |
Mar
(48) |
Apr
(32) |
May
(20) |
Jun
(15) |
Jul
(32) |
Aug
(13) |
Sep
(33) |
Oct
(54) |
Nov
(25) |
Dec
(16) |
| 2013 |
Jan
(45) |
Feb
(39) |
Mar
(38) |
Apr
(50) |
May
(29) |
Jun
(30) |
Jul
(33) |
Aug
(12) |
Sep
(9) |
Oct
(25) |
Nov
(29) |
Dec
(20) |
| 2014 |
Jan
(25) |
Feb
(19) |
Mar
(16) |
Apr
(33) |
May
(27) |
Jun
(37) |
Jul
(29) |
Aug
(27) |
Sep
(37) |
Oct
(58) |
Nov
(109) |
Dec
(26) |
| 2015 |
Jan
(4) |
Feb
(35) |
Mar
(22) |
Apr
(35) |
May
(28) |
Jun
(20) |
Jul
(4) |
Aug
(16) |
Sep
(37) |
Oct
(13) |
Nov
(13) |
Dec
(14) |
| 2016 |
Jan
(22) |
Feb
(7) |
Mar
(23) |
Apr
(30) |
May
(10) |
Jun
(10) |
Jul
(15) |
Aug
(12) |
Sep
(22) |
Oct
(31) |
Nov
(5) |
Dec
(5) |
| 2017 |
Jan
(30) |
Feb
(25) |
Mar
(28) |
Apr
(4) |
May
(19) |
Jun
(13) |
Jul
(7) |
Aug
(1) |
Sep
(2) |
Oct
(5) |
Nov
(12) |
Dec
(2) |
| 2018 |
Jan
(7) |
Feb
|
Mar
(7) |
Apr
(2) |
May
(8) |
Jun
(18) |
Jul
(6) |
Aug
(3) |
Sep
(15) |
Oct
(33) |
Nov
(13) |
Dec
(7) |
| 2019 |
Jan
(5) |
Feb
(7) |
Mar
(30) |
Apr
(5) |
May
(4) |
Jun
(69) |
Jul
(86) |
Aug
(22) |
Sep
(6) |
Oct
(7) |
Nov
(5) |
Dec
(3) |
| 2020 |
Jan
(10) |
Feb
(12) |
Mar
(22) |
Apr
(5) |
May
(1) |
Jun
(4) |
Jul
(6) |
Aug
|
Sep
(9) |
Oct
|
Nov
|
Dec
(1) |
| 2021 |
Jan
(4) |
Feb
(11) |
Mar
(7) |
Apr
(7) |
May
|
Jun
(3) |
Jul
(10) |
Aug
(6) |
Sep
|
Oct
|
Nov
(18) |
Dec
(2) |
| 2022 |
Jan
(1) |
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
(2) |
Jul
|
Aug
(4) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2023 |
Jan
|
Feb
|
Mar
|
Apr
(1) |
May
(1) |
Jun
|
Jul
|
Aug
(5) |
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <oh...@ya...> - 2019-07-30 19:45:38
|
Also, FYI, here is the response I get when I test the OCSP request using "openssl ocsp":
OCSP Request Data:
Version: 1 (0x0)
Requestor List:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: DD109D9D80B22984C50240DF37F6C75E70E2DEDD
Issuer Key Hash: BC0F770B8DA3B38543C2369366AC02A977C33D52
Serial Number: 3732
Request Extensions:
OCSP Nonce:
04109186E755667555C98040988194088E5D
Responder Error: unauthorized (6)
NOTICE the "Responder Error: unauthorized (6)" error.
I have even deleted the CA from EJBCA OCSP responder and then re-imported that CA's cert and the latest CRL and I am still getting the same error.
Thanks,Jim
On Tuesday, July 30, 2019, 4:37:49 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
I am circling back and trying to do some OCSP response testing with the EJBCA OCSP responder, but when I run "openssl ocsp" testing, I am getting an error (from the EJBCA logging):
16:25:35,230 INFO [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean] (default task-7) Received OCSP request for certificate with serNo: 3a1b, and issuerNameHash: dd109d9d80b22984c50240df37f6c75e70e2dedd. Client ip 192.168.xx.yy.
16:25:35,236 ERROR [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean] (default task-7) Unable to find CA certificate by issuer name hash: dd109d9d80b22984c50240df37f6c75e70e2dedd, or even the default responder: CN=xxxx.
I think that I have that CA imported into EJBCA and also the latest CRL.
Is there a way to find out what that issuer name that it is looking for from the "issuer name hash"?
I'm guessing there probably isn't, so how can I debug why it is not able to find the CA (and CRL from that CA) in EJBCA?
Thanks,Jim
|
|
From: <oh...@ya...> - 2019-07-30 16:37:57
|
Hi, I am circling back and trying to do some OCSP response testing with the EJBCA OCSP responder, but when I run "openssl ocsp" testing, I am getting an error (from the EJBCA logging): 16:25:35,230 INFO [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean] (default task-7) Received OCSP request for certificate with serNo: 3a1b, and issuerNameHash: dd109d9d80b22984c50240df37f6c75e70e2dedd. Client ip 192.168.xx.yy. 16:25:35,236 ERROR [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean] (default task-7) Unable to find CA certificate by issuer name hash: dd109d9d80b22984c50240df37f6c75e70e2dedd, or even the default responder: CN=xxxx. I think that I have that CA imported into EJBCA and also the latest CRL. Is there a way to find out what that issuer name that it is looking for from the "issuer name hash"? I'm guessing there probably isn't, so how can I debug why it is not able to find the CA (and CRL from that CA) in EJBCA? Thanks,Jim |
|
From: <oh...@ya...> - 2019-07-30 16:11:19
|
Aaarh! I accidentally killed the import process after about 18.5 hours :(!!
So now I am trying to decide if I should try to migrate the EJBCA to a larger/faster AWS machine, as I don't think that what appears to be probably a 20+ hour import is reasonable.
Jim
On Tuesday, July 30, 2019, 1:01:37 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
Yes, I did that and also am using the modified java class (I even tagged the code so that the output identifies the change, just to make sure I knew for sure I was using the modified class).
As I mentioned, it was running better at the beginning (~75k entries/hour) but it has been slowly slowing down overnight.
Here's the top (now):
top - 13:00:05 up 22:24, 4 users, load average: 1.29, 1.30, 1.29
Tasks: 104 total, 1 running, 103 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.8 us, 0.7 sy, 0.0 ni, 57.0 id, 37.0 wa, 0.0 hi, 0.3 si, 0.2 st
KiB Mem : 3880256 total, 105380 free, 3393632 used, 381244 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 224116 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3060 root 20 0 3288836 1.043g 0 S 9.0 28.2 198:34.75 java
1250 mysql 20 0 1580252 373048 2304 S 4.7 9.6 95:54.29 mysqld
3382 root 20 0 4671464 1.725g 1556 S 2.7 46.6 60:43.70 java
1 root 20 0 125404 1868 616 S 0.0 0.0 0:07.10 systemd
Jim
On Tuesday, July 30, 2019, 12:42:04 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
If it's a new machine, did you add the database indexes?
On July 30, 2019 1:42:00 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
The machine I am using now is on AWS, a t2-medium, with 2 CPUs and 4GB RAM and 70GB drive, and RHEL7.4(?). I modified the ejbca.sh for "-Xms2g Xmx2g" to avoid the outofmemory error I got previously.
It is still not finished now, after 18.5 hours. The import rate went from about ~75k/hour at the beginning to ~23.5k/hour now, and it is only processed ~435k entries so far, out of ~980k entries total in the CRL file. Maybe I should've picked a larger machine (probably more memory), but it may be too late for that now, since it's already been running for 18.5 hours.
Jim
On Tuesday, July 30, 2019, 7:47:33 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Our team in California has also managed to import the CRL, running about 25 entries per second. If you need any helo, don't hesitate to reach out to them.
In the longer run, with minor code changes, it's possible to make it orders of magnitude faster.
Regards,
Tomas
On July 30, 2019 12:05:36 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
It is still running now, about 5 hours so far, and only about 228K entries.
So far, processing rate has been between 45K per hour (now) and 75K per hour (earlier). I am guessing that it will take about 20 hours to import the whole CRL, if it doesn't blow up.
Jim
On Monday, July 29, 2019, 4:48:19 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
From looking at the entries in CertificateData table, it looks like it was only able to import 130635 entries before the import process died/ended...
Jim
On Monday, July 29, 2019, 3:37:06 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
It looks like the import that I started before going on vacation failed for some reason (possibly the machine got shutdown by our automatic shutdowns... I am not sure).
So I will start the import AGAIN today...
Bottom line is that I am still not able to import the large CRL successfully so far....
Jim
On Monday, July 22, 2019, 4:47:50 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I had already changed that and started a new run before I saw your email, but then I have left for a vacation, and I don't have access to our system until I get back.
I will post when I check it after I get back home.
Jim
On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
tool itself.
-Xmx4096m -Xms4096m
i,e.
exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
Regards,
Tomas
On 2019-07-21 17:19, oh...@ya... wrote:
> Hi,
>
> The import processing crashed :(....
>
> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
> missing in the database
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
> at
> org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>
>
> It got through 242720 entries.
>
> Jim
>
> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I built the new class and JAR and am testing. This looks better. It's
> not quite the rate that you are seeing but it's much better than what I
> was seeing before.
>
> So now it looks like I am getting about 1073 per minute, which is about
> 17 per second. I added some text to the class before I built it (not a
> lot, just some additional strings so I could verify I was using the
> modified class), so I know for sure that I am using the modified Java class.
>
> So anyway, it looks like we are down to about 15 hours to import that
> one CRL now :) ...
>
> Jim
>
>
>
> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I was doing a diff/fc file compare between the one you attached, and the
> last one I had that I used before, and it seems like there is a
> difference between those. Is the code that you just attached different
> than the patch you gave me before? Here's the file compare output (the
> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>
> Comparing files
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> and CAIMPORTCRLCOMMAND.JAVA
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> ***** CAIMPORTCRLCOMMAND.JAVA
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> *****
>
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>
> private KeyPair getStaticRSAKeyPair() {
> // A switch to use different keys depending on the sigAlg so we
> can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
> return keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException parsing hard
> coded presign key. This should never happen: ", e);
> }
> }
> ***** CAIMPORTCRLCOMMAND.JAVA
>
> private static KeyPair staticKp = null;
> private KeyPair getStaticRSAKeyPair() {
> if (staticKp == null) {
> synchronized (this) {
> if (staticKp == null) {
> // A switch to use different keys depending on the
> sigAlg so we can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair)
> pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
> ME);
> staticKp = keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException
> parsing hard coded presign key. This should never happen:
> ", e);
> }
> }
> }
> }
> return staticKp;
> }
> *****
>
>
>
> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> AACK! Yes, I forgot all about that and just used the vanilla software
> :(! Now, if I can remember how to do that patch, I will try it :(...
>
> Thanks,
> Jim
>
> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> Did you forget to patch the java file? The top output suggest you did.
> Attached the latest patched file that I used for the import.
>
> Regards,
> Tomas
>
> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
>> just started testing with that same CRL, and unfortunately, it is not
>> doing very well... actually even slower than on my earlier dev machine
>> under VBox :(!!
>>
>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
>> CPUs. I converted the tables to InnoDB and also ran the create index
>> SQL script.
>>
>> It looks like the import is only processing about 5000 entries per hour.
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-30 13:01:20
|
Hi,
Yes, I did that and also am using the modified java class (I even tagged the code so that the output identifies the change, just to make sure I knew for sure I was using the modified class).
As I mentioned, it was running better at the beginning (~75k entries/hour) but it has been slowly slowing down overnight.
Here's the top (now):
top - 13:00:05 up 22:24, 4 users, load average: 1.29, 1.30, 1.29
Tasks: 104 total, 1 running, 103 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.8 us, 0.7 sy, 0.0 ni, 57.0 id, 37.0 wa, 0.0 hi, 0.3 si, 0.2 st
KiB Mem : 3880256 total, 105380 free, 3393632 used, 381244 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 224116 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3060 root 20 0 3288836 1.043g 0 S 9.0 28.2 198:34.75 java
1250 mysql 20 0 1580252 373048 2304 S 4.7 9.6 95:54.29 mysqld
3382 root 20 0 4671464 1.725g 1556 S 2.7 46.6 60:43.70 java
1 root 20 0 125404 1868 616 S 0.0 0.0 0:07.10 systemd
Jim
On Tuesday, July 30, 2019, 12:42:04 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
If it's a new machine, did you add the database indexes?
On July 30, 2019 1:42:00 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
The machine I am using now is on AWS, a t2-medium, with 2 CPUs and 4GB RAM and 70GB drive, and RHEL7.4(?). I modified the ejbca.sh for "-Xms2g Xmx2g" to avoid the outofmemory error I got previously.
It is still not finished now, after 18.5 hours. The import rate went from about ~75k/hour at the beginning to ~23.5k/hour now, and it is only processed ~435k entries so far, out of ~980k entries total in the CRL file. Maybe I should've picked a larger machine (probably more memory), but it may be too late for that now, since it's already been running for 18.5 hours.
Jim
On Tuesday, July 30, 2019, 7:47:33 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Our team in California has also managed to import the CRL, running about 25 entries per second. If you need any helo, don't hesitate to reach out to them.
In the longer run, with minor code changes, it's possible to make it orders of magnitude faster.
Regards,
Tomas
On July 30, 2019 12:05:36 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
It is still running now, about 5 hours so far, and only about 228K entries.
So far, processing rate has been between 45K per hour (now) and 75K per hour (earlier). I am guessing that it will take about 20 hours to import the whole CRL, if it doesn't blow up.
Jim
On Monday, July 29, 2019, 4:48:19 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
From looking at the entries in CertificateData table, it looks like it was only able to import 130635 entries before the import process died/ended...
Jim
On Monday, July 29, 2019, 3:37:06 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
It looks like the import that I started before going on vacation failed for some reason (possibly the machine got shutdown by our automatic shutdowns... I am not sure).
So I will start the import AGAIN today...
Bottom line is that I am still not able to import the large CRL successfully so far....
Jim
On Monday, July 22, 2019, 4:47:50 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I had already changed that and started a new run before I saw your email, but then I have left for a vacation, and I don't have access to our system until I get back.
I will post when I check it after I get back home.
Jim
On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
tool itself.
-Xmx4096m -Xms4096m
i,e.
exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
Regards,
Tomas
On 2019-07-21 17:19, oh...@ya... wrote:
> Hi,
>
> The import processing crashed :(....
>
> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
> missing in the database
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
> at
> org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>
>
> It got through 242720 entries.
>
> Jim
>
> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I built the new class and JAR and am testing. This looks better. It's
> not quite the rate that you are seeing but it's much better than what I
> was seeing before.
>
> So now it looks like I am getting about 1073 per minute, which is about
> 17 per second. I added some text to the class before I built it (not a
> lot, just some additional strings so I could verify I was using the
> modified class), so I know for sure that I am using the modified Java class.
>
> So anyway, it looks like we are down to about 15 hours to import that
> one CRL now :) ...
>
> Jim
>
>
>
> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I was doing a diff/fc file compare between the one you attached, and the
> last one I had that I used before, and it seems like there is a
> difference between those. Is the code that you just attached different
> than the patch you gave me before? Here's the file compare output (the
> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>
> Comparing files
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> and CAIMPORTCRLCOMMAND.JAVA
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> ***** CAIMPORTCRLCOMMAND.JAVA
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> *****
>
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>
> private KeyPair getStaticRSAKeyPair() {
> // A switch to use different keys depending on the sigAlg so we
> can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
> return keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException parsing hard
> coded presign key. This should never happen: ", e);
> }
> }
> ***** CAIMPORTCRLCOMMAND.JAVA
>
> private static KeyPair staticKp = null;
> private KeyPair getStaticRSAKeyPair() {
> if (staticKp == null) {
> synchronized (this) {
> if (staticKp == null) {
> // A switch to use different keys depending on the
> sigAlg so we can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair)
> pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
> ME);
> staticKp = keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException
> parsing hard coded presign key. This should never happen:
> ", e);
> }
> }
> }
> }
> return staticKp;
> }
> *****
>
>
>
> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> AACK! Yes, I forgot all about that and just used the vanilla software
> :(! Now, if I can remember how to do that patch, I will try it :(...
>
> Thanks,
> Jim
>
> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> Did you forget to patch the java file? The top output suggest you did.
> Attached the latest patched file that I used for the import.
>
> Regards,
> Tomas
>
> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
>> just started testing with that same CRL, and unfortunately, it is not
>> doing very well... actually even slower than on my earlier dev machine
>> under VBox :(!!
>>
>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
>> CPUs. I converted the tables to InnoDB and also ran the create index
>> SQL script.
>>
>> It looks like the import is only processing about 5000 entries per hour.
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: Tomas G. <to...@pr...> - 2019-07-30 12:42:11
|
If it's a new machine, did you add the database indexes?
On July 30, 2019 1:42:00 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
> Hi,
>The machine I am using now is on AWS, a t2-medium, with 2 CPUs and 4GB
>RAM and 70GB drive, and RHEL7.4(?). I modified the ejbca.sh for
>"-Xms2g Xmx2g" to avoid the outofmemory error I got previously.
>
>It is still not finished now, after 18.5 hours. The import rate went
>from about ~75k/hour at the beginning to ~23.5k/hour now, and it is
>only processed ~435k entries so far, out of ~980k entries total in the
>CRL file. Maybe I should've picked a larger machine (probably more
>memory), but it may be too late for that now, since it's already been
>running for 18.5 hours.
>Jim
>
>On Tuesday, July 30, 2019, 7:47:33 AM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>Our team in California has also managed to import the CRL, running
>about 25 entries per second. If you need any helo, don't hesitate to
>reach out to them.
>
>In the longer run, with minor code changes, it's possible to make it
>orders of magnitude faster.
>
>Regards,
>Tomas
>
>
>On July 30, 2019 12:05:36 AM GMT+02:00, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>It is still running now, about 5 hours so far, and only about 228K
>entries.
>
>So far, processing rate has been between 45K per hour (now) and 75K per
>hour (earlier). I am guessing that it will take about 20 hours to
>import the whole CRL, if it doesn't blow up.
>
>Jim
>
>On Monday, July 29, 2019, 4:48:19 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
>From looking at the entries in CertificateData table, it looks like it
>was only able to import 130635 entries before the import process
>died/ended...
>Jim
>
>
>
>On Monday, July 29, 2019, 3:37:06 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>It looks like the import that I started before going on vacation failed
>for some reason (possibly the machine got shutdown by our automatic
>shutdowns... I am not sure).
>So I will start the import AGAIN today...
>Bottom line is that I am still not able to import the large CRL
>successfully so far....
>
>Jim
>
>
>On Monday, July 22, 2019, 4:47:50 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>I had already changed that and started a new run before I saw your
>email, but then I have left for a vacation, and I don't have access to
>our system until I get back.
>I will post when I check it after I get back home.
>Jim
>
>
>On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>
>I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
>tool itself.
>
> -Xmx4096m -Xms4096m
>
>i,e.
>
>exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
>
>
>Regards,
>Tomas
>
>On 2019-07-21 17:19, oh...@ya... wrote:
>> Hi,
>>
>> The import processing crashed :(....
>>
>> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
>> missing in the database
>> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead
>limit
>> exceeded
>> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
>> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
>> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
>> at
>>
>org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
>> at
>>
>org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
>> at
>>
>org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
>> at
>>
>org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
>> at
>>
>org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
>> at
>>
>org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
>> at
>>
>org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
>> at
>>
>org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
>> at
>>
>org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
>> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>>
>>
>> It got through 242720 entries.
>>
>> Jim
>>
>> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> Hi,
>>
>> I built the new class and JAR and am testing. This looks better.
>It's
>> not quite the rate that you are seeing but it's much better than what
>I
>> was seeing before.
>>
>> So now it looks like I am getting about 1073 per minute, which is
>about
>> 17 per second. I added some text to the class before I built it (not
>a
>> lot, just some additional strings so I could verify I was using the
>> modified class), so I know for sure that I am using the modified Java
>class.
>>
>> So anyway, it looks like we are down to about 15 hours to import that
>> one CRL now :) ...
>>
>> Jim
>>
>>
>>
>> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via
>Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> Hi,
>>
>> I was doing a diff/fc file compare between the one you attached, and
>the
>> last one I had that I used before, and it seems like there is a
>> difference between those. Is the code that you just attached
>different
>> than the patch you gave me before? Here's the file compare output
>(the
>> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>>
>> Comparing files
>>
>CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>> and CAIMPORTCRLCOMMAND.JAVA
>> *****
>>
>CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>> final EndEntityInformation
>> missingUserEndEntityInformation =
>EjbRemoteHelper.INSTANCE.getRemoteSession(
>>
>>
>>
>EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
>> missing_user_name);
>> ***** CAIMPORTCRLCOMMAND.JAVA
>> final EndEntityInformation
>> missingUserEndEntityInformation =
>EjbRemoteHelper.INSTANCE.getRemoteSession(
>>
>>
>EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
>> missing_user_name);
>> *****
>>
>> *****
>>
>CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>>
>> private KeyPair getStaticRSAKeyPair() {
>> // A switch to use different keys depending on the sigAlg so
>we
>> can sign using the CAs signature algorithm
>> final StringReader reader = new
>> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
>> try (PEMParser pemParser = new PEMParser(reader)) {
>> PEMKeyPair pemKeyPair = (PEMKeyPair)
>pemParser.readObject();
>> JcaPEMKeyConverter keyConverter = new
>> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
>> return keyConverter.getKeyPair(pemKeyPair);
>> } catch (IOException e) {
>> throw new IllegalStateException("IOException parsing hard
>> coded presign key. This should never happen: ", e);
>> }
>> }
>> ***** CAIMPORTCRLCOMMAND.JAVA
>>
>> private static KeyPair staticKp = null;
>> private KeyPair getStaticRSAKeyPair() {
>> if (staticKp == null) {
>> synchronized (this) {
>> if (staticKp == null) {
>> // A switch to use different keys depending on
>the
>> sigAlg so we can sign using the CAs signature algorithm
>> final StringReader reader = new
>> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
>> try (PEMParser pemParser = new PEMParser(reader))
>{
>> PEMKeyPair pemKeyPair = (PEMKeyPair)
>> pemParser.readObject();
>> JcaPEMKeyConverter keyConverter = new
>> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
>> ME);
>> staticKp =
>keyConverter.getKeyPair(pemKeyPair);
>> } catch (IOException e) {
>> throw new IllegalStateException("IOException
>> parsing hard coded presign key. This should never happen:
>> ", e);
>> }
>> }
>> }
>> }
>> return staticKp;
>> }
>> *****
>>
>>
>>
>> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via
>Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> AACK! Yes, I forgot all about that and just used the vanilla
>software
>> :(! Now, if I can remember how to do that patch, I will try it :(...
>>
>> Thanks,
>> Jim
>>
>> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
>> <to...@pr...> wrote:
>>
>>
>>
>> Did you forget to patch the java file? The top output suggest you
>did.
>> Attached the latest patched file that I used for the import.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>>> Hi,
>>>
>>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and
>I
>>> just started testing with that same CRL, and unfortunately, it is
>not
>>> doing very well... actually even slower than on my earlier dev
>machine
>>> under VBox :(!!
>>>
>>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and
>2
>>> CPUs. I converted the tables to InnoDB and also ran the create
>index
>>> SQL script.
>>>
>>> It looks like the import is only processing about 5000 entries per
>hour.
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
|
|
From: <oh...@ya...> - 2019-07-30 11:42:18
|
Hi,
The machine I am using now is on AWS, a t2-medium, with 2 CPUs and 4GB RAM and 70GB drive, and RHEL7.4(?). I modified the ejbca.sh for "-Xms2g Xmx2g" to avoid the outofmemory error I got previously.
It is still not finished now, after 18.5 hours. The import rate went from about ~75k/hour at the beginning to ~23.5k/hour now, and it is only processed ~435k entries so far, out of ~980k entries total in the CRL file. Maybe I should've picked a larger machine (probably more memory), but it may be too late for that now, since it's already been running for 18.5 hours.
Jim
On Tuesday, July 30, 2019, 7:47:33 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Our team in California has also managed to import the CRL, running about 25 entries per second. If you need any helo, don't hesitate to reach out to them.
In the longer run, with minor code changes, it's possible to make it orders of magnitude faster.
Regards,
Tomas
On July 30, 2019 12:05:36 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
It is still running now, about 5 hours so far, and only about 228K entries.
So far, processing rate has been between 45K per hour (now) and 75K per hour (earlier). I am guessing that it will take about 20 hours to import the whole CRL, if it doesn't blow up.
Jim
On Monday, July 29, 2019, 4:48:19 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
From looking at the entries in CertificateData table, it looks like it was only able to import 130635 entries before the import process died/ended...
Jim
On Monday, July 29, 2019, 3:37:06 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
It looks like the import that I started before going on vacation failed for some reason (possibly the machine got shutdown by our automatic shutdowns... I am not sure).
So I will start the import AGAIN today...
Bottom line is that I am still not able to import the large CRL successfully so far....
Jim
On Monday, July 22, 2019, 4:47:50 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I had already changed that and started a new run before I saw your email, but then I have left for a vacation, and I don't have access to our system until I get back.
I will post when I check it after I get back home.
Jim
On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
tool itself.
-Xmx4096m -Xms4096m
i,e.
exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
Regards,
Tomas
On 2019-07-21 17:19, oh...@ya... wrote:
> Hi,
>
> The import processing crashed :(....
>
> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
> missing in the database
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
> at
> org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>
>
> It got through 242720 entries.
>
> Jim
>
> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I built the new class and JAR and am testing. This looks better. It's
> not quite the rate that you are seeing but it's much better than what I
> was seeing before.
>
> So now it looks like I am getting about 1073 per minute, which is about
> 17 per second. I added some text to the class before I built it (not a
> lot, just some additional strings so I could verify I was using the
> modified class), so I know for sure that I am using the modified Java class.
>
> So anyway, it looks like we are down to about 15 hours to import that
> one CRL now :) ...
>
> Jim
>
>
>
> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I was doing a diff/fc file compare between the one you attached, and the
> last one I had that I used before, and it seems like there is a
> difference between those. Is the code that you just attached different
> than the patch you gave me before? Here's the file compare output (the
> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>
> Comparing files
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> and CAIMPORTCRLCOMMAND.JAVA
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> ***** CAIMPORTCRLCOMMAND.JAVA
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> *****
>
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>
> private KeyPair getStaticRSAKeyPair() {
> // A switch to use different keys depending on the sigAlg so we
> can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
> return keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException parsing hard
> coded presign key. This should never happen: ", e);
> }
> }
> ***** CAIMPORTCRLCOMMAND.JAVA
>
> private static KeyPair staticKp = null;
> private KeyPair getStaticRSAKeyPair() {
> if (staticKp == null) {
> synchronized (this) {
> if (staticKp == null) {
> // A switch to use different keys depending on the
> sigAlg so we can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair)
> pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
> ME);
> staticKp = keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException
> parsing hard coded presign key. This should never happen:
> ", e);
> }
> }
> }
> }
> return staticKp;
> }
> *****
>
>
>
> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> AACK! Yes, I forgot all about that and just used the vanilla software
> :(! Now, if I can remember how to do that patch, I will try it :(...
>
> Thanks,
> Jim
>
> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> Did you forget to patch the java file? The top output suggest you did.
> Attached the latest patched file that I used for the import.
>
> Regards,
> Tomas
>
> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
>> just started testing with that same CRL, and unfortunately, it is not
>> doing very well... actually even slower than on my earlier dev machine
>> under VBox :(!!
>>
>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
>> CPUs. I converted the tables to InnoDB and also ran the create index
>> SQL script.
>>
>> It looks like the import is only processing about 5000 entries per hour.
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: Tomas G. <to...@pr...> - 2019-07-30 07:47:41
|
Our team in California has also managed to import the CRL, running about 25 entries per second. If you need any helo, don't hesitate to reach out to them.
In the longer run, with minor code changes, it's possible to make it orders of magnitude faster.
Regards,
Tomas
On July 30, 2019 12:05:36 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
>It is still running now, about 5 hours so far, and only about 228K
>entries.
>
>So far, processing rate has been between 45K per hour (now) and 75K per
>hour (earlier). I am guessing that it will take about 20 hours to
>import the whole CRL, if it doesn't blow up.
>
>Jim
>
>On Monday, July 29, 2019, 4:48:19 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
>From looking at the entries in CertificateData table, it looks like it
>was only able to import 130635 entries before the import process
>died/ended...
>Jim
>
>
>
>On Monday, July 29, 2019, 3:37:06 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>It looks like the import that I started before going on vacation failed
>for some reason (possibly the machine got shutdown by our automatic
>shutdowns... I am not sure).
>So I will start the import AGAIN today...
>Bottom line is that I am still not able to import the large CRL
>successfully so far....
>
>Jim
>
>
>On Monday, July 22, 2019, 4:47:50 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>I had already changed that and started a new run before I saw your
>email, but then I have left for a vacation, and I don't have access to
>our system until I get back.
>I will post when I check it after I get back home.
>Jim
>
>
>On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>
>I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
>tool itself.
>
> -Xmx4096m -Xms4096m
>
>i,e.
>
>exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
>
>
>Regards,
>Tomas
>
>On 2019-07-21 17:19, oh...@ya... wrote:
>> Hi,
>>
>> The import processing crashed :(....
>>
>> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
>> missing in the database
>> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead
>limit
>> exceeded
>> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
>> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
>> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
>> at
>>
>org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
>> at
>>
>org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
>> at
>>
>org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
>> at
>>
>org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
>> at
>>
>org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
>> at
>>
>org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
>> at
>>
>org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
>> at
>>
>org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
>> at
>>
>org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
>> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>>
>>
>> It got through 242720 entries.
>>
>> Jim
>>
>> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> Hi,
>>
>> I built the new class and JAR and am testing. This looks better.
>It's
>> not quite the rate that you are seeing but it's much better than what
>I
>> was seeing before.
>>
>> So now it looks like I am getting about 1073 per minute, which is
>about
>> 17 per second. I added some text to the class before I built it (not
>a
>> lot, just some additional strings so I could verify I was using the
>> modified class), so I know for sure that I am using the modified Java
>class.
>>
>> So anyway, it looks like we are down to about 15 hours to import that
>> one CRL now :) ...
>>
>> Jim
>>
>>
>>
>> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via
>Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> Hi,
>>
>> I was doing a diff/fc file compare between the one you attached, and
>the
>> last one I had that I used before, and it seems like there is a
>> difference between those. Is the code that you just attached
>different
>> than the patch you gave me before? Here's the file compare output
>(the
>> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>>
>> Comparing files
>>
>CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>> and CAIMPORTCRLCOMMAND.JAVA
>> *****
>>
>CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>> final EndEntityInformation
>> missingUserEndEntityInformation =
>EjbRemoteHelper.INSTANCE.getRemoteSession(
>>
>>
>>
>EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
>> missing_user_name);
>> ***** CAIMPORTCRLCOMMAND.JAVA
>> final EndEntityInformation
>> missingUserEndEntityInformation =
>EjbRemoteHelper.INSTANCE.getRemoteSession(
>>
>>
>EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
>> missing_user_name);
>> *****
>>
>> *****
>>
>CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>>
>> private KeyPair getStaticRSAKeyPair() {
>> // A switch to use different keys depending on the sigAlg so
>we
>> can sign using the CAs signature algorithm
>> final StringReader reader = new
>> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
>> try (PEMParser pemParser = new PEMParser(reader)) {
>> PEMKeyPair pemKeyPair = (PEMKeyPair)
>pemParser.readObject();
>> JcaPEMKeyConverter keyConverter = new
>> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
>> return keyConverter.getKeyPair(pemKeyPair);
>> } catch (IOException e) {
>> throw new IllegalStateException("IOException parsing hard
>> coded presign key. This should never happen: ", e);
>> }
>> }
>> ***** CAIMPORTCRLCOMMAND.JAVA
>>
>> private static KeyPair staticKp = null;
>> private KeyPair getStaticRSAKeyPair() {
>> if (staticKp == null) {
>> synchronized (this) {
>> if (staticKp == null) {
>> // A switch to use different keys depending on
>the
>> sigAlg so we can sign using the CAs signature algorithm
>> final StringReader reader = new
>> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
>> try (PEMParser pemParser = new PEMParser(reader))
>{
>> PEMKeyPair pemKeyPair = (PEMKeyPair)
>> pemParser.readObject();
>> JcaPEMKeyConverter keyConverter = new
>> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
>> ME);
>> staticKp =
>keyConverter.getKeyPair(pemKeyPair);
>> } catch (IOException e) {
>> throw new IllegalStateException("IOException
>> parsing hard coded presign key. This should never happen:
>> ", e);
>> }
>> }
>> }
>> }
>> return staticKp;
>> }
>> *****
>>
>>
>>
>> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via
>Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> AACK! Yes, I forgot all about that and just used the vanilla
>software
>> :(! Now, if I can remember how to do that patch, I will try it :(...
>>
>> Thanks,
>> Jim
>>
>> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
>> <to...@pr...> wrote:
>>
>>
>>
>> Did you forget to patch the java file? The top output suggest you
>did.
>> Attached the latest patched file that I used for the import.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>>> Hi,
>>>
>>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and
>I
>>> just started testing with that same CRL, and unfortunately, it is
>not
>>> doing very well... actually even slower than on my earlier dev
>machine
>>> under VBox :(!!
>>>
>>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and
>2
>>> CPUs. I converted the tables to InnoDB and also ran the create
>index
>>> SQL script.
>>>
>>> It looks like the import is only processing about 5000 entries per
>hour.
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
|
|
From: <oh...@ya...> - 2019-07-29 22:05:49
|
It is still running now, about 5 hours so far, and only about 228K entries.
So far, processing rate has been between 45K per hour (now) and 75K per hour (earlier). I am guessing that it will take about 20 hours to import the whole CRL, if it doesn't blow up.
Jim
On Monday, July 29, 2019, 4:48:19 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
From looking at the entries in CertificateData table, it looks like it was only able to import 130635 entries before the import process died/ended...
Jim
On Monday, July 29, 2019, 3:37:06 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
It looks like the import that I started before going on vacation failed for some reason (possibly the machine got shutdown by our automatic shutdowns... I am not sure).
So I will start the import AGAIN today...
Bottom line is that I am still not able to import the large CRL successfully so far....
Jim
On Monday, July 22, 2019, 4:47:50 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I had already changed that and started a new run before I saw your email, but then I have left for a vacation, and I don't have access to our system until I get back.
I will post when I check it after I get back home.
Jim
On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
tool itself.
-Xmx4096m -Xms4096m
i,e.
exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
Regards,
Tomas
On 2019-07-21 17:19, oh...@ya... wrote:
> Hi,
>
> The import processing crashed :(....
>
> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
> missing in the database
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
> at
> org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>
>
> It got through 242720 entries.
>
> Jim
>
> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I built the new class and JAR and am testing. This looks better. It's
> not quite the rate that you are seeing but it's much better than what I
> was seeing before.
>
> So now it looks like I am getting about 1073 per minute, which is about
> 17 per second. I added some text to the class before I built it (not a
> lot, just some additional strings so I could verify I was using the
> modified class), so I know for sure that I am using the modified Java class.
>
> So anyway, it looks like we are down to about 15 hours to import that
> one CRL now :) ...
>
> Jim
>
>
>
> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I was doing a diff/fc file compare between the one you attached, and the
> last one I had that I used before, and it seems like there is a
> difference between those. Is the code that you just attached different
> than the patch you gave me before? Here's the file compare output (the
> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>
> Comparing files
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> and CAIMPORTCRLCOMMAND.JAVA
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> ***** CAIMPORTCRLCOMMAND.JAVA
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> *****
>
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>
> private KeyPair getStaticRSAKeyPair() {
> // A switch to use different keys depending on the sigAlg so we
> can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
> return keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException parsing hard
> coded presign key. This should never happen: ", e);
> }
> }
> ***** CAIMPORTCRLCOMMAND.JAVA
>
> private static KeyPair staticKp = null;
> private KeyPair getStaticRSAKeyPair() {
> if (staticKp == null) {
> synchronized (this) {
> if (staticKp == null) {
> // A switch to use different keys depending on the
> sigAlg so we can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair)
> pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
> ME);
> staticKp = keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException
> parsing hard coded presign key. This should never happen:
> ", e);
> }
> }
> }
> }
> return staticKp;
> }
> *****
>
>
>
> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> AACK! Yes, I forgot all about that and just used the vanilla software
> :(! Now, if I can remember how to do that patch, I will try it :(...
>
> Thanks,
> Jim
>
> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> Did you forget to patch the java file? The top output suggest you did.
> Attached the latest patched file that I used for the import.
>
> Regards,
> Tomas
>
> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
>> just started testing with that same CRL, and unfortunately, it is not
>> doing very well... actually even slower than on my earlier dev machine
>> under VBox :(!!
>>
>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
>> CPUs. I converted the tables to InnoDB and also ran the create index
>> SQL script.
>>
>> It looks like the import is only processing about 5000 entries per hour.
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-29 16:48:00
|
From looking at the entries in CertificateData table, it looks like it was only able to import 130635 entries before the import process died/ended...
Jim
On Monday, July 29, 2019, 3:37:06 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
It looks like the import that I started before going on vacation failed for some reason (possibly the machine got shutdown by our automatic shutdowns... I am not sure).
So I will start the import AGAIN today...
Bottom line is that I am still not able to import the large CRL successfully so far....
Jim
On Monday, July 22, 2019, 4:47:50 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I had already changed that and started a new run before I saw your email, but then I have left for a vacation, and I don't have access to our system until I get back.
I will post when I check it after I get back home.
Jim
On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
tool itself.
-Xmx4096m -Xms4096m
i,e.
exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
Regards,
Tomas
On 2019-07-21 17:19, oh...@ya... wrote:
> Hi,
>
> The import processing crashed :(....
>
> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
> missing in the database
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
> at
> org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>
>
> It got through 242720 entries.
>
> Jim
>
> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I built the new class and JAR and am testing. This looks better. It's
> not quite the rate that you are seeing but it's much better than what I
> was seeing before.
>
> So now it looks like I am getting about 1073 per minute, which is about
> 17 per second. I added some text to the class before I built it (not a
> lot, just some additional strings so I could verify I was using the
> modified class), so I know for sure that I am using the modified Java class.
>
> So anyway, it looks like we are down to about 15 hours to import that
> one CRL now :) ...
>
> Jim
>
>
>
> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I was doing a diff/fc file compare between the one you attached, and the
> last one I had that I used before, and it seems like there is a
> difference between those. Is the code that you just attached different
> than the patch you gave me before? Here's the file compare output (the
> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>
> Comparing files
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> and CAIMPORTCRLCOMMAND.JAVA
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> ***** CAIMPORTCRLCOMMAND.JAVA
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> *****
>
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>
> private KeyPair getStaticRSAKeyPair() {
> // A switch to use different keys depending on the sigAlg so we
> can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
> return keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException parsing hard
> coded presign key. This should never happen: ", e);
> }
> }
> ***** CAIMPORTCRLCOMMAND.JAVA
>
> private static KeyPair staticKp = null;
> private KeyPair getStaticRSAKeyPair() {
> if (staticKp == null) {
> synchronized (this) {
> if (staticKp == null) {
> // A switch to use different keys depending on the
> sigAlg so we can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair)
> pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
> ME);
> staticKp = keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException
> parsing hard coded presign key. This should never happen:
> ", e);
> }
> }
> }
> }
> return staticKp;
> }
> *****
>
>
>
> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> AACK! Yes, I forgot all about that and just used the vanilla software
> :(! Now, if I can remember how to do that patch, I will try it :(...
>
> Thanks,
> Jim
>
> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> Did you forget to patch the java file? The top output suggest you did.
> Attached the latest patched file that I used for the import.
>
> Regards,
> Tomas
>
> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
>> just started testing with that same CRL, and unfortunately, it is not
>> doing very well... actually even slower than on my earlier dev machine
>> under VBox :(!!
>>
>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
>> CPUs. I converted the tables to InnoDB and also ran the create index
>> SQL script.
>>
>> It looks like the import is only processing about 5000 entries per hour.
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-29 15:36:46
|
Hi,
It looks like the import that I started before going on vacation failed for some reason (possibly the machine got shutdown by our automatic shutdowns... I am not sure).
So I will start the import AGAIN today...
Bottom line is that I am still not able to import the large CRL successfully so far....
Jim
On Monday, July 22, 2019, 4:47:50 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I had already changed that and started a new run before I saw your email, but then I have left for a vacation, and I don't have access to our system until I get back.
I will post when I check it after I get back home.
Jim
On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
tool itself.
-Xmx4096m -Xms4096m
i,e.
exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
Regards,
Tomas
On 2019-07-21 17:19, oh...@ya... wrote:
> Hi,
>
> The import processing crashed :(....
>
> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
> missing in the database
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
> at
> org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>
>
> It got through 242720 entries.
>
> Jim
>
> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I built the new class and JAR and am testing. This looks better. It's
> not quite the rate that you are seeing but it's much better than what I
> was seeing before.
>
> So now it looks like I am getting about 1073 per minute, which is about
> 17 per second. I added some text to the class before I built it (not a
> lot, just some additional strings so I could verify I was using the
> modified class), so I know for sure that I am using the modified Java class.
>
> So anyway, it looks like we are down to about 15 hours to import that
> one CRL now :) ...
>
> Jim
>
>
>
> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I was doing a diff/fc file compare between the one you attached, and the
> last one I had that I used before, and it seems like there is a
> difference between those. Is the code that you just attached different
> than the patch you gave me before? Here's the file compare output (the
> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>
> Comparing files
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> and CAIMPORTCRLCOMMAND.JAVA
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> ***** CAIMPORTCRLCOMMAND.JAVA
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> *****
>
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>
> private KeyPair getStaticRSAKeyPair() {
> // A switch to use different keys depending on the sigAlg so we
> can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
> return keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException parsing hard
> coded presign key. This should never happen: ", e);
> }
> }
> ***** CAIMPORTCRLCOMMAND.JAVA
>
> private static KeyPair staticKp = null;
> private KeyPair getStaticRSAKeyPair() {
> if (staticKp == null) {
> synchronized (this) {
> if (staticKp == null) {
> // A switch to use different keys depending on the
> sigAlg so we can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair)
> pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
> ME);
> staticKp = keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException
> parsing hard coded presign key. This should never happen:
> ", e);
> }
> }
> }
> }
> return staticKp;
> }
> *****
>
>
>
> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> AACK! Yes, I forgot all about that and just used the vanilla software
> :(! Now, if I can remember how to do that patch, I will try it :(...
>
> Thanks,
> Jim
>
> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> Did you forget to patch the java file? The top output suggest you did.
> Attached the latest patched file that I used for the import.
>
> Regards,
> Tomas
>
> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
>> just started testing with that same CRL, and unfortunately, it is not
>> doing very well... actually even slower than on my earlier dev machine
>> under VBox :(!!
>>
>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
>> CPUs. I converted the tables to InnoDB and also ran the create index
>> SQL script.
>>
>> It looks like the import is only processing about 5000 entries per hour.
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: Jaime H. <hab...@gm...> - 2019-07-26 00:25:39
|
PublishQueueProcessWorker is currently always reporting a NO_ACTION result,
even when there are successes or failures. Patch follows:
---
modules/ejbca-common-web/src/org/ejbca/core/model/services/workers/PublishQueueProcessWorker.java
(revision 32884)
+++
modules/ejbca-common-web/src/org/ejbca/core/model/services/workers/PublishQueueProcessWorker.java
(date 1564026429457)
@@ -111,7 +111,7 @@
int publisherId = Integer.valueOf(ids[i]);
// Get everything from the queue for this
publisher id
BasePublisher publisher =
publisherSession.getPublisher(publisherId);
-
publisherQueueSession.plainFifoTryAlwaysLimit100EntriesOrderByTimeCreated(getAdmin(),
publisher);
+
publishingResult.append(publisherQueueSession.plainFifoTryAlwaysLimit100EntriesOrderByTimeCreated(getAdmin(),
publisher));
}
} else {
log.debug("No publisher ids configured for worker.");
@@ -121,23 +121,25 @@
runmap.put(this.serviceName, Boolean.FALSE);
}
}
- } else {
- log.info(InternalEjbcaResources.getInstance().getLocalizedMessage("services.alreadyrunninginvm",
PublishQueueProcessWorker.class.getName()));
- }
- log.trace("<work");
- if (publishingResult.getSuccesses() == 0 &&
publishingResult.getFailures() == 0) {
- return new ServiceExecutionResult(Result.NO_ACTION,
- "Publishing Queue Service " + serviceName + " ran, but
the publishing queue was either empty or the publisher(s) could not
connect.");
- } else {
- if (publishingResult.getFailures() != 0) {
- return new ServiceExecutionResult(Result.FAILURE,
- "Publishing Queue Service " + serviceName + " ran
with " + publishingResult.getFailures() + " failed publishing operations"
- + (publishingResult.getSuccesses() == 0 ?
"."
- : " and " +
publishingResult.getSuccesses() + " successful publishing operations."));
- } else {
- return new ServiceExecutionResult(Result.SUCCESS,
"Publishing Queue Service " + serviceName + " ran with "
- + publishingResult.getSuccesses() + " successful
publishing operations.");
+ log.trace("<work");
+ if (publishingResult.getSuccesses() == 0 &&
publishingResult.getFailures() == 0) {
+ return new ServiceExecutionResult(Result.NO_ACTION,
+ "Publishing Queue Service " + serviceName + " ran,
but the publishing queue was either empty or the publisher(s) could not
connect.");
+ } else {
+ if (publishingResult.getFailures() != 0) {
+ return new ServiceExecutionResult(Result.FAILURE,
+ "Publishing Queue Service " + serviceName + "
ran with " + publishingResult.getFailures() + " failed publishing
operations"
+ + (publishingResult.getSuccesses() ==
0 ? "."
+ : " and " +
publishingResult.getSuccesses() + " successful publishing operations."));
+ } else {
+ return new ServiceExecutionResult(Result.SUCCESS,
"Publishing Queue Service " + serviceName + " ran with "
+ + publishingResult.getSuccesses() + "
successful publishing operations.");
+ }
}
+ } else {
+ String msg =
InternalEjbcaResources.getInstance().getLocalizedMessage("services.alreadyrunninginvm",
PublishQueueProcessWorker.class.getName());
+ log.info(msg);
+ return new ServiceExecutionResult(Result.NO_ACTION, msg);
}
}
--
Jaime Hablutzel - +51 994690880
|
|
From: Jaime H. <hab...@gm...> - 2019-07-25 23:36:45
|
Since r32384, around 20k queries to the DB are being produced from
PublisherQueueSessionBean#plainFifoTryAlwaysLimit100EntriesOrderByTimeCreated
if it succeds at least once in publishing qeued data.
Below is a patch where it is worth noting that the 20K limit has been
removed as it doesn't make sense anymore cause the transactions are
currently circumscribed to the chunks of 100 records processed with each
call to PublisherQueueSessionBean#doChunk:
---
modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionLocal.java
(revision 32884)
+++
modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionLocal.java
(date 1564089225333)
@@ -102,12 +102,10 @@
/**
* Intended for use from PublishQueueProcessWorker.
*
- * Publishing algorithm that is a plain fifo queue, but limited to
selecting entries to republish at 100 records at a time. It will select
from the database for this particular publisher id, and process
+ * Publishing algorithm that is a plain fifo queue, but limited to
selecting entries to publish at 100 records at a time. It will select from
the database for this particular publisher, and process
* the record that is returned one by one. The records are ordered by
date, descending so the oldest record is returned first.
* Publishing is tried every time for every record returned, with no
limit.
* Repeat this process as long as we actually manage to publish
something this is because when publishing starts to work we want to publish
everything in one go, if possible.
- * However we don't want to publish more than 20000 certificates each
time, because we want to commit to the database some time as well.
- * Now, the OCSP publisher uses a non-transactional data source so it
commits every time so...
*/
PublishingResult
plainFifoTryAlwaysLimit100EntriesOrderByTimeCreated(AuthenticationToken
admin, BasePublisher publisher);
---
modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublishingResult.java
(revision 32884)
+++
modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublishingResult.java
(date 1564088515443)
@@ -18,8 +18,7 @@
/**
* Represents a return type for publishing operations. Successes and
failures are stored in sets containing the fingerprints of what's being
stored
- * due to the fact that the publisher will by default retry 20,000 times
on a failed attempt, so the "failures" will be per attempted certificate
- * and not all attempts.
+ * where the "failures" will be per attempted certificate and not all
attempts.
*
* @version $Id: PublishingResult.java 32392 2019-05-22 18:45:21Z
mikekushner $
*
---
modules/ejbca-ejb/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionBean.java
(revision 32884)
+++
modules/ejbca-ejb/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionBean.java
(date 1564093297113)
@@ -278,12 +278,11 @@
PublishingResult result = new PublishingResult();
// Repeat this process as long as we actually manage to publish
something
// this is because when publishing starts to work we want to
publish everything in one go, if possible.
- // However we don't want to publish more than 20000 certificates
each time, because we want to commit to the database some time as well.
- int totalcount = 0;
+ PublishingResult chunkResult;
do {
- result.append(publisherQueueSession.doChunk(admin, publisher));
- totalcount += result.getSuccesses();
- } while ((result.getSuccesses() > 0) && (totalcount < 20000));
+ chunkResult = publisherQueueSession.doChunk(admin, publisher);
+ result.append(chunkResult);
+ } while (chunkResult.getSuccesses() > 0);
return result;
}
@@ -296,7 +295,6 @@
/**
* @param admin
- * @param publisherId
* @param publisher
*
* @return how many publishing operations that succeeded and failed */
--
Jaime Hablutzel - +51 994690880
|
|
From: Jaime H. <hab...@gm...> - 2019-07-25 22:59:10
|
If a publisher with the following settings is configured:
- *No direct publishing, only use queue:* Disabled
- *Keep successfully published items in database:* Enabled
- *Use queue for CRLs:* Enabled
And it publishes a CRL successfully, the CRL will be kept in the publisher
queue with STATUS_PENDING anyway and something like a Publish Queue Process
Service will be required to process it again and mark it with STATUS_SUCCESS,
resulting in the CRL published twice, instead of only once.
A simple patch follows:
---
modules/ejbca-ejb/src/org/ejbca/core/ejb/ca/publisher/PublisherSessionBean.java
(revision 32884)
+++
modules/ejbca-ejb/src/org/ejbca/core/ejb/ca/publisher/PublisherSessionBean.java
(date 1563994959367)
@@ -309,7 +309,7 @@
pqvd.setUserDN(issuerDn);
String fp = CertTools.getFingerprintAsString(incrl);
try {
- publisherQueueSession.addQueueData(id,
PublisherConst.PUBLISH_TYPE_CRL, fp, pqvd, PublisherConst.STATUS_PENDING);
+ publisherQueueSession.addQueueData(id,
PublisherConst.PUBLISH_TYPE_CRL, fp, pqvd, publishStatus);
String msg =
intres.getLocalizedMessage("publisher.storequeue", name, fp, "CRL");
log.info(msg);
} catch (CreateException e) {
--
Jaime Hablutzel - +51 994690880
|
|
From: Tomas G. <to...@pr...> - 2019-07-23 10:55:41
|
Thanks Jaime, I think you are completely right. I've forwarded it to the dev team. Cheers, Tomas On 2019-07-20 06:53, Jaime Hablutzel wrote: > The following changes were introduced in the source code in relation > with https://jira.primekey.se/browse/ECA-7912: > > _modules/build-properties.xml_ > > <condition property="variant.ra.enabled"> > <available > file="${mod.peerconnector.path}/src-ra/org/ejbca/peerconnector/ra/RaMasterApiPeerImpl.java"/> > </condition> > > _modules/build.xml_ > > <target name="ra-gui" *if="${variant.ra.enabled}"* > depends="ejbca-common-web, ejbca-ejb" description="Build the EJBCA RA > GUI WAR"> > <ant antfile="${mod.ra-gui.path}/build.xml" target="build" > inheritall="true" inheritrefs="true"/> > </target> > > Which, given that the 'peerconnector' module is not available in the CE > source code, avoid the build of the 'ra-gui' module even when its own > source code is available at modules/ra-gui. > > This isn't the expected behaviour, right?. > > Regards. > > -- > Jaime Hablutzel - +51 994690880 > > > _______________________________________________ > Ejbca-develop mailing list > Ejb...@li... > https://lists.sourceforge.net/lists/listinfo/ejbca-develop > |
|
From: Tomas G. <to...@pr...> - 2019-07-23 09:26:53
|
Hi Jaime, The requirements were developed within CAB Forum, and CABF now has taken over/back the code signing certificate minimum requirements document (in addition to the EV guidelines currently in CABF). A ballot (see link) in the Code Signing WG in CABF was just completed adopting the "old" version, so they can start working on a new version. New updated versions of the code signing certificate minimum requirements will be released by CAB Forum. Therefore I will leave the text as is. I believe the ballot has passed now so it is actually a CAB Forum document now, albeit originally published in CA Security Council. https://cabforum.org/pipermail/cscwg-public/2019-April/000009.html Regards, Tomas On 2019-07-22 01:17, Jaime Hablutzel wrote: > The following from > https://download.primekey.com/docs/EJBCA-Enterprise/6_15_2/CA_Fields.html#src-23859254_id-.CAFieldsv6.15.0-KeepExpiredCertificatesonCRLKeep_Expired_Certificates_on_CRL: > > For example *CABForum* specifies this in their 'Minimum Requirements > for the Issuance and Management of Publicly-Trusted Code Signing > Certificates', version 1.1, section 13.2.1. > > > Should be modified to: > > For example, *the CA Security Council* specifies this in their > 'Minimum Requirements for the Issuance and Management of > Publicly-Trusted Code Signing Certificates', version 1.1, section > 13.2.1. > > > -- > Jaime Hablutzel - +51 994690880 > > > _______________________________________________ > Ejbca-develop mailing list > Ejb...@li... > https://lists.sourceforge.net/lists/listinfo/ejbca-develop > |
|
From: <oh...@ya...> - 2019-07-22 16:47:34
|
Hi,
I had already changed that and started a new run before I saw your email, but then I have left for a vacation, and I don't have access to our system until I get back.
I will post when I check it after I get back home.
Jim
On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
tool itself.
-Xmx4096m -Xms4096m
i,e.
exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
Regards,
Tomas
On 2019-07-21 17:19, oh...@ya... wrote:
> Hi,
>
> The import processing crashed :(....
>
> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
> missing in the database
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
> at
> org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>
>
> It got through 242720 entries.
>
> Jim
>
> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I built the new class and JAR and am testing. This looks better. It's
> not quite the rate that you are seeing but it's much better than what I
> was seeing before.
>
> So now it looks like I am getting about 1073 per minute, which is about
> 17 per second. I added some text to the class before I built it (not a
> lot, just some additional strings so I could verify I was using the
> modified class), so I know for sure that I am using the modified Java class.
>
> So anyway, it looks like we are down to about 15 hours to import that
> one CRL now :) ...
>
> Jim
>
>
>
> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I was doing a diff/fc file compare between the one you attached, and the
> last one I had that I used before, and it seems like there is a
> difference between those. Is the code that you just attached different
> than the patch you gave me before? Here's the file compare output (the
> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>
> Comparing files
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> and CAIMPORTCRLCOMMAND.JAVA
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> ***** CAIMPORTCRLCOMMAND.JAVA
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> *****
>
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>
> private KeyPair getStaticRSAKeyPair() {
> // A switch to use different keys depending on the sigAlg so we
> can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
> return keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException parsing hard
> coded presign key. This should never happen: ", e);
> }
> }
> ***** CAIMPORTCRLCOMMAND.JAVA
>
> private static KeyPair staticKp = null;
> private KeyPair getStaticRSAKeyPair() {
> if (staticKp == null) {
> synchronized (this) {
> if (staticKp == null) {
> // A switch to use different keys depending on the
> sigAlg so we can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair)
> pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
> ME);
> staticKp = keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException
> parsing hard coded presign key. This should never happen:
> ", e);
> }
> }
> }
> }
> return staticKp;
> }
> *****
>
>
>
> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> AACK! Yes, I forgot all about that and just used the vanilla software
> :(! Now, if I can remember how to do that patch, I will try it :(...
>
> Thanks,
> Jim
>
> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> Did you forget to patch the java file? The top output suggest you did.
> Attached the latest patched file that I used for the import.
>
> Regards,
> Tomas
>
> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
>> just started testing with that same CRL, and unfortunately, it is not
>> doing very well... actually even slower than on my earlier dev machine
>> under VBox :(!!
>>
>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
>> CPUs. I converted the tables to InnoDB and also ran the create index
>> SQL script.
>>
>> It looks like the import is only processing about 5000 entries per hour.
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: Jaime H. <hab...@gm...> - 2019-07-21 23:17:47
|
The following from https://download.primekey.com/docs/EJBCA-Enterprise/6_15_2/CA_Fields.html#src-23859254_id-.CAFieldsv6.15.0-KeepExpiredCertificatesonCRLKeep_Expired_Certificates_on_CRL : For example *CABForum* specifies this in their 'Minimum Requirements for > the Issuance and Management of Publicly-Trusted Code Signing Certificates', > version 1.1, section 13.2.1. Should be modified to: For example, *the CA Security Council* specifies this in their 'Minimum > Requirements for the Issuance and Management of Publicly-Trusted Code > Signing Certificates', version 1.1, section 13.2.1. -- Jaime Hablutzel - +51 994690880 |
|
From: Jaime H. <hab...@gm...> - 2019-07-21 20:08:00
|
Are there any other methods available for querying the audit log different than going through the Admin GUI's "Supervision Functions > Audit Log"?, maybe a CLI tool or a WS API?. I'm especially interested in a method that would allow to filter log entries based on some text in the details message too, for example, to identify CAA validation events. -- Jaime Hablutzel - +51 994690880 |
|
From: Tomas G. <to...@pr...> - 2019-07-21 17:32:37
|
I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
tool itself.
-Xmx4096m -Xms4096m
i,e.
exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
Regards,
Tomas
On 2019-07-21 17:19, oh...@ya... wrote:
> Hi,
>
> The import processing crashed :(....
>
> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
> missing in the database
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
> at
> org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
> at
> org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
> at
> org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
> at
> org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>
>
> It got through 242720 entries.
>
> Jim
>
> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I built the new class and JAR and am testing. This looks better. It's
> not quite the rate that you are seeing but it's much better than what I
> was seeing before.
>
> So now it looks like I am getting about 1073 per minute, which is about
> 17 per second. I added some text to the class before I built it (not a
> lot, just some additional strings so I could verify I was using the
> modified class), so I know for sure that I am using the modified Java class.
>
> So anyway, it looks like we are down to about 15 hours to import that
> one CRL now :) ...
>
> Jim
>
>
>
> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I was doing a diff/fc file compare between the one you attached, and the
> last one I had that I used before, and it seems like there is a
> difference between those. Is the code that you just attached different
> than the patch you gave me before? Here's the file compare output (the
> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>
> Comparing files
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> and CAIMPORTCRLCOMMAND.JAVA
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> ***** CAIMPORTCRLCOMMAND.JAVA
> final EndEntityInformation
> missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
>
> EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
> missing_user_name);
> *****
>
> *****
> CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>
> private KeyPair getStaticRSAKeyPair() {
> // A switch to use different keys depending on the sigAlg so we
> can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
> return keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException parsing hard
> coded presign key. This should never happen: ", e);
> }
> }
> ***** CAIMPORTCRLCOMMAND.JAVA
>
> private static KeyPair staticKp = null;
> private KeyPair getStaticRSAKeyPair() {
> if (staticKp == null) {
> synchronized (this) {
> if (staticKp == null) {
> // A switch to use different keys depending on the
> sigAlg so we can sign using the CAs signature algorithm
> final StringReader reader = new
> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
> try (PEMParser pemParser = new PEMParser(reader)) {
> PEMKeyPair pemKeyPair = (PEMKeyPair)
> pemParser.readObject();
> JcaPEMKeyConverter keyConverter = new
> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
> ME);
> staticKp = keyConverter.getKeyPair(pemKeyPair);
> } catch (IOException e) {
> throw new IllegalStateException("IOException
> parsing hard coded presign key. This should never happen:
> ", e);
> }
> }
> }
> }
> return staticKp;
> }
> *****
>
>
>
> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> AACK! Yes, I forgot all about that and just used the vanilla software
> :(! Now, if I can remember how to do that patch, I will try it :(...
>
> Thanks,
> Jim
>
> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> Did you forget to patch the java file? The top output suggest you did.
> Attached the latest patched file that I used for the import.
>
> Regards,
> Tomas
>
> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
>> just started testing with that same CRL, and unfortunately, it is not
>> doing very well... actually even slower than on my earlier dev machine
>> under VBox :(!!
>>
>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
>> CPUs. I converted the tables to InnoDB and also ran the create index
>> SQL script.
>>
>> It looks like the import is only processing about 5000 entries per hour.
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-21 15:19:40
|
Hi,
The import processing crashed :(....
+++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED' missing in the database
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
at org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
at org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
at org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
at org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
at org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
at org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
at org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
at org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
at org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
It got through 242720 entries.
Jim
On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I built the new class and JAR and am testing. This looks better. It's not quite the rate that you are seeing but it's much better than what I was seeing before.
So now it looks like I am getting about 1073 per minute, which is about 17 per second. I added some text to the class before I built it (not a lot, just some additional strings so I could verify I was using the modified class), so I know for sure that I am using the modified Java class.
So anyway, it looks like we are down to about 15 hours to import that one CRL now :) ...
Jim
On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I was doing a diff/fc file compare between the one you attached, and the last one I had that I used before, and it seems like there is a difference between those. Is the code that you just attached different than the patch you gave me before? Here's the file compare output (the "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
Comparing files CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20 and CAIMPORTCRLCOMMAND.JAVA
***** CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
final EndEntityInformation missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(), missing_user_name);
***** CAIMPORTCRLCOMMAND.JAVA
final EndEntityInformation missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(), missing_user_name);
*****
***** CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
private KeyPair getStaticRSAKeyPair() {
// A switch to use different keys depending on the sigAlg so we can sign using the CAs signature algorithm
final StringReader reader = new StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
try (PEMParser pemParser = new PEMParser(reader)) {
PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
JcaPEMKeyConverter keyConverter = new JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
return keyConverter.getKeyPair(pemKeyPair);
} catch (IOException e) {
throw new IllegalStateException("IOException parsing hard coded presign key. This should never happen: ", e);
}
}
***** CAIMPORTCRLCOMMAND.JAVA
private static KeyPair staticKp = null;
private KeyPair getStaticRSAKeyPair() {
if (staticKp == null) {
synchronized (this) {
if (staticKp == null) {
// A switch to use different keys depending on the sigAlg so we can sign using the CAs signature algorithm
final StringReader reader = new StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
try (PEMParser pemParser = new PEMParser(reader)) {
PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
JcaPEMKeyConverter keyConverter = new JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
ME);
staticKp = keyConverter.getKeyPair(pemKeyPair);
} catch (IOException e) {
throw new IllegalStateException("IOException parsing hard coded presign key. This should never happen:
", e);
}
}
}
}
return staticKp;
}
*****
On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
AACK! Yes, I forgot all about that and just used the vanilla software :(! Now, if I can remember how to do that patch, I will try it :(...
Thanks,Jim
On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Did you forget to patch the java file? The top output suggest you did.
Attached the latest patched file that I used for the import.
Regards,
Tomas
On 2019-07-20 17:45, oh...@ya... wrote:
> Hi,
>
> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
> just started testing with that same CRL, and unfortunately, it is not
> doing very well... actually even slower than on my earlier dev machine
> under VBox :(!!
>
> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
> CPUs. I converted the tables to InnoDB and also ran the create index
> SQL script.
>
> It looks like the import is only processing about 5000 entries per hour.
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-21 01:52:55
|
Hi,
I built the new class and JAR and am testing. This looks better. It's not quite the rate that you are seeing but it's much better than what I was seeing before.
So now it looks like I am getting about 1073 per minute, which is about 17 per second. I added some text to the class before I built it (not a lot, just some additional strings so I could verify I was using the modified class), so I know for sure that I am using the modified Java class.
So anyway, it looks like we are down to about 15 hours to import that one CRL now :) ...
Jim
On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I was doing a diff/fc file compare between the one you attached, and the last one I had that I used before, and it seems like there is a difference between those. Is the code that you just attached different than the patch you gave me before? Here's the file compare output (the "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
Comparing files CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20 and CAIMPORTCRLCOMMAND.JAVA
***** CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
final EndEntityInformation missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(), missing_user_name);
***** CAIMPORTCRLCOMMAND.JAVA
final EndEntityInformation missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(), missing_user_name);
*****
***** CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
private KeyPair getStaticRSAKeyPair() {
// A switch to use different keys depending on the sigAlg so we can sign using the CAs signature algorithm
final StringReader reader = new StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
try (PEMParser pemParser = new PEMParser(reader)) {
PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
JcaPEMKeyConverter keyConverter = new JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
return keyConverter.getKeyPair(pemKeyPair);
} catch (IOException e) {
throw new IllegalStateException("IOException parsing hard coded presign key. This should never happen: ", e);
}
}
***** CAIMPORTCRLCOMMAND.JAVA
private static KeyPair staticKp = null;
private KeyPair getStaticRSAKeyPair() {
if (staticKp == null) {
synchronized (this) {
if (staticKp == null) {
// A switch to use different keys depending on the sigAlg so we can sign using the CAs signature algorithm
final StringReader reader = new StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
try (PEMParser pemParser = new PEMParser(reader)) {
PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
JcaPEMKeyConverter keyConverter = new JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
ME);
staticKp = keyConverter.getKeyPair(pemKeyPair);
} catch (IOException e) {
throw new IllegalStateException("IOException parsing hard coded presign key. This should never happen:
", e);
}
}
}
}
return staticKp;
}
*****
On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
AACK! Yes, I forgot all about that and just used the vanilla software :(! Now, if I can remember how to do that patch, I will try it :(...
Thanks,Jim
On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Did you forget to patch the java file? The top output suggest you did.
Attached the latest patched file that I used for the import.
Regards,
Tomas
On 2019-07-20 17:45, oh...@ya... wrote:
> Hi,
>
> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
> just started testing with that same CRL, and unfortunately, it is not
> doing very well... actually even slower than on my earlier dev machine
> under VBox :(!!
>
> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
> CPUs. I converted the tables to InnoDB and also ran the create index
> SQL script.
>
> It looks like the import is only processing about 5000 entries per hour.
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-20 21:51:30
|
Hi,
I was doing a diff/fc file compare between the one you attached, and the last one I had that I used before, and it seems like there is a difference between those. Is the code that you just attached different than the patch you gave me before? Here's the file compare output (the "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
Comparing files CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20 and CAIMPORTCRLCOMMAND.JAVA
***** CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
final EndEntityInformation missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(), missing_user_name);
***** CAIMPORTCRLCOMMAND.JAVA
final EndEntityInformation missingUserEndEntityInformation = EjbRemoteHelper.INSTANCE.getRemoteSession(
EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(), missing_user_name);
*****
***** CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
private KeyPair getStaticRSAKeyPair() {
// A switch to use different keys depending on the sigAlg so we can sign using the CAs signature algorithm
final StringReader reader = new StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
try (PEMParser pemParser = new PEMParser(reader)) {
PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
JcaPEMKeyConverter keyConverter = new JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
return keyConverter.getKeyPair(pemKeyPair);
} catch (IOException e) {
throw new IllegalStateException("IOException parsing hard coded presign key. This should never happen: ", e);
}
}
***** CAIMPORTCRLCOMMAND.JAVA
private static KeyPair staticKp = null;
private KeyPair getStaticRSAKeyPair() {
if (staticKp == null) {
synchronized (this) {
if (staticKp == null) {
// A switch to use different keys depending on the sigAlg so we can sign using the CAs signature algorithm
final StringReader reader = new StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
try (PEMParser pemParser = new PEMParser(reader)) {
PEMKeyPair pemKeyPair = (PEMKeyPair) pemParser.readObject();
JcaPEMKeyConverter keyConverter = new JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
ME);
staticKp = keyConverter.getKeyPair(pemKeyPair);
} catch (IOException e) {
throw new IllegalStateException("IOException parsing hard coded presign key. This should never happen:
", e);
}
}
}
}
return staticKp;
}
*****
On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
AACK! Yes, I forgot all about that and just used the vanilla software :(! Now, if I can remember how to do that patch, I will try it :(...
Thanks,Jim
On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Did you forget to patch the java file? The top output suggest you did.
Attached the latest patched file that I used for the import.
Regards,
Tomas
On 2019-07-20 17:45, oh...@ya... wrote:
> Hi,
>
> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
> just started testing with that same CRL, and unfortunately, it is not
> doing very well... actually even slower than on my earlier dev machine
> under VBox :(!!
>
> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
> CPUs. I converted the tables to InnoDB and also ran the create index
> SQL script.
>
> It looks like the import is only processing about 5000 entries per hour.
|
|
From: <oh...@ya...> - 2019-07-20 21:32:11
|
AACK! Yes, I forgot all about that and just used the vanilla software :(! Now, if I can remember how to do that patch, I will try it :(...
Thanks,Jim
On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Did you forget to patch the java file? The top output suggest you did.
Attached the latest patched file that I used for the import.
Regards,
Tomas
On 2019-07-20 17:45, oh...@ya... wrote:
> Hi,
>
> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
> just started testing with that same CRL, and unfortunately, it is not
> doing very well... actually even slower than on my earlier dev machine
> under VBox :(!!
>
> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
> CPUs. I converted the tables to InnoDB and also ran the create index
> SQL script.
>
> It looks like the import is only processing about 5000 entries per hour.
>
> Here's the top:
>
> top - 15:43:29 up 17:25, 7 users, load average: 1.70, 1.60, 0.98
> Tasks: 127 total, 1 running, 126 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 75.4 us, 0.5 sy, 0.0 ni, 23.2 id, 0.5 wa, 0.0 hi, 0.2 si,
> 0.2 st
> KiB Mem : 3880256 total, 103696 free, 2799736 used, 976824 buff/cache
> KiB Swap: 0 total, 0 free, 0 used. 788552 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 12115 root 20 0 3529260 1.074g 15368 S 143.0 29.0 18:35.88 java
> 7326 root 20 0 3290332 1.038g 21880 S 7.7 28.1 4:02.67 java
> 1384 mysql 20 0 1580240 231652 8260 S 3.0 6.0 0:41.18 mysqld
> 264 root 20 0 0 0 0 S 0.3 0.0 0:12.29
> xfsaild/xvda2
> 1 root 20 0 125572 3952 2572 S 0.0 0.1 0:03.04 systemd
> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
> 3 root 20 0 0 0 0 S 0.0 0.0 0:00.03
> ksoftirqd/0
> 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00
> kworker/0:0H
>
>
> Tomas, you had mentioned that there were some optimizations for MySQL?
> Could those be the reason that your import is running so much faster
> than when I run the import?
>
> Thanks,
> Jim
>
>
>
> On Friday, July 19, 2019, 8:18:13 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> I'm trying to stand up a new EJBCA instance on AWS, to try to see if I
> can replicate the import speed you are seeing, but ran into an error
> (see the new thread I just posted).
>
> Thanks,
> Jim
>
> On Friday, July 19, 2019, 6:39:52 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Hi,
>
> I just finished importing your CRL.
> 996549 records in approx. 12 hours.
>
> This is on my laptop, starting with 400K records in the database already
> (i.e ending with 1408370). There is no slowdown as number of records grow).
>
> It failed in the end storing the actual CRL due to a limitation on
> packet size in my MySQL config, but that could be changed easily in
> MySQL (MariaDB) config. Actually the CRL itself is not needed in the
> database in order to answer OCSP queries, so there should optimally be a
> switch to not import the CRL itself, only the revocation records (to
> save a bit on database storage).
>
> The import speed can with some rather simple development be made orders
> of magnitude faster (you don't want a roundtrip for every record for
> example).
>
> Cheers,
> Tomas
>
>
> On 2019-07-19 18:33, oh...@ya... <mailto:oh...@ya...> wrote:
>> Started testing with the VM on an SSD and now I am getting ~500 - 550
>> per minute, or about 30K per hour. That is faster than before, but
>> still way slower than the 100K per hour you mentioned.
>>
>> Jim
>>
>> On Friday, July 19, 2019, 2:56:11 PM UTC, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> I don't have any SSDs on the machine that I use as my server... only
>> 7200RPM hard drives, except for the OS drive, which is SSD. I can try
>> to add an SSD (I have lots of SSDs, just not on that machine).
>>
>> Jim
>>
>> On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>>
>> I have now imported 600K of the entries form the CRL. Still running at
>> 25 entries per second.
>>
>> The logging does not normally affect performance, logging is a normal
>> part. Not running an SSD for the database typically hugely affect DB
>> performance. It's possible to tune logging a lot, For example disable
>> logging to console and only log to the file (if your console is really
>> slow). It's also possible to set the database to not synchronize on disk
>> for every write, during database restore and certain operations for
> example.
>>
>> I actually have a worse performing log config than you. I have info log
>> to the console and DEBUG log to file, so it's logging a lot. I have a
>> SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
>> top). But it's still a three year old laptop, so not the fastest you can
>> get.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-19 15:36, oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>> wrote:
>>> Hi,
>>>
>>> I got the index done and now am running the CRL import, which has been
>>> running about 1/2 hour now, but so far, only about 5500 entries
>>> imported, so it doesn't look like it hugely faster than before?
>>>
>>> I was wondering... on the server side, I think outputting messages like
>>> the following to stdout for every entry in the CRL:
>>>
>>> 09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
>>> 09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>>> 09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
>>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
>>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>>> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
>>> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
>>> 'CN=XXXXXX', serialNo=8A296.
>>> 09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>>> 09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-2) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>>> 09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-2) 2019-07-19
>>>
>>
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
>>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
>>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>>> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>>>
>>>
>>> Is it possible that that logging is slowing the CRL import?
>>>
>>> How is your logging setup (log level) compared to what mine is?
>>>
>>> Other that, it just may be that my system is much slower than your
>>> laptop :)??
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
>>> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
>>>
>>>
>>> Ah, eap 7.2 corresponds to wildly 14 as base.
>>>
>>> Yes, run that to convert, then you should be able to create the index.
>>>
>>> Cheers,
>>> Tomas
>>>
>>>
>>> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
>>> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>>>
>>> Hi Tomas,
>>>
>>> No, I am not using Wildfly, but rather JBOSS... specifically
>>> "jboss-eap-7.2".
>>>
>>> So I restored my snapshot from just after I had deleted all the rows
>>> in CertificateData that were created before.
>>>
>>> And it looks like all the tables in the "ejbca" database are MyISAM.
>>>
>>> So I found a one-liner that will convert them all to InnoDB:
>>>
>>> mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
>>> | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql
>>> -u root -p ejbca
>>>
>>> Should I run that and convert all the tables in the "ejbca" to
>> InnoDB???
>>>
>>>
>>> And then try to run the create indexes SQL script?
>>>
>>>
>>> Thanks for all your help!
>>>
>>> Jim
>>>
>>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
> <mailto:Ejb...@li...>
>> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: Tomas G. <to...@pr...> - 2019-07-20 20:35:14
|
Did you forget to patch the java file? The top output suggest you did.
Attached the latest patched file that I used for the import.
Regards,
Tomas
On 2019-07-20 17:45, oh...@ya... wrote:
> Hi,
>
> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I
> just started testing with that same CRL, and unfortunately, it is not
> doing very well... actually even slower than on my earlier dev machine
> under VBox :(!!
>
> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2
> CPUs. I converted the tables to InnoDB and also ran the create index
> SQL script.
>
> It looks like the import is only processing about 5000 entries per hour.
>
> Here's the top:
>
> top - 15:43:29 up 17:25, 7 users, load average: 1.70, 1.60, 0.98
> Tasks: 127 total, 1 running, 126 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 75.4 us, 0.5 sy, 0.0 ni, 23.2 id, 0.5 wa, 0.0 hi, 0.2 si,
> 0.2 st
> KiB Mem : 3880256 total, 103696 free, 2799736 used, 976824 buff/cache
> KiB Swap: 0 total, 0 free, 0 used. 788552 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 12115 root 20 0 3529260 1.074g 15368 S 143.0 29.0 18:35.88 java
> 7326 root 20 0 3290332 1.038g 21880 S 7.7 28.1 4:02.67 java
> 1384 mysql 20 0 1580240 231652 8260 S 3.0 6.0 0:41.18 mysqld
> 264 root 20 0 0 0 0 S 0.3 0.0 0:12.29
> xfsaild/xvda2
> 1 root 20 0 125572 3952 2572 S 0.0 0.1 0:03.04 systemd
> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
> 3 root 20 0 0 0 0 S 0.0 0.0 0:00.03
> ksoftirqd/0
> 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00
> kworker/0:0H
>
>
> Tomas, you had mentioned that there were some optimizations for MySQL?
> Could those be the reason that your import is running so much faster
> than when I run the import?
>
> Thanks,
> Jim
>
>
>
> On Friday, July 19, 2019, 8:18:13 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> I'm trying to stand up a new EJBCA instance on AWS, to try to see if I
> can replicate the import speed you are seeing, but ran into an error
> (see the new thread I just posted).
>
> Thanks,
> Jim
>
> On Friday, July 19, 2019, 6:39:52 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Hi,
>
> I just finished importing your CRL.
> 996549 records in approx. 12 hours.
>
> This is on my laptop, starting with 400K records in the database already
> (i.e ending with 1408370). There is no slowdown as number of records grow).
>
> It failed in the end storing the actual CRL due to a limitation on
> packet size in my MySQL config, but that could be changed easily in
> MySQL (MariaDB) config. Actually the CRL itself is not needed in the
> database in order to answer OCSP queries, so there should optimally be a
> switch to not import the CRL itself, only the revocation records (to
> save a bit on database storage).
>
> The import speed can with some rather simple development be made orders
> of magnitude faster (you don't want a roundtrip for every record for
> example).
>
> Cheers,
> Tomas
>
>
> On 2019-07-19 18:33, oh...@ya... <mailto:oh...@ya...> wrote:
>> Started testing with the VM on an SSD and now I am getting ~500 - 550
>> per minute, or about 30K per hour. That is faster than before, but
>> still way slower than the 100K per hour you mentioned.
>>
>> Jim
>>
>> On Friday, July 19, 2019, 2:56:11 PM UTC, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> I don't have any SSDs on the machine that I use as my server... only
>> 7200RPM hard drives, except for the OS drive, which is SSD. I can try
>> to add an SSD (I have lots of SSDs, just not on that machine).
>>
>> Jim
>>
>> On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>>
>> I have now imported 600K of the entries form the CRL. Still running at
>> 25 entries per second.
>>
>> The logging does not normally affect performance, logging is a normal
>> part. Not running an SSD for the database typically hugely affect DB
>> performance. It's possible to tune logging a lot, For example disable
>> logging to console and only log to the file (if your console is really
>> slow). It's also possible to set the database to not synchronize on disk
>> for every write, during database restore and certain operations for
> example.
>>
>> I actually have a worse performing log config than you. I have info log
>> to the console and DEBUG log to file, so it's logging a lot. I have a
>> SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
>> top). But it's still a three year old laptop, so not the fastest you can
>> get.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-19 15:36, oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>> wrote:
>>> Hi,
>>>
>>> I got the index done and now am running the CRL import, which has been
>>> running about 1/2 hour now, but so far, only about 5500 entries
>>> imported, so it doesn't look like it hugely faster than before?
>>>
>>> I was wondering... on the server side, I think outputting messages like
>>> the following to stdout for every entry in the CRL:
>>>
>>> 09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
>>> 09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>>> 09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
>>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
>>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>>> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
>>> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
>>> 'CN=XXXXXX', serialNo=8A296.
>>> 09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-1) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>>> 09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-2) 2019-07-19
>>>
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>>> 09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>>> task-2) 2019-07-19
>>>
>>
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
>>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
>>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>>> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>>>
>>>
>>> Is it possible that that logging is slowing the CRL import?
>>>
>>> How is your logging setup (log level) compared to what mine is?
>>>
>>> Other that, it just may be that my system is much slower than your
>>> laptop :)??
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
>>> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
>>>
>>>
>>> Ah, eap 7.2 corresponds to wildly 14 as base.
>>>
>>> Yes, run that to convert, then you should be able to create the index.
>>>
>>> Cheers,
>>> Tomas
>>>
>>>
>>> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
>>> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>>>
>>> Hi Tomas,
>>>
>>> No, I am not using Wildfly, but rather JBOSS... specifically
>>> "jboss-eap-7.2".
>>>
>>> So I restored my snapshot from just after I had deleted all the rows
>>> in CertificateData that were created before.
>>>
>>> And it looks like all the tables in the "ejbca" database are MyISAM.
>>>
>>> So I found a one-liner that will convert them all to InnoDB:
>>>
>>> mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
>>> | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql
>>> -u root -p ejbca
>>>
>>> Should I run that and convert all the tables in the "ejbca" to
>> InnoDB???
>>>
>>>
>>> And then try to run the create indexes SQL script?
>>>
>>>
>>> Thanks for all your help!
>>>
>>> Jim
>>>
>>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
> <mailto:Ejb...@li...>
>> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-20 15:45:28
|
Hi,
FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and I just started testing with that same CRL, and unfortunately, it is not doing very well... actually even slower than on my earlier dev machine under VBox :(!!
The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and 2 CPUs. I converted the tables to InnoDB and also ran the create index SQL script.
It looks like the import is only processing about 5000 entries per hour.
Here's the top:
top - 15:43:29 up 17:25, 7 users, load average: 1.70, 1.60, 0.98
Tasks: 127 total, 1 running, 126 sleeping, 0 stopped, 0 zombie
%Cpu(s): 75.4 us, 0.5 sy, 0.0 ni, 23.2 id, 0.5 wa, 0.0 hi, 0.2 si, 0.2 st
KiB Mem : 3880256 total, 103696 free, 2799736 used, 976824 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 788552 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12115 root 20 0 3529260 1.074g 15368 S 143.0 29.0 18:35.88 java
7326 root 20 0 3290332 1.038g 21880 S 7.7 28.1 4:02.67 java
1384 mysql 20 0 1580240 231652 8260 S 3.0 6.0 0:41.18 mysqld
264 root 20 0 0 0 0 S 0.3 0.0 0:12.29 xfsaild/xvda2
1 root 20 0 125572 3952 2572 S 0.0 0.1 0:03.04 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
Tomas, you had mentioned that there were some optimizations for MySQL? Could those be the reason that your import is running so much faster than when I run the import?
Thanks,Jim
On Friday, July 19, 2019, 8:18:13 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
I'm trying to stand up a new EJBCA instance on AWS, to try to see if I can replicate the import speed you are seeing, but ran into an error (see the new thread I just posted).
Thanks,Jim
On Friday, July 19, 2019, 6:39:52 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
I just finished importing your CRL.
996549 records in approx. 12 hours.
This is on my laptop, starting with 400K records in the database already
(i.e ending with 1408370). There is no slowdown as number of records grow).
It failed in the end storing the actual CRL due to a limitation on
packet size in my MySQL config, but that could be changed easily in
MySQL (MariaDB) config. Actually the CRL itself is not needed in the
database in order to answer OCSP queries, so there should optimally be a
switch to not import the CRL itself, only the revocation records (to
save a bit on database storage).
The import speed can with some rather simple development be made orders
of magnitude faster (you don't want a roundtrip for every record for
example).
Cheers,
Tomas
On 2019-07-19 18:33, oh...@ya... wrote:
> Started testing with the VM on an SSD and now I am getting ~500 - 550
> per minute, or about 30K per hour. That is faster than before, but
> still way slower than the 100K per hour you mentioned.
>
> Jim
>
> On Friday, July 19, 2019, 2:56:11 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I don't have any SSDs on the machine that I use as my server... only
> 7200RPM hard drives, except for the OS drive, which is SSD. I can try
> to add an SSD (I have lots of SSDs, just not on that machine).
>
> Jim
>
> On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> I have now imported 600K of the entries form the CRL. Still running at
> 25 entries per second.
>
> The logging does not normally affect performance, logging is a normal
> part. Not running an SSD for the database typically hugely affect DB
> performance. It's possible to tune logging a lot, For example disable
> logging to console and only log to the file (if your console is really
> slow). It's also possible to set the database to not synchronize on disk
> for every write, during database restore and certain operations for example.
>
> I actually have a worse performing log config than you. I have info log
> to the console and DEBUG log to file, so it's logging a lot. I have a
> SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
> top). But it's still a three year old laptop, so not the fastest you can
> get.
>
> Regards,
> Tomas
>
> On 2019-07-19 15:36, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> I got the index done and now am running the CRL import, which has been
>> running about 1/2 hour now, but so far, only about 5500 entries
>> imported, so it doesn't look like it hugely faster than before?
>>
>> I was wondering... on the server side, I think outputting messages like
>> the following to stdout for every entry in the CRL:
>>
>> 09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
>> 09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
>> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
>> 'CN=XXXXXX', serialNo=8A296.
>> 09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-2) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-2) 2019-07-19
>>
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>>
>>
>> Is it possible that that logging is slowing the CRL import?
>>
>> How is your logging setup (log level) compared to what mine is?
>>
>> Other that, it just may be that my system is much slower than your
>> laptop :)??
>>
>> Jim
>>
>>
>>
>>
>> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Ah, eap 7.2 corresponds to wildly 14 as base.
>>
>> Yes, run that to convert, then you should be able to create the index.
>>
>> Cheers,
>> Tomas
>>
>>
>> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi Tomas,
>>
>> No, I am not using Wildfly, but rather JBOSS... specifically
>> "jboss-eap-7.2".
>>
>> So I restored my snapshot from just after I had deleted all the rows
>> in CertificateData that were created before.
>>
>> And it looks like all the tables in the "ejbca" database are MyISAM.
>>
>> So I found a one-liner that will convert them all to InnoDB:
>>
>> mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
>> | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql
>> -u root -p ejbca
>>
>> Should I run that and convert all the tables in the "ejbca" to
> InnoDB???
>>
>>
>> And then try to run the create indexes SQL script?
>>
>>
>> Thanks for all your help!
>>
>> Jim
>>
>>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
106 messages has been excluded from this view by a project administrator.
