Menu
▾
▴
Re: [Ejbca-develop] EJBCA OCSP Responder does not appear to
|
From: Tomas G. <to...@pr...> - 2019-07-30 12:42:11
|
If it's a new machine, did you add the database indexes?
On July 30, 2019 1:42:00 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
> Hi,
>The machine I am using now is on AWS, a t2-medium, with 2 CPUs and 4GB
>RAM and 70GB drive, and RHEL7.4(?). I modified the ejbca.sh for
>"-Xms2g Xmx2g" to avoid the outofmemory error I got previously.
>
>It is still not finished now, after 18.5 hours. The import rate went
>from about ~75k/hour at the beginning to ~23.5k/hour now, and it is
>only processed ~435k entries so far, out of ~980k entries total in the
>CRL file. Maybe I should've picked a larger machine (probably more
>memory), but it may be too late for that now, since it's already been
>running for 18.5 hours.
>Jim
>
>On Tuesday, July 30, 2019, 7:47:33 AM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>Our team in California has also managed to import the CRL, running
>about 25 entries per second. If you need any helo, don't hesitate to
>reach out to them.
>
>In the longer run, with minor code changes, it's possible to make it
>orders of magnitude faster.
>
>Regards,
>Tomas
>
>
>On July 30, 2019 12:05:36 AM GMT+02:00, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>It is still running now, about 5 hours so far, and only about 228K
>entries.
>
>So far, processing rate has been between 45K per hour (now) and 75K per
>hour (earlier). I am guessing that it will take about 20 hours to
>import the whole CRL, if it doesn't blow up.
>
>Jim
>
>On Monday, July 29, 2019, 4:48:19 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
>From looking at the entries in CertificateData table, it looks like it
>was only able to import 130635 entries before the import process
>died/ended...
>Jim
>
>
>
>On Monday, July 29, 2019, 3:37:06 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>It looks like the import that I started before going on vacation failed
>for some reason (possibly the machine got shutdown by our automatic
>shutdowns... I am not sure).
>So I will start the import AGAIN today...
>Bottom line is that I am still not able to import the large CRL
>successfully so far....
>
>Jim
>
>
>On Monday, July 22, 2019, 4:47:50 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>I had already changed that and started a new run before I saw your
>email, but then I have left for a vacation, and I don't have access to
>our system until I get back.
>I will post when I check it after I get back home.
>Jim
>
>
>On Sunday, July 21, 2019, 5:32:31 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>
>I edited bin/ejbca.sh and added these parameters to use 4GB for the CLI
>tool itself.
>
> -Xmx4096m -Xms4096m
>
>i,e.
>
>exec "$JAVACMD" -Xmx4096m -Xms4096m -jar "$CLI_JAR" "$@"
>
>
>Regards,
>Tomas
>
>On 2019-07-21 17:19, oh...@ya... wrote:
>> Hi,
>>
>> The import processing crashed :(....
>>
>> +++++ V2.00 SMALLER PRIVATE KEY BY TOMAS +++++ Certificate '273BED'
>> missing in the database
>> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead
>limit
>> exceeded
>> at org.ejbca.util.crypto.BCrypt.initKey(BCrypt.java:547)
>> at org.ejbca.util.crypto.BCrypt.cryptRaw(BCrypt.java:635)
>> at org.ejbca.util.crypto.BCrypt.hashpw(BCrypt.java:700)
>> at
>>
>org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.generateSha1Hash(CliAuthenticationToken.java:102)
>> at
>>
>org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromHashedPassword(CliAuthenticationToken.java:168)
>> at
>>
>org.ejbca.core.ejb.authentication.cli.CliAuthenticationToken.setSha1HashFromCleartextPassword(CliAuthenticationToken.java:187)
>> at
>>
>org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.getAuthenticationToken(PasswordUsingCommandBase.java:246)
>> at
>>
>org.ejbca.ui.cli.ca.CaImportCRLCommand.execute(CaImportCRLCommand.java:179)
>> at
>>
>org.ejbca.ui.cli.infrastructure.command.PasswordUsingCommandBase.execute(PasswordUsingCommandBase.java:202)
>> at
>>
>org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:287)
>> at
>>
>org.ejbca.ui.cli.infrastructure.library.CommandLibrary$Branch.execute(CommandLibrary.java:297)
>> at
>>
>org.ejbca.ui.cli.infrastructure.library.CommandLibrary.findAndExecuteCommandFromParameters(CommandLibrary.java:78)
>> at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:33)
>>
>>
>> It got through 242720 entries.
>>
>> Jim
>>
>> On Sunday, July 21, 2019, 1:53:13 AM UTC, ohaya--- via Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> Hi,
>>
>> I built the new class and JAR and am testing. This looks better.
>It's
>> not quite the rate that you are seeing but it's much better than what
>I
>> was seeing before.
>>
>> So now it looks like I am getting about 1073 per minute, which is
>about
>> 17 per second. I added some text to the class before I built it (not
>a
>> lot, just some additional strings so I could verify I was using the
>> modified class), so I know for sure that I am using the modified Java
>class.
>>
>> So anyway, it looks like we are down to about 15 hours to import that
>> one CRL now :) ...
>>
>> Jim
>>
>>
>>
>> On Saturday, July 20, 2019, 9:51:43 PM UTC, ohaya--- via
>Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> Hi,
>>
>> I was doing a diff/fc file compare between the one you attached, and
>the
>> last one I had that I used before, and it seems like there is a
>> difference between those. Is the code that you just attached
>different
>> than the patch you gave me before? Here's the file compare output
>(the
>> "CAIMPORTCRLCOMMAND.JAVA" is the one you just attached):
>>
>> Comparing files
>>
>CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>> and CAIMPORTCRLCOMMAND.JAVA
>> *****
>>
>CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>> final EndEntityInformation
>> missingUserEndEntityInformation =
>EjbRemoteHelper.INSTANCE.getRemoteSession(
>>
>>
>>
>EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
>> missing_user_name);
>> ***** CAIMPORTCRLCOMMAND.JAVA
>> final EndEntityInformation
>> missingUserEndEntityInformation =
>EjbRemoteHelper.INSTANCE.getRemoteSession(
>>
>>
>EndEntityAccessSessionRemote.class).findUser(getAuthenticationToken(),
>> missing_user_name);
>> *****
>>
>> *****
>>
>CaImportCRLCommand.java-C-WORKING-PATCH-B4-TOMAS-GAVE-NEW-ONE-ON-2019-07-20
>>
>> private KeyPair getStaticRSAKeyPair() {
>> // A switch to use different keys depending on the sigAlg so
>we
>> can sign using the CAs signature algorithm
>> final StringReader reader = new
>> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
>> try (PEMParser pemParser = new PEMParser(reader)) {
>> PEMKeyPair pemKeyPair = (PEMKeyPair)
>pemParser.readObject();
>> JcaPEMKeyConverter keyConverter = new
>> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NAME);
>> return keyConverter.getKeyPair(pemKeyPair);
>> } catch (IOException e) {
>> throw new IllegalStateException("IOException parsing hard
>> coded presign key. This should never happen: ", e);
>> }
>> }
>> ***** CAIMPORTCRLCOMMAND.JAVA
>>
>> private static KeyPair staticKp = null;
>> private KeyPair getStaticRSAKeyPair() {
>> if (staticKp == null) {
>> synchronized (this) {
>> if (staticKp == null) {
>> // A switch to use different keys depending on
>the
>> sigAlg so we can sign using the CAs signature algorithm
>> final StringReader reader = new
>> StringReader(CaImportCRLCommand.PRESIGN_VALIDATION_KEY_RSA_PRIV);
>> try (PEMParser pemParser = new PEMParser(reader))
>{
>> PEMKeyPair pemKeyPair = (PEMKeyPair)
>> pemParser.readObject();
>> JcaPEMKeyConverter keyConverter = new
>> JcaPEMKeyConverter().setProvider(BouncyCastleProvider.PROVIDER_NA
>> ME);
>> staticKp =
>keyConverter.getKeyPair(pemKeyPair);
>> } catch (IOException e) {
>> throw new IllegalStateException("IOException
>> parsing hard coded presign key. This should never happen:
>> ", e);
>> }
>> }
>> }
>> }
>> return staticKp;
>> }
>> *****
>>
>>
>>
>> On Saturday, July 20, 2019, 9:32:26 PM UTC, ohaya--- via
>Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> AACK! Yes, I forgot all about that and just used the vanilla
>software
>> :(! Now, if I can remember how to do that patch, I will try it :(...
>>
>> Thanks,
>> Jim
>>
>> On Saturday, July 20, 2019, 8:35:09 PM UTC, Tomas Gustavsson
>> <to...@pr...> wrote:
>>
>>
>>
>> Did you forget to patch the java file? The top output suggest you
>did.
>> Attached the latest patched file that I used for the import.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-20 17:45, oh...@ya... <mailto:oh...@ya...> wrote:
>>> Hi,
>>>
>>> FYI, I got the EJBCA installed on a Redhat 7.4 instance on AWS, and
>I
>>> just started testing with that same CRL, and unfortunately, it is
>not
>>> doing very well... actually even slower than on my earlier dev
>machine
>>> under VBox :(!!
>>>
>>> The AWS machine is a t2.medium instance with 70GB disk, 4GB RAM, and
>2
>>> CPUs. I converted the tables to InnoDB and also ran the create
>index
>>> SQL script.
>>>
>>> It looks like the import is only processing about 5000 entries per
>hour.
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
|
