Menu
▾
▴
ejbca-develop — Development discussions
You can subscribe to this list here.
| 2001 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
(3) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2002 |
Jan
(3) |
Feb
(2) |
Mar
(8) |
Apr
(3) |
May
(6) |
Jun
(1) |
Jul
(15) |
Aug
(6) |
Sep
|
Oct
(10) |
Nov
(2) |
Dec
(4) |
| 2003 |
Jan
(1) |
Feb
(7) |
Mar
(3) |
Apr
(6) |
May
(7) |
Jun
(5) |
Jul
(5) |
Aug
(25) |
Sep
(14) |
Oct
(2) |
Nov
|
Dec
(2) |
| 2004 |
Jan
(7) |
Feb
(4) |
Mar
(12) |
Apr
(16) |
May
(43) |
Jun
(56) |
Jul
(43) |
Aug
(40) |
Sep
(66) |
Oct
(12) |
Nov
(26) |
Dec
(10) |
| 2005 |
Jan
(13) |
Feb
(33) |
Mar
(16) |
Apr
(7) |
May
(10) |
Jun
(34) |
Jul
(41) |
Aug
(8) |
Sep
(4) |
Oct
(32) |
Nov
(20) |
Dec
(25) |
| 2006 |
Jan
(30) |
Feb
(101) |
Mar
(5) |
Apr
(75) |
May
(74) |
Jun
(22) |
Jul
(6) |
Aug
(70) |
Sep
(19) |
Oct
(21) |
Nov
(31) |
Dec
(50) |
| 2007 |
Jan
(15) |
Feb
(20) |
Mar
(24) |
Apr
(33) |
May
(13) |
Jun
(18) |
Jul
(13) |
Aug
(7) |
Sep
(63) |
Oct
(68) |
Nov
(29) |
Dec
(68) |
| 2008 |
Jan
(30) |
Feb
(33) |
Mar
(30) |
Apr
(103) |
May
(78) |
Jun
(48) |
Jul
(72) |
Aug
(24) |
Sep
(62) |
Oct
(63) |
Nov
(70) |
Dec
(37) |
| 2009 |
Jan
(34) |
Feb
(35) |
Mar
(64) |
Apr
(34) |
May
(34) |
Jun
(58) |
Jul
(30) |
Aug
(30) |
Sep
(46) |
Oct
(52) |
Nov
(12) |
Dec
(23) |
| 2010 |
Jan
(121) |
Feb
(18) |
Mar
(53) |
Apr
(62) |
May
(62) |
Jun
(20) |
Jul
(33) |
Aug
(20) |
Sep
(36) |
Oct
(35) |
Nov
(44) |
Dec
(63) |
| 2011 |
Jan
(19) |
Feb
(32) |
Mar
(94) |
Apr
(41) |
May
(47) |
Jun
(25) |
Jul
(34) |
Aug
(20) |
Sep
(9) |
Oct
(41) |
Nov
(33) |
Dec
(24) |
| 2012 |
Jan
(12) |
Feb
(36) |
Mar
(48) |
Apr
(32) |
May
(20) |
Jun
(15) |
Jul
(32) |
Aug
(13) |
Sep
(33) |
Oct
(54) |
Nov
(25) |
Dec
(16) |
| 2013 |
Jan
(45) |
Feb
(39) |
Mar
(38) |
Apr
(50) |
May
(29) |
Jun
(30) |
Jul
(33) |
Aug
(12) |
Sep
(9) |
Oct
(25) |
Nov
(29) |
Dec
(20) |
| 2014 |
Jan
(25) |
Feb
(19) |
Mar
(16) |
Apr
(33) |
May
(27) |
Jun
(37) |
Jul
(29) |
Aug
(27) |
Sep
(37) |
Oct
(58) |
Nov
(109) |
Dec
(26) |
| 2015 |
Jan
(4) |
Feb
(35) |
Mar
(22) |
Apr
(35) |
May
(28) |
Jun
(20) |
Jul
(4) |
Aug
(16) |
Sep
(37) |
Oct
(13) |
Nov
(13) |
Dec
(14) |
| 2016 |
Jan
(22) |
Feb
(7) |
Mar
(23) |
Apr
(30) |
May
(10) |
Jun
(10) |
Jul
(15) |
Aug
(12) |
Sep
(22) |
Oct
(31) |
Nov
(5) |
Dec
(5) |
| 2017 |
Jan
(30) |
Feb
(25) |
Mar
(28) |
Apr
(4) |
May
(19) |
Jun
(13) |
Jul
(7) |
Aug
(1) |
Sep
(2) |
Oct
(5) |
Nov
(12) |
Dec
(2) |
| 2018 |
Jan
(7) |
Feb
|
Mar
(7) |
Apr
(2) |
May
(8) |
Jun
(18) |
Jul
(6) |
Aug
(3) |
Sep
(15) |
Oct
(33) |
Nov
(13) |
Dec
(7) |
| 2019 |
Jan
(5) |
Feb
(7) |
Mar
(30) |
Apr
(5) |
May
(4) |
Jun
(69) |
Jul
(86) |
Aug
(22) |
Sep
(6) |
Oct
(7) |
Nov
(5) |
Dec
(3) |
| 2020 |
Jan
(10) |
Feb
(12) |
Mar
(22) |
Apr
(5) |
May
(1) |
Jun
(4) |
Jul
(6) |
Aug
|
Sep
(9) |
Oct
|
Nov
|
Dec
(1) |
| 2021 |
Jan
(4) |
Feb
(11) |
Mar
(7) |
Apr
(7) |
May
|
Jun
(3) |
Jul
(10) |
Aug
(6) |
Sep
|
Oct
|
Nov
(18) |
Dec
(2) |
| 2022 |
Jan
(1) |
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
(2) |
Jul
|
Aug
(4) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2023 |
Jan
|
Feb
|
Mar
|
Apr
(1) |
May
(1) |
Jun
|
Jul
|
Aug
(5) |
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Jaime H. <hab...@gm...> - 2019-07-20 04:53:56
|
The following changes were introduced in the source code in relation with https://jira.primekey.se/browse/ECA-7912: *modules/build-properties.xml* <condition property="variant.ra.enabled"> <available file="${mod.peerconnector.path}/src-ra/org/ejbca/peerconnector/ra/RaMasterApiPeerImpl.java"/> </condition> *modules/build.xml* <target name="ra-gui" *if="${variant.ra.enabled}"* depends="ejbca-common-web, ejbca-ejb" description="Build the EJBCA RA GUI WAR"> <ant antfile="${mod.ra-gui.path}/build.xml" target="build" inheritall="true" inheritrefs="true"/> </target> Which, given that the 'peerconnector' module is not available in the CE source code, avoid the build of the 'ra-gui' module even when its own source code is available at modules/ra-gui. This isn't the expected behaviour, right?. Regards. -- Jaime Hablutzel - +51 994690880 |
|
From: Jaime H. <hab...@gm...> - 2019-07-20 00:37:12
|
>From RFC 5280, "3.3. Revocation":
An entry MUST NOT be removed
> from the CRL until it appears on one regularly scheduled CRL issued
> beyond the revoked certificate’s validity period.
But, if you look carefully at the following code from
org.ejbca.core.ejb.crl.PublishingCrlSessionBean#internalCreateCRL (r32721):
for (final RevokedCertInfo revokedCertInfo : revokedCertificates) {
...
if ( !keepexpiredcertsoncrl && revokedCertInfo.getExpireDate() != null
&& *revokedCertInfo.getExpireDate().before(lastCrlCreationDate)* ) {
...
noConflictCertificateStoreSession.setStatus(archiveAdmin,
revokedCertInfo.getCertificateFingerprint(),
CertificateConstants.CERT_ARCHIVED);
} else {
...
}
}
final byte[] crlBytes = generateAndStoreCRL(admin, ca, crlPartitionIndex,
revokedCertificates, lastBaseCrlInfo, false);
You can see that the highlighted snippet won't become true until it is
executed for a second time after a given certificate has expired, producing
that expired certificate entries get into two CRLs instead of only one.
So, is there are reason to keep that code like that instead of replacing it
with something like the following:
if ( !keepexpiredcertsoncrl && revokedCertInfo.getExpireDate() != null &&
*revokedCertInfo.getExpireDate().before(now)* ) {
Which would allow to include the expired certificate entries in only one
CRL beyond the certificate validity period, providing this way a more
precise RFC 5280 implementation?.
PS: The previous is not a production ready patch as it could have side
effects that haven't been checked in detail.
--
Jaime Hablutzel - +51 994690880
|
|
From: <oh...@ya...> - 2019-07-20 00:28:22
|
Hi,
I think I was able to get it working. Something must have been messed up before, plus the "p12" directory was not created, so I created <EJBCA>/p12 directory, then I did "ant clean", then the install and deployment and then it created the JKS etc. in the <EJBCA>/p12 directory and the server was able to start. I still need to figure out how to set it up because we don't have a GUI for this machine.
Jim
On Friday, July 19, 2019, 8:16:59 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I am trying to stand up a new EJBCA (OCSP responder). This is similar to the instance I stood up previously, but this one is in AWS, so I have pretty much been following the same steps, but when I do the "ant runinstall", I am getting the following:
ejbca:init:
[echo]
[echo] ------------------- CA Properties ----------------
[echo] ca.name : ManagementCA
[echo] ca.dn : CN=ManagementCA,O=EJBCA Sample,C=SE
[echo] ca.tokentype : soft
[echo] ca.keytype : RSA
[echo] ca.keyspec : 2048
[echo] ca.signaturealgorithm : SHA256WithRSA
[echo] ca.validity : 3650
[echo] ca.policy : null
[echo] ca.tokenproperties : ${ca.tokenproperties}
[echo] httpsserver.hostname : localhost
[echo] httpsserver.dn : CN=localhost,O=EJBCA Sample,C=SE
[echo] superadmin.cn : SuperAdmin
[echo] superadmin.dn : CN=SuperAdmin
[echo] superadmin.batch : true
[echo] appserver.home : /apps/jboss-eap-7.2
[echo]
ejbca:install:
ejbca:initCA:
[echo] Initializing CA with 'ManagementCA' 'CN=ManagementCA,O=EJBCA Sample,C=SE' 'soft' '<ca.tokenpassword hidden>' '2048' 'RSA' '3650' 'null' 'SHA256WithRSA' -superadmincn 'SuperAdmin'...
[java] Exception in thread "main" java.util.ServiceConfigurationError: org.ejbca.ui.cli.infrastructure.command.CliCommandPlugin: Provider org.ejbca.ui.cli.keybind.InternalKeyBindingCreateCommand could not be instantiated
[java] at java.util.ServiceLoader.fail(ServiceLoader.java:232)
[java] at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
[java] at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
[java] at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
[java] at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
[java] at org.ejbca.ui.cli.infrastructure.library.CommandLibrary.<init>(CommandLibrary.java:55)
[java] at org.ejbca.ui.cli.infrastructure.library.CommandLibrary.<clinit>(CommandLibrary.java:39)
[java] at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:29)
[java] Caused by: javax.ejb.NoSuchEJBException: EJBCLIENT000079: Unable to discover destination for request for EJB StatelessEJBLocator for "ejbca/cesecore-ejb/InternalKeyBindingMgmtSessionBean", view is interface org.cesecore.keybind.InternalKeyBindingMgmtSessionRemote, affinity is None
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:592)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocationResult(RemotingEJBClientInterceptor.java:56)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocationResult(TransactionPostDiscoveryInterceptor.java:133)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocationResult(DiscoveryEJBClientInterceptor.java:115)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocationResult(NamingEJBClientInterceptor.java:79)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:172)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:938)
[java] at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:177)
[java] at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
[java] at com.sun.proxy.$Proxy0.getAvailableTypesAndProperties(Unknown Source)
[java] at org.ejbca.ui.cli.keybind.InternalKeyBindingCreateCommand.<init>(InternalKeyBindingCreateCommand.java:69)
[java] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[java] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[java] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[java] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[java] at java.lang.Class.newInstance(Class.java:442)
[java] at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
[java] ... 5 more
[java] Suppressed: javax.ejb.NoSuchEJBException: No such EJB: ejbca/cesecore-ejb/InternalKeyBindingMgmtSessionBean @ http-remoting://localhost:4447
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel$MethodInvocation.handleResponse(EJBClientChannel.java:1078)
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel$MethodInvocation.handleResponse(EJBClientChannel.java:1004)
[java] at org.jboss.remoting3.util.InvocationTracker.signalResponse(InvocationTracker.java:167)
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel.processMessage(EJBClientChannel.java:189)
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel.access$100(EJBClientChannel.java:115)
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel$1$1.handleMessage(EJBClientChannel.java:683)
[java] at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$handleMessageData$3(RemoteConnectionChannel.java:430)
[java] at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:949)
[java] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[java] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[java] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[java] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[java] at java.lang.Thread.run(Thread.java:748)
BUILD FAILED
/apps/ejbca_ce_6_15_2_1/build.xml:70: The following error occurred while executing this line:
/apps/ejbca_ce_6_15_2_1/bin/cli.xml:97: The following error occurred while executing this line:
/apps/ejbca_ce_6_15_2_1/bin/cli.xml:115: The following error occurred while executing this line:
/apps/ejbca_ce_6_15_2_1/bin/cli.xml:189: Java returned: 1
Can anyone tell me what might be causing this error?
Thanks,Jim
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-19 20:18:02
|
I'm trying to stand up a new EJBCA instance on AWS, to try to see if I can replicate the import speed you are seeing, but ran into an error (see the new thread I just posted).
Thanks,Jim
On Friday, July 19, 2019, 6:39:52 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
I just finished importing your CRL.
996549 records in approx. 12 hours.
This is on my laptop, starting with 400K records in the database already
(i.e ending with 1408370). There is no slowdown as number of records grow).
It failed in the end storing the actual CRL due to a limitation on
packet size in my MySQL config, but that could be changed easily in
MySQL (MariaDB) config. Actually the CRL itself is not needed in the
database in order to answer OCSP queries, so there should optimally be a
switch to not import the CRL itself, only the revocation records (to
save a bit on database storage).
The import speed can with some rather simple development be made orders
of magnitude faster (you don't want a roundtrip for every record for
example).
Cheers,
Tomas
On 2019-07-19 18:33, oh...@ya... wrote:
> Started testing with the VM on an SSD and now I am getting ~500 - 550
> per minute, or about 30K per hour. That is faster than before, but
> still way slower than the 100K per hour you mentioned.
>
> Jim
>
> On Friday, July 19, 2019, 2:56:11 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I don't have any SSDs on the machine that I use as my server... only
> 7200RPM hard drives, except for the OS drive, which is SSD. I can try
> to add an SSD (I have lots of SSDs, just not on that machine).
>
> Jim
>
> On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> I have now imported 600K of the entries form the CRL. Still running at
> 25 entries per second.
>
> The logging does not normally affect performance, logging is a normal
> part. Not running an SSD for the database typically hugely affect DB
> performance. It's possible to tune logging a lot, For example disable
> logging to console and only log to the file (if your console is really
> slow). It's also possible to set the database to not synchronize on disk
> for every write, during database restore and certain operations for example.
>
> I actually have a worse performing log config than you. I have info log
> to the console and DEBUG log to file, so it's logging a lot. I have a
> SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
> top). But it's still a three year old laptop, so not the fastest you can
> get.
>
> Regards,
> Tomas
>
> On 2019-07-19 15:36, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> I got the index done and now am running the CRL import, which has been
>> running about 1/2 hour now, but so far, only about 5500 entries
>> imported, so it doesn't look like it hugely faster than before?
>>
>> I was wondering... on the server side, I think outputting messages like
>> the following to stdout for every entry in the CRL:
>>
>> 09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
>> 09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
>> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
>> 'CN=XXXXXX', serialNo=8A296.
>> 09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-2) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-2) 2019-07-19
>>
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>>
>>
>> Is it possible that that logging is slowing the CRL import?
>>
>> How is your logging setup (log level) compared to what mine is?
>>
>> Other that, it just may be that my system is much slower than your
>> laptop :)??
>>
>> Jim
>>
>>
>>
>>
>> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Ah, eap 7.2 corresponds to wildly 14 as base.
>>
>> Yes, run that to convert, then you should be able to create the index.
>>
>> Cheers,
>> Tomas
>>
>>
>> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi Tomas,
>>
>> No, I am not using Wildfly, but rather JBOSS... specifically
>> "jboss-eap-7.2".
>>
>> So I restored my snapshot from just after I had deleted all the rows
>> in CertificateData that were created before.
>>
>> And it looks like all the tables in the "ejbca" database are MyISAM.
>>
>> So I found a one-liner that will convert them all to InnoDB:
>>
>> mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
>> | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql
>> -u root -p ejbca
>>
>> Should I run that and convert all the tables in the "ejbca" to
> InnoDB???
>>
>>
>> And then try to run the create indexes SQL script?
>>
>>
>> Thanks for all your help!
>>
>> Jim
>>
>>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-19 20:16:44
|
Hi,
I am trying to stand up a new EJBCA (OCSP responder). This is similar to the instance I stood up previously, but this one is in AWS, so I have pretty much been following the same steps, but when I do the "ant runinstall", I am getting the following:
ejbca:init:
[echo]
[echo] ------------------- CA Properties ----------------
[echo] ca.name : ManagementCA
[echo] ca.dn : CN=ManagementCA,O=EJBCA Sample,C=SE
[echo] ca.tokentype : soft
[echo] ca.keytype : RSA
[echo] ca.keyspec : 2048
[echo] ca.signaturealgorithm : SHA256WithRSA
[echo] ca.validity : 3650
[echo] ca.policy : null
[echo] ca.tokenproperties : ${ca.tokenproperties}
[echo] httpsserver.hostname : localhost
[echo] httpsserver.dn : CN=localhost,O=EJBCA Sample,C=SE
[echo] superadmin.cn : SuperAdmin
[echo] superadmin.dn : CN=SuperAdmin
[echo] superadmin.batch : true
[echo] appserver.home : /apps/jboss-eap-7.2
[echo]
ejbca:install:
ejbca:initCA:
[echo] Initializing CA with 'ManagementCA' 'CN=ManagementCA,O=EJBCA Sample,C=SE' 'soft' '<ca.tokenpassword hidden>' '2048' 'RSA' '3650' 'null' 'SHA256WithRSA' -superadmincn 'SuperAdmin'...
[java] Exception in thread "main" java.util.ServiceConfigurationError: org.ejbca.ui.cli.infrastructure.command.CliCommandPlugin: Provider org.ejbca.ui.cli.keybind.InternalKeyBindingCreateCommand could not be instantiated
[java] at java.util.ServiceLoader.fail(ServiceLoader.java:232)
[java] at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
[java] at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
[java] at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
[java] at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
[java] at org.ejbca.ui.cli.infrastructure.library.CommandLibrary.<init>(CommandLibrary.java:55)
[java] at org.ejbca.ui.cli.infrastructure.library.CommandLibrary.<clinit>(CommandLibrary.java:39)
[java] at org.ejbca.ui.cli.EjbcaEjbCli.main(EjbcaEjbCli.java:29)
[java] Caused by: javax.ejb.NoSuchEJBException: EJBCLIENT000079: Unable to discover destination for request for EJB StatelessEJBLocator for "ejbca/cesecore-ejb/InternalKeyBindingMgmtSessionBean", view is interface org.cesecore.keybind.InternalKeyBindingMgmtSessionRemote, affinity is None
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:592)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocationResult(RemotingEJBClientInterceptor.java:56)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocationResult(TransactionPostDiscoveryInterceptor.java:133)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocationResult(DiscoveryEJBClientInterceptor.java:115)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocationResult(NamingEJBClientInterceptor.java:79)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:172)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:594)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:528)
[java] at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:938)
[java] at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:177)
[java] at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:112)
[java] at com.sun.proxy.$Proxy0.getAvailableTypesAndProperties(Unknown Source)
[java] at org.ejbca.ui.cli.keybind.InternalKeyBindingCreateCommand.<init>(InternalKeyBindingCreateCommand.java:69)
[java] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[java] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[java] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[java] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[java] at java.lang.Class.newInstance(Class.java:442)
[java] at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
[java] ... 5 more
[java] Suppressed: javax.ejb.NoSuchEJBException: No such EJB: ejbca/cesecore-ejb/InternalKeyBindingMgmtSessionBean @ http-remoting://localhost:4447
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel$MethodInvocation.handleResponse(EJBClientChannel.java:1078)
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel$MethodInvocation.handleResponse(EJBClientChannel.java:1004)
[java] at org.jboss.remoting3.util.InvocationTracker.signalResponse(InvocationTracker.java:167)
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel.processMessage(EJBClientChannel.java:189)
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel.access$100(EJBClientChannel.java:115)
[java] at org.jboss.ejb.protocol.remote.EJBClientChannel$1$1.handleMessage(EJBClientChannel.java:683)
[java] at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$handleMessageData$3(RemoteConnectionChannel.java:430)
[java] at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:949)
[java] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[java] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[java] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[java] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
[java] at java.lang.Thread.run(Thread.java:748)
BUILD FAILED
/apps/ejbca_ce_6_15_2_1/build.xml:70: The following error occurred while executing this line:
/apps/ejbca_ce_6_15_2_1/bin/cli.xml:97: The following error occurred while executing this line:
/apps/ejbca_ce_6_15_2_1/bin/cli.xml:115: The following error occurred while executing this line:
/apps/ejbca_ce_6_15_2_1/bin/cli.xml:189: Java returned: 1
Can anyone tell me what might be causing this error?
Thanks,Jim
|
|
From: Tomas G. <to...@pr...> - 2019-07-19 18:39:54
|
Hi,
I just finished importing your CRL.
996549 records in approx. 12 hours.
This is on my laptop, starting with 400K records in the database already
(i.e ending with 1408370). There is no slowdown as number of records grow).
It failed in the end storing the actual CRL due to a limitation on
packet size in my MySQL config, but that could be changed easily in
MySQL (MariaDB) config. Actually the CRL itself is not needed in the
database in order to answer OCSP queries, so there should optimally be a
switch to not import the CRL itself, only the revocation records (to
save a bit on database storage).
The import speed can with some rather simple development be made orders
of magnitude faster (you don't want a roundtrip for every record for
example).
Cheers,
Tomas
On 2019-07-19 18:33, oh...@ya... wrote:
> Started testing with the VM on an SSD and now I am getting ~500 - 550
> per minute, or about 30K per hour. That is faster than before, but
> still way slower than the 100K per hour you mentioned.
>
> Jim
>
> On Friday, July 19, 2019, 2:56:11 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> I don't have any SSDs on the machine that I use as my server... only
> 7200RPM hard drives, except for the OS drive, which is SSD. I can try
> to add an SSD (I have lots of SSDs, just not on that machine).
>
> Jim
>
> On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> I have now imported 600K of the entries form the CRL. Still running at
> 25 entries per second.
>
> The logging does not normally affect performance, logging is a normal
> part. Not running an SSD for the database typically hugely affect DB
> performance. It's possible to tune logging a lot, For example disable
> logging to console and only log to the file (if your console is really
> slow). It's also possible to set the database to not synchronize on disk
> for every write, during database restore and certain operations for example.
>
> I actually have a worse performing log config than you. I have info log
> to the console and DEBUG log to file, so it's logging a lot. I have a
> SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
> top). But it's still a three year old laptop, so not the fastest you can
> get.
>
> Regards,
> Tomas
>
> On 2019-07-19 15:36, oh...@ya... <mailto:oh...@ya...> wrote:
>> Hi,
>>
>> I got the index done and now am running the CRL import, which has been
>> running about 1/2 hour now, but so far, only about 5500 entries
>> imported, so it doesn't look like it hugely faster than before?
>>
>> I was wondering... on the server side, I think outputting messages like
>> the following to stdout for every entry in the CRL:
>>
>> 09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
>> 09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
>> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
>> 'CN=XXXXXX', serialNo=8A296.
>> 09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-1) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-2) 2019-07-19
>>
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
>> 09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
>> task-2) 2019-07-19
>>
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
>> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
>> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
>> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>>
>>
>> Is it possible that that logging is slowing the CRL import?
>>
>> How is your logging setup (log level) compared to what mine is?
>>
>> Other that, it just may be that my system is much slower than your
>> laptop :)??
>>
>> Jim
>>
>>
>>
>>
>> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Ah, eap 7.2 corresponds to wildly 14 as base.
>>
>> Yes, run that to convert, then you should be able to create the index.
>>
>> Cheers,
>> Tomas
>>
>>
>> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi Tomas,
>>
>> No, I am not using Wildfly, but rather JBOSS... specifically
>> "jboss-eap-7.2".
>>
>> So I restored my snapshot from just after I had deleted all the rows
>> in CertificateData that were created before.
>>
>> And it looks like all the tables in the "ejbca" database are MyISAM.
>>
>> So I found a one-liner that will convert them all to InnoDB:
>>
>> mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
>> | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql
>> -u root -p ejbca
>>
>> Should I run that and convert all the tables in the "ejbca" to
> InnoDB???
>>
>>
>> And then try to run the create indexes SQL script?
>>
>>
>> Thanks for all your help!
>>
>> Jim
>>
>>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-19 16:33:34
|
Started testing with the VM on an SSD and now I am getting ~500 - 550 per minute, or about 30K per hour. That is faster than before, but still way slower than the 100K per hour you mentioned.
Jim
On Friday, July 19, 2019, 2:56:11 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I don't have any SSDs on the machine that I use as my server... only 7200RPM hard drives, except for the OS drive, which is SSD. I can try to add an SSD (I have lots of SSDs, just not on that machine).
Jim
On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I have now imported 600K of the entries form the CRL. Still running at
25 entries per second.
The logging does not normally affect performance, logging is a normal
part. Not running an SSD for the database typically hugely affect DB
performance. It's possible to tune logging a lot, For example disable
logging to console and only log to the file (if your console is really
slow). It's also possible to set the database to not synchronize on disk
for every write, during database restore and certain operations for example.
I actually have a worse performing log config than you. I have info log
to the console and DEBUG log to file, so it's logging a lot. I have a
SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
top). But it's still a three year old laptop, so not the fastest you can
get.
Regards,
Tomas
On 2019-07-19 15:36, oh...@ya... wrote:
> Hi,
>
> I got the index done and now am running the CRL import, which has been
> running about 1/2 hour now, but so far, only about 5500 entries
> imported, so it doesn't look like it hugely faster than before?
>
> I was wondering... on the server side, I think outputting messages like
> the following to stdout for every entry in the CRL:
>
> 09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
> 09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
> 'CN=XXXXXX', serialNo=8A296.
> 09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-2) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-2) 2019-07-19
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>
>
> Is it possible that that logging is slowing the CRL import?
>
> How is your logging setup (log level) compared to what mine is?
>
> Other that, it just may be that my system is much slower than your
> laptop :)??
>
> Jim
>
>
>
>
> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Ah, eap 7.2 corresponds to wildly 14 as base.
>
> Yes, run that to convert, then you should be able to create the index.
>
> Cheers,
> Tomas
>
>
> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi Tomas,
>
> No, I am not using Wildfly, but rather JBOSS... specifically
> "jboss-eap-7.2".
>
> So I restored my snapshot from just after I had deleted all the rows
> in CertificateData that were created before.
>
> And it looks like all the tables in the "ejbca" database are MyISAM.
>
> So I found a one-liner that will convert them all to InnoDB:
>
> mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
> | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql
> -u root -p ejbca
>
> Should I run that and convert all the tables in the "ejbca" to InnoDB???
>
>
> And then try to run the create indexes SQL script?
>
>
> Thanks for all your help!
>
> Jim
>
>
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-19 15:33:23
|
Ok, I installed an SSD on my machine and am moving the vm files to the SSD now.
Will try the CRL import once I get the machine moved.
On Friday, July 19, 2019, 2:56:11 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I don't have any SSDs on the machine that I use as my server... only 7200RPM hard drives, except for the OS drive, which is SSD. I can try to add an SSD (I have lots of SSDs, just not on that machine).
Jim
On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I have now imported 600K of the entries form the CRL. Still running at
25 entries per second.
The logging does not normally affect performance, logging is a normal
part. Not running an SSD for the database typically hugely affect DB
performance. It's possible to tune logging a lot, For example disable
logging to console and only log to the file (if your console is really
slow). It's also possible to set the database to not synchronize on disk
for every write, during database restore and certain operations for example.
I actually have a worse performing log config than you. I have info log
to the console and DEBUG log to file, so it's logging a lot. I have a
SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
top). But it's still a three year old laptop, so not the fastest you can
get.
Regards,
Tomas
On 2019-07-19 15:36, oh...@ya... wrote:
> Hi,
>
> I got the index done and now am running the CRL import, which has been
> running about 1/2 hour now, but so far, only about 5500 entries
> imported, so it doesn't look like it hugely faster than before?
>
> I was wondering... on the server side, I think outputting messages like
> the following to stdout for every entry in the CRL:
>
> 09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
> 09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
> 'CN=XXXXXX', serialNo=8A296.
> 09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-2) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-2) 2019-07-19
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>
>
> Is it possible that that logging is slowing the CRL import?
>
> How is your logging setup (log level) compared to what mine is?
>
> Other that, it just may be that my system is much slower than your
> laptop :)??
>
> Jim
>
>
>
>
> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Ah, eap 7.2 corresponds to wildly 14 as base.
>
> Yes, run that to convert, then you should be able to create the index.
>
> Cheers,
> Tomas
>
>
> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi Tomas,
>
> No, I am not using Wildfly, but rather JBOSS... specifically
> "jboss-eap-7.2".
>
> So I restored my snapshot from just after I had deleted all the rows
> in CertificateData that were created before.
>
> And it looks like all the tables in the "ejbca" database are MyISAM.
>
> So I found a one-liner that will convert them all to InnoDB:
>
> mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
> | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql
> -u root -p ejbca
>
> Should I run that and convert all the tables in the "ejbca" to InnoDB???
>
>
> And then try to run the create indexes SQL script?
>
>
> Thanks for all your help!
>
> Jim
>
>
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-19 14:55:56
|
Hi,
I don't have any SSDs on the machine that I use as my server... only 7200RPM hard drives, except for the OS drive, which is SSD. I can try to add an SSD (I have lots of SSDs, just not on that machine).
Jim
On Friday, July 19, 2019, 2:16:04 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
I have now imported 600K of the entries form the CRL. Still running at
25 entries per second.
The logging does not normally affect performance, logging is a normal
part. Not running an SSD for the database typically hugely affect DB
performance. It's possible to tune logging a lot, For example disable
logging to console and only log to the file (if your console is really
slow). It's also possible to set the database to not synchronize on disk
for every write, during database restore and certain operations for example.
I actually have a worse performing log config than you. I have info log
to the console and DEBUG log to file, so it's logging a lot. I have a
SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
top). But it's still a three year old laptop, so not the fastest you can
get.
Regards,
Tomas
On 2019-07-19 15:36, oh...@ya... wrote:
> Hi,
>
> I got the index done and now am running the CRL import, which has been
> running about 1/2 hour now, but so far, only about 5500 entries
> imported, so it doesn't look like it hugely faster than before?
>
> I was wondering... on the server side, I think outputting messages like
> the following to stdout for every entry in the CRL:
>
> 09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
> 09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
> 'CN=XXXXXX', serialNo=8A296.
> 09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-2) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-2) 2019-07-19
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>
>
> Is it possible that that logging is slowing the CRL import?
>
> How is your logging setup (log level) compared to what mine is?
>
> Other that, it just may be that my system is much slower than your
> laptop :)??
>
> Jim
>
>
>
>
> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Ah, eap 7.2 corresponds to wildly 14 as base.
>
> Yes, run that to convert, then you should be able to create the index.
>
> Cheers,
> Tomas
>
>
> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi Tomas,
>
> No, I am not using Wildfly, but rather JBOSS... specifically
> "jboss-eap-7.2".
>
> So I restored my snapshot from just after I had deleted all the rows
> in CertificateData that were created before.
>
> And it looks like all the tables in the "ejbca" database are MyISAM.
>
> So I found a one-liner that will convert them all to InnoDB:
>
> mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
> | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql
> -u root -p ejbca
>
> Should I run that and convert all the tables in the "ejbca" to InnoDB???
>
>
> And then try to run the create indexes SQL script?
>
>
> Thanks for all your help!
>
> Jim
>
>
|
|
From: Tomas G. <to...@pr...> - 2019-07-19 14:16:07
|
I have now imported 600K of the entries form the CRL. Still running at
25 entries per second.
The logging does not normally affect performance, logging is a normal
part. Not running an SSD for the database typically hugely affect DB
performance. It's possible to tune logging a lot, For example disable
logging to console and only log to the file (if your console is really
slow). It's also possible to set the database to not synchronize on disk
for every write, during database restore and certain operations for example.
I actually have a worse performing log config than you. I have info log
to the console and DEBUG log to file, so it's logging a lot. I have a
SSD disk in my laptop, and 4 cores (multithreaded so shows up as 8 in
top). But it's still a three year old laptop, so not the fastest you can
get.
Regards,
Tomas
On 2019-07-19 15:36, oh...@ya... wrote:
> Hi,
>
> I got the index done and now am running the CRL import, which has been
> running about 1/2 hour now, but so far, only about 5500 entries
> imported, so it doesn't look like it hugely faster than before?
>
> I was wondering... on the server side, I think outputting messages like
> the following to stdout for every entry in the CRL:
>
> 09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
> 09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;***
> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate
> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
> fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1,
> subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN
> 'CN=XXXXXX', serialNo=8A296.
> 09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-1) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-2) 2019-07-19
> 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
> 09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default
> task-2) 2019-07-19
> 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;***
> Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored
> for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41',
> fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
>
>
> Is it possible that that logging is slowing the CRL import?
>
> How is your logging setup (log level) compared to what mine is?
>
> Other that, it just may be that my system is much slower than your
> laptop :)??
>
> Jim
>
>
>
>
> On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Ah, eap 7.2 corresponds to wildly 14 as base.
>
> Yes, run that to convert, then you should be able to create the index.
>
> Cheers,
> Tomas
>
>
> On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi Tomas,
>
> No, I am not using Wildfly, but rather JBOSS... specifically
> "jboss-eap-7.2".
>
> So I restored my snapshot from just after I had deleted all the rows
> in CertificateData that were created before.
>
> And it looks like all the tables in the "ejbca" database are MyISAM.
>
> So I found a one-liner that will convert them all to InnoDB:
>
> mysql -u root -p ejbca -e "show table status where Engine='MyISAM';"
> | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql
> -u root -p ejbca
>
> Should I run that and convert all the tables in the "ejbca" to InnoDB???
>
>
> And then try to run the create indexes SQL script?
>
>
> Thanks for all your help!
>
> Jim
>
>
|
|
From: <oh...@ya...> - 2019-07-19 13:37:13
|
Hi,
I got the index done and now am running the CRL import, which has been running about 1/2 hour now, but so far, only about 5500 entries imported, so it doesn't look like it hugely faster than before?
I was wondering... on the server side, I think outputting messages like the following to stdout for every entry in the CRL:
09:32:37,008 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default task-1) 2019-07-19 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/endentityprofilesrules/1/revoke_end_entity;resource1=/ra_functionality/revoke_end_entity
09:32:37,025 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default task-1) 2019-07-19 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
09:32:37,035 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default task-1) 2019-07-19 09:32:37-04:00;CERT_REVOKED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;8A296;*** Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Revoked certificate for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41', fp=667d63d9060a424dd67ed19bdcd25917106eba5a, revocationReason=1, subjectDN 'CN=Dummy Missing in Imported CRL,SN=8A296', issuerDN 'CN=XXXXXX', serialNo=8A296.
09:32:37,079 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default task-1) 2019-07-19 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
09:32:37,115 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default task-2) 2019-07-19 09:32:37-04:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;ejbca;;;;resource0=/ca/1164433895
09:32:37,133 INFO [org.cesecore.audit.impl.log4j.Log4jDevice] (default task-2) 2019-07-19 09:32:37-04:00;CERT_STORED;SUCCESS;CERTIFICATE;CORE;ejbca;1164433895;4B78C;*** Missing During CRL Import to: XXXXXXXXXXXCA_41;msg=Certificate stored for username '*** Missing During CRL Import to: XXXXXXXXXXXCA_41', fp=bf50e62fe1bde64cca1a4ffa050d99bc750c68b1, subjectDN 'CN=Dummy Missin
Is it possible that that logging is slowing the CRL import?
How is your logging setup (log level) compared to what mine is?
Other that, it just may be that my system is much slower than your laptop :)??
Jim
On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Ah, eap 7.2 corresponds to wildly 14 as base.
Yes, run that to convert, then you should be able to create the index.
Cheers,
Tomas
On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi Tomas,
No, I am not using Wildfly, but rather JBOSS... specifically "jboss-eap-7.2".
So I restored my snapshot from just after I had deleted all the rows in CertificateData that were created before.
And it looks like all the tables in the "ejbca" database are MyISAM.
So I found a one-liner that will convert them all to InnoDB:
mysql -u root -p ejbca -e "show table status where Engine='MyISAM';" | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql -u root -p ejbca
Should I run that and convert all the tables in the "ejbca" to InnoDB???
And then try to run the create indexes SQL script?
Thanks for all your help!
Jim
|
|
From: <oh...@ya...> - 2019-07-19 09:13:10
|
I have been running the conversion since my last post ---- is the conversion supposed to take that long? It is still running :(!!
Jim
On Friday, July 19, 2019, 8:18:07 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Ah, eap 7.2 corresponds to wildly 14 as base.
Yes, run that to convert, then you should be able to create the index.
Cheers,
Tomas
On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi Tomas,
No, I am not using Wildfly, but rather JBOSS... specifically "jboss-eap-7.2".
So I restored my snapshot from just after I had deleted all the rows in CertificateData that were created before.
And it looks like all the tables in the "ejbca" database are MyISAM.
So I found a one-liner that will convert them all to InnoDB:
mysql -u root -p ejbca -e "show table status where Engine='MyISAM';" | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql -u root -p ejbca
Should I run that and convert all the tables in the "ejbca" to InnoDB???
And then try to run the create indexes SQL script?
Thanks for all your help!
Jim
On Friday, July 19, 2019, 8:01:15 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, MyISAM is horrible and can not be used.
I'm not sure about the exact commands to change it. But I think there
are good internet guides.
The reason you have MyISAM is probably because you are using WildFly 14
(are you?). There was a change in Hibernate that caused this issue with
automatic database creation.
One way to modify is to mysqldump the database, change enging in the sql
file and restore it.
Regards,
Tomas
On 2019-07-19 09:19, ohaya--- via Ejbca-develop wrote:
> Hi,
>
> Mysql version: Server version: 5.7.26 MySQL Community Server (GPL)
>
>
>
>
> mysql> show create table CertificateData;
>
>
>
> | CertificateData | CREATE TABLE `CertificateData` (
> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `certificateProfileId` int(11) NOT NULL,
> `endEntityProfileId` int(11) DEFAULT NULL,
> `expireDate` bigint(20) NOT NULL,
> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `notBefore` bigint(20) DEFAULT NULL,
> `revocationDate` bigint(20) NOT NULL,
> `revocationReason` int(11) NOT NULL,
> `rowVersion` int(11) NOT NULL,
> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `status` int(11) NOT NULL,
> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `type` int(11) NOT NULL,
> `updateTime` bigint(20) NOT NULL,
> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `base64Cert` longtext,
> `rowProtection` longtext,
> PRIMARY KEY (`fingerprint`),
> KEY `certificatedata_idx2` (`username`)
> ) ENGINE=MyISAM DEFAULT CHARSET=utf8 |
>
>
>
> AH! Yes mine is showing as MyISAM and not InnoDB!
>
>
>
> Do I need to change the engine to InnoDB? And, if so, how do I do that?
>
> Do I need to do that for every table in the ejbca database?
>
>
> Thanks,
> Jim
>
>
>
>
> On Friday, July 19, 2019, 6:20:23 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> That looks about right. What version of database are you using? I just
> tried the index command (on line 23 in the SQL file) myself and it works.
>
> CREATE INDEX certificatedata_idx4 ON CertificateData (subjectDN);
>
>
> What does your database creation say?
>
>>show create table CertiifcateData;
>
> It should look something like this:
>
> | CertificateData | CREATE TABLE `CertificateData` (
> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `certificateProfileId` int(11) NOT NULL,
> `crlPartitionIndex` int(11) DEFAULT NULL,
> `endEntityProfileId` int(11) DEFAULT NULL,
> `expireDate` bigint(20) NOT NULL,
> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `notBefore` bigint(20) DEFAULT NULL,
> `revocationDate` bigint(20) NOT NULL,
> `revocationReason` int(11) NOT NULL,
> `rowVersion` int(11) NOT NULL,
> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `status` int(11) NOT NULL,
> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `type` int(11) NOT NULL,
> `updateTime` bigint(20) NOT NULL,
> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `base64Cert` longtext,
> `certificateRequest` longtext,
> `rowProtection` longtext,
> PRIMARY KEY (`fingerprint`)
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
>
> Is the database engine wrong? I.e. not InnoDB?
>
> I am importing the CRL right now and processing about 27 records per
> second on my laptop.
>
> Regards,
> Tomas
>
>
> On 2019-07-18 18:09, ohaya--- via Ejbca-develop wrote:
>> According to my notes I used these commands:
>>
>> |*/CREATE DATABASE ejbca CHARACTER SET utf8 COLLATE utf8_general_ci;/*|
>>
>> |*/ /*|
>>
>> |*/GRANT ALL PRIVILEGES ON ejbca.* TO 'ejbca'@'localhost' IDENTIFIED BY
>> 'ejbca';/*|
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 3:37:51 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> What command did you use to create the database?
>>
>> On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> This is really getting frustrating!!!
>>
>> [root@ejbca <mailto:root@ejbca> sql-scripts]# mysql -u root
> -p'xxxxxx' ejbca <
>> create-index-ejbca.sql
>> mysql: [Warning] Using a password on the command line interface can
>> be insecure.
>> ERROR 1071 (42000) at line 23: Specified key was too long; max key
>> length is 1000 bytes
>>
>> Jim
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> The "machine" is a virtual machine under VirtualBox on a machine I
>> have that has 64GB physical RAM. Drives are all 7200 RPM drives.
>>
>> I will restore the snapshot, then increase the memory and # of
>> processors and try to run the create index again.
>>
>> Jim
>>
>> On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Hi,
>>
>> Just run create index script.
>>
>> Create index should run pretty fast even with millions of records in
>> the database. I suspect that your database is configured very slow,
>> perhaps running everything on disl, without RAM, using slow disks?
>>
>> A standard installation of mysql/mariadb on a centos machine should
>> run faster. Then you have pretty low amount of ram for running
>> everything on one box, handing millions of records.
>>
>> Regards,
>> Tomas
>>
>> On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> I had taken a snapshot of the machine before I tried to do the
>> index earlier, so I restored that snapshot.
>>
>> Then, I tried running the "optimize-ejbca-mysql.sql" and I got a
>> different/new error:
>>
>> ERROR 1091 (42000) at line 27: Can't DROP
>> 'certificatedata_idx5'; check that column/key exists
>>
>> Jim
>>
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> Oh oh, I just got this from the mysql execution:
>>
>> ERROR 1071 (42000) at line 23: Specified key was too long; max
>> key length is 1000 bytes
>>
>> Jim
>>
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> FYI, I am running the create-index-ejbca.sql script now, but
>> that seems to be taking FOREVER?
>>
>> I had stopped the earlier import, and then deleted all the rows
>> for that CA DN in CertificateData table, and then bounced the
>> machine, then I ran the create-index-ejbca.sql.
>>
>> Why is it taking so long? Should I have done something in
>> addition to deleting the rows? I am guessing that the reason
>> the sql script is taking so long is that it's somehow still
>> trying to index the rows were deleted or something like that?
>>
>> Thanks,
>> Jim
>>
>>
>>
>> On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> I notice that there is:
>>
>> - create-index-ejbca.sql
>> - optimize-ejbca-mysql.sql
>>
>> Should I run BOTH of them, or just the "create-index-ejbca.sql "?
>>
>> Thanks,
>> Jim
>>
>>
>> On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Yes, i could see from your output from top that mysql takes all
>> CPU. This indicates that you don't have any indexes in your
>> database, which makes it horribly slow as numbers grow.
>>
>> Apply database indexes from doc/sql-scripts.
>>
>> Regards,
>> Tomas
>>
>>
>> On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> So far, processed only 185706 entries, and it looks like the
>> processing is going to a crawl (really slow now).
>>
>> Which kind of makes me wonder if maybe part of reason for
>> the slowness is that it is having to search the certificate
>> database for the serial number, and now, as there are more
>> dummy certs in the database, that search is taking longer
>> and longer?
>>
>> Jim
>>
>>
>> On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> This is what top is showing now (the import is still running):
>>
>> top - 13:56:19 up 1 day, 1:09, 3 users, load average:
>> 1.18, 1.11, 1.07
>> Tasks: 167 total, 2 running, 165 sleeping, 0 stopped,
>> 0 zombie
>> %Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0
>> hi, 0.3 si, 0.0 st
>> KiB Mem : 4045880 total, 123192 free, 3053072 used,
>> 869616 buff/cache
>> KiB Swap: 3905532 total, 3753980 free, 151552 used.
>> 688136 avail Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU
>> %MEM TIME+ COMMAND
>> 1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2
>> 1227:41 mysqld
>> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2
>> 139:05.67 java
>> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3
>> 176:26.68 java
>> 22466 root 20 0 0 0 0 S 0.7 0.0
>> 0:00.73 kworker/u4:1
>>
>> That is running on 2-CPU machine with 4GB RAM.
>>
>> Jim
>>
>>
>>
>>
>> On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas
>> Gustavsson <to...@pr... <mailto:to...@pr...>>
> wrote:
>>
>>
>> Yes, like top.
>>
>>
>> On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi Tomas,
>>
>> I'm sorry that I missed the message below, but I am not
>> clear what you are asking me to check/monitor? Do you
>> mean like running something like "top" command?
>>
>> Thanks,
>> Jim
>>
>> On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas
>> Gustavsson <to...@pr...
> <mailto:to...@pr...>> wrote:
>>
>>
>> To verify that the changes actually happen...
>>
>> Do you monitor CPU usage to see what processes use CPU
>> and memory?
>>
>>
>> On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> The import has been running for about an hour now
>> and has imported 17836 entries.
>>
>> Per what I had said previously, this CRL file has
>> 1002255 entries, that means it would take about 56+
>> hours to import this CRL.
>>
>> So, this import processing is still way too slow.
>>
>> I don't know what is involved, or even how much it
>> would improve things, but I think that the next
>> thing to try is to eliminated both the check for the
>> serial number in the DB AND the dummy cert creation.
>>
>> Jim
>>
>>
>> On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya---
>> via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> Ok, I rebuilt the JAR using your Java class, and it
>> is running now, and creating rows in the
>> CertificateData, but it still SEEMS kind of slow.
>>
>> I am running now and will try to get a measurement
>> of revoked entries/time and will post back.
>>
>> Jim
>>
>>
>>
>>
>> On Tuesday, July 16, 2019, 5:25:32 PM UTC,
>> oh...@ya... <mailto:oh...@ya...>
> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> So I just tried what I said (emptying the "if") and
>> you are right, it doesn't create the rows in the
>> CeritificateData table :(...
>>
>> Ok, I will try your version...
>>
>> Jim
>>
>> On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas
>> Gustavsson <to...@pr...
> <mailto:to...@pr...>> wrote:
>>
>>
>>
>> Hi,
>>
>> I don't think so, no quite actually. In order to use
>> the API to populate
>> the database you need the certificate object it
>> seems. Something like
>> the attached file should work, it still creates a
>> dummy certificate, but
>> uses a static hard coded key, so no key generation.
>>
>> The file assumes you are on 6.15.x.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
>> > For testing (only...), I am thinking that if I
>> modify this If clause in
>> > CaImportCrlCommand.java:
>> >
>> > if (adaptive)
>> > {
>> > Date time = new Date();
>> > KeyPair key_pair =
>> KeyTools.genKeys("2048", "RSA");
>> >
>> > SubjectPublicKeyInfo pkinfo =
>> >
>>
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
>> > X500Name dnName = new
>> X500Name("CN=Dummy Missing in
>> > Imported CRL, serialNumber=" + serialHex);
>> > Date notAfter = new
>> Date(time.getTime() + 315360000000L);
>> > X509v3CertificateBuilder certbuilder
>> = new
>> > X509v3CertificateBuilder(X500Name.getInstance(cacert
>> >
>> .getSubjectX500Principal().getEncoded()), serialNr,
>> > time, notAfter, dnName, pkinfo);
>> > ContentSigner signer = new
>> BufferingContentSigner(new
>> >
>>
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
>> > .getPrivate()), 20480);
>> > X509CertificateHolder certHolder =
>> certbuilder.build(signer);
>> > X509Certificate certificate =
>> >
>>
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
>> > X509Certificate.class);
>> >
>> > String fingerprint =
>> > CertTools.getFingerprintAsString(certificate);
>> >
>> > EndEntityInformation
>> missingUserEndEntityInformation =
>> >
>>
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
>> > missing_user_name);
>> > if (missingUserEndEntityInformation
>> == null)
>> > {
>> > log.debug("Loading/updating user "
>> + missing_user_name);
>> >
>> > EndEntityInformation userdataNew
> = new
>> > EndEntityInformation(missing_user_name,
>> > CertTools.getSubjectDN(certificate),
>> cainfo.getCAId(), null, null, 10,
>> > new EndEntityType(EndEntityTypes.ENDUSER), 1, 1,
>> null, null, 1, 0, null);
>> >
>> > userdataNew.setPassword("foo123");
>> >
>> >
>>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
>> > userdataNew, false);
>> >
>> > log.info("User '" +
>> missing_user_name + "' has been
>> > added.");
>> >
>> >
>>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
>> > missing_user_name, 50);
>> >
>> > log.info("User '" +
>> missing_user_name + "' has been
>> > updated.");
>> > }
>> >
>> >
>>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>> >
>> > EJBTools.wrap(certificate),
>> missing_user_name,
>> > fingerprint, 20, 1, 1, 0, null, new Date()
>> >
>> > .getTime());
>> > log.info("Dummy certificate '" +
>> serialHex + "' has been
>> > stored.");
>> > }
>> >
>> > to (for example):
>> >
>> > if (adaptive)
>> > {
>> >
>> > log.info("*** SKIPPING *** Dummy
>> certificate '" +
>> > serialHex + "' has been stored.");
>> > }
>> >
>> >
>> > Would THAT work??
>> >
>> > Thanks,
>> > Jim
>> >
>> >
>> >
>> >
>> > On Tuesday, July 16, 2019, 10:59:31 AM UTC,
>> ohaya--- via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> >
>> > Hi Tomas (or anyone else),
>> >
>> > Sorry - I was typing my last msg on phone, and it
>> wasn't as detailed or
>> > clear as I wanted.
>> >
>> > Basically what I was saying is I was thinking
>> that, as a quick sanity
>> > test, if there was one or two places where the
> code in
>> > the CaImportCRLCommand.java was maybe making a
>> method call to create the
>> > dummy cert and put it into the database, then
>> maybe I could decompile
>> > and modify the CaImportCRLCommand.java, and
>> comment out those method
>> > calls, then, re-do my large CRL import test and
>> time it?
>> >
>> > If so, can you let me know specifically where
>> those calls are, and if I
>> > just comment them out, would that work?
>> >
>> > Ironically, that (not doing the dummy cert
>> creation and writing) was
>> > what I *thought* the "LENIENT" option was doing).
>> >
>> > Thanks,
>> > Jim
>> >
>> > P. S. I have time today to try this if I figure
>> out what needs to be done.
>> >
>> >
>> >
>> > On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya---
>> via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> >
>> >
>> > Yes, we are definitely interested/willing to test
>> patches. I actually
>> > was wondering if it would be a quick test to mod
>> the code somewhere to
>> > avoid a call that creates the dummy cert, to get
>> an idea if that would
>> > reduce the import time enough??
>> >
>> >
>> > Sent from Yahoo Mail on Android
>> >
>>
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>> >
>> > On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>>
>> wrote:
>> > Most large users, some very very large,
>> typically uses on line
>> > updates directly from the CA. Many CAs today
>> don't accept only
>> > importing CRLs, as they need to keep track of
>> issued certificates as
>> > well not only revoked (can't use "non existing
>> is good").
>> > Having said thst, it's interesting to be able
>> to import large CRLs,
>> > so improving it is of interest.
>> >
>> > Importing CRLs would actually not have to
>> generate a dummy cert at
>> > all, since a certificate object is not needed
>> on the responder,
>> > which should make it very fast.
>> >
>> > If you are willing to test patches, we can work
>> with you on improving.
>> >
>> > Regards,
>> > Tomas
>> >
>> >
>> > On July 15, 2019 5:26:08 PM GMT+02:00, ohaya---
>> via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> > Hi,
>> >
>> > I am curious whether you have other users
>> that import large CRLs
>> > like this?
>> >
>> > Are we the only ones who would encounter
>> these really long
>> > import times, because the other users who
>> might have large CRLs
>> > use the EJBCA CA also, so they don't run
>> into having the dummy
>> > certs being created for every entry in the
> CRL?
>> >
>> > Jim
>> >
>> > On Monday, July 15, 2019, 1:42:57 PM UTC,
>> Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>> wrote:
>> >
>> >
>> >
>> > I see...
>> >
>> > I can see that a new key pair is being
>> generated for each dummy
>> > cert.
>> >
>> > In CaImportCRLCommand.java:
>> >
>> > �� final KeyPair key_pair =
>> KeyTools.genKeys("2048",
>> > AlgorithmConstants.KEYALGORITHM_RSA);
>> >
>> > Chaning that to 1024 or 512 would be an
>> easy improvement. A bit more
>> > complex, but even larger speedup would be
>> to use a "hard coded"
>> > dummy
>> > key, i.e. same key for all dummy certs.
>> >
>> > You can experiment by changing 2048 to 512
>> (not secure but it
>> > doesn't
>> > have to be for this dummy cert).
>> >
>> > Regards,
>> > Tomas
>> >
>> > On 2019-07-11 14:19, ohaya--- via
>> Ejbca-develop wrote:
>> > > FYI, the is running, but it is REALLY
>> slow. The CRL file has
>> > 1002255
>> > > entries in it, and the import is only up
>> to 8899 now, after
>> > about 40
>> > > minutes... which would take about 75
> hours!!
>> > >
>> > > In our production systems, we are
>> required to update some CRLs
>> > every
>> > > hour, so that would be a problem.
>> > >
>> > > What can I do to speed that up??
>> > >
>> > > Thanks,
>> > > Jim
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 11:09:33 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
>> > >
>> > >
>> > > AHHH... Ok, it is running now, and from
>> checking the
>> > CertificateData in
>> > > the Mysql database, I can see that it is
>> now populating the
>> > > CertificateData!!
>> > >
>> > > Thanks!
>> > >
>> > > Jim
>> > >
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 11:07:22 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
>> > >
>> > >
>> > > I wasn't sure if we should use ADAPTIVE,
>> since we are not
>> > using the CA part.
>> > >
>> > >
>> > > I will try with "-o ADAPTIVE" now.
>> > >
>> > >
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 7:42:14 AM
>> UTC, Tomas Gustavsson
>> > > <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>> wrote:
>> > >
>> > >
>> > >
>> > > It should be ADAPTIVE.
>> > >
>> > > The documentation says:
>> > > bin/ejbca.sh ca importcrl --help
>> > > ---
>> > > ADAPTIVE means that missing certficates
>> will be replaced by dummy
>> > > certificates to cater for proper CRLs for
>> missing certificates.
>> > > ---
>> > >
>> > > Regards,
>> > > Tomas
>> > >
>> > > On 2019-07-11 08:05, ohaya--- via
>> Ejbca-develop wrote:
>> > >> Hi,
>> > >>
>> > >> I've been looking at the code in
>> CaImportCRLCommand, and I am
>> > wondering
>> > >> if, when the command is run in LENIENT
>> mode and it doesn't
>> > find the cert
>> > >> in the database (and outputs the
>> "missing in database" error
>> > message),
>> > >> what is supposed to happen?
>> > >>
>> > >> Mind you, the only error messages I am
>> seeing are the "missing in
>> > >> database" error messages. I don't see
>> any of the other error
>> > messages
>> > >> that are in/from CaImportCRLCommand.java.
>> > >>
>> > >> I am wondering if, when the command is
>> run with "-o LENIENT",
>> > and when
>> > >> it outputs the "missing in database"
>> error message, then the
>> > ejbca.sh is
>> > >> basically not doing anything (and not
>> storing anything into the
>> > >> CertificateData table)?
>> > >>
>> > >> From the lack of any other error
>> messages (vs. what I see in the
>> > >> CaImportCRLCommand.java code), that is
>> what it looks like is
>> > happening.
>> > >>
>> > >> So given I am using EJBCA for OCSP
>> responder only, is running the
>> > >> ejbca.sh in the "-o LENIENT" mode correct?
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >>
>> > >> On Thursday, July 11, 2019, 5:43:37 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> I've been doing a bunch of testing with
>> trying to import the
>> > large CRL,
>> > >> and wanted to summarize:
>> > >>
>> > >> - The ejbca.sh runs fine - it completes
>> without outputting an
>> > errors,
>> > >> except for outputting a bunch of the
>> "cert xxxx missing in
>> > database"
>> > >> messages.
>> > >>
>> > >> - When the import completes, and I check
>> the Mysql database,
>> > the crlData
>> > >> (the full PEM of the CRL) is in the
>> database, but there is no
>> > rows in
>> > >> the CertificateData table for that CA at
>> all.
>> > >>
>> > >> - I even figured out how to get the
>> ejbca.sh to log using
>> > >> log4j.properties, and it output a bunch
>> of logging, mainly
>> > the missing
>> > >> in database errors, but other than that
>> I didn't see any
>> > obvious errors.
>> > >>
>> > >>
>> > >> So, bottom line is that even though the
>> ejbca.sh seems to run
>> > without
>> > >> errors (other than the missing in
>> database errors), the import is
>> > >> causing the CRL PEM to be stored in the
>> database, but the
>> > >> CertificateData table is not being
>> populated at all for that CA.
>> > >>
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >>
>> > >> On Wednesday, July 10, 2019, 7:40:16 PM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> Ok, I found Tomas' original message...
>> it was in my Spam
>> > folder....
>> > >>
>> > >> I also tried re-running the "./ejbca.sh
>> ca importcrl" command
>> > a bunch of
>> > >> ways. I think it is running without
>> outputting any errors,
>> > but still,
>> > >> no rows are appearing with that issuerDN
>> at all.
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Tuesday, July 9, 2019, 10:45:01 AM
>> UTC, Tomas Gustavsson
>> > >> <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>> > <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> OCSP responses does not have anything to
>> do with the CRL
>> > itself. Once
>> > >> the CRL has been imported each
>> certificate status is in its
>> > own record
>> > >> in the CertificateData table. This is
>> where status is read
>> > when reponses
>> > >> are created.
>> > >> If there are issues, it is in that case
>> related to the import
>> > of the CRL.
>> > >>
>> > >> Verify that certificate records are
>> created in
>> > CertificateData when
>> > >> importing the CRL.
>> > >> Also, a debug log when making OCSP
>> request will tell exact
>> > details of
>> > >> what the responder finds in the database
>> or not.
>> > >>
>> > >> Regards,
>> > >> Tomas
>> > >>
>> > >>
>> > >> On July 8, 2019 4:59:06 PM GMT+02:00,
>> ohaya--- via Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >> [Re-sending, as this (handling of
>> large CRLs) is still a
>> > problem and
>> > >> is still not working.]
>> > >>
>> > >>
>> > >>
>> > >> On Monday, July 1, 2019, 12:47:23 PM
>> UTC, <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> I think that I've found how to
>> set/change the
>> > max_allowed_packet on
>> > >> the client (i.e., EJBCA) side, by
>> editing the Mysql URL in the
>> > >>
>> standalone/configuration/standalone.xml to:
>> > >>
>> > >>
>> > >
>> >
>>
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
>> > >>
>> > >> However, when I test the OCSP
>> Responder with a serial
>> > number (and CA
>> > >> cert) from the CA with the large
>> (>71MB) CRL file, I am still
>> > >> getting "unknown" and EJBCA OCSP
>> Responder is still NOT
>> > FINDING the
>> > >> cert serial number.
>> > >>
>> > >> NOTE, again, that I can test with
>> certs (and CA cert) that
>> > have
>> > >> smaller CRL files, and those tests work.
>> > >>
>> > >> So it is only when testing with the
>> large CRL file that is
>> > failing
>> > >> with the OCSP Responder.
>> > >>
>> > >> This seems like a potential BUG for
>> EJBCA OCSP Responder?
>> > >>
>> > >> Please advise.
>> > >>
>> > >> Thanks,
>> > >> Jim
>> > >> On Saturday, June 29, 2019, 3:36:18
>> PM EDT,
>> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> BTW, just to check, I have actually
>> gone through the
>> > process of
>> > >> getting the crlData from the MySQL
>> using mysql client (I
>> > had to add
>> > >> the --max_allowed-packet=100M to the
>> mysql command) and I
>> > was able
>> > >> to actually get the PEM from the
>> Mysql database that way.
>> > >>
>> > >> So, I KNOW that the entire CRL is in
>> the Mysql database,
>> > and yet
>> > >> when I run an "openssl ocsp" test
>> against the EJBCA OCSP
>> > Responder,
>> > >> it says it cannot find the revocation.
>> > >>
>> > >> OCSP Request Data:
>> > >> Version: 1 (0x0)
>> > >> Requestor List:
>> > >> Certificate ID:
>> > >> Hash Algorithm: sha1
>> > >> Issuer Name Hash:
>> 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
>> > >> Issuer Key Hash:
>> 3929A80864E28F83AA816147119133F85A37A6E4
>> > >> Serial Number: 2FFFF4
>> > >> Request Extensions:
>> > >> OCSP Nonce:
>> > >> 04103054A60A55E389B239A55228033D8F62
>> > >> Response verify OK
>> > >> 0x2FFFF4: unknown
>> > >> This Update: Jun 29 19:31:02 2019 GMT
>> > >>
>> > >>
>> > >> In the server log:
>> > >>
>> > >> 2019-06-29 15:31:02,150 DEBUG
>> > >>
>> >
>>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
>> > >> (default task-1) Unable to find
>> revocation information for
>> > >> certificate with serial '2ffff4' from
>> issuer 'CN=XXX....'
>> > >>
>> > >>
>> > >> Honestly, I am no 100% sure that the
>> problem is that
>> > >> max_allowe_packet is too small on the
>> mysql client side,
>> > but that is
>> > >> my best guess.
>> > >>
>> > >> What I AM sure is that the CRL is in
>> mysql, but when I do
>> > a test
>> > >> with OCSP Responder, it cannot find
>> the cert serial number
>> > in the CRL.
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >> On Saturday, June 29, 2019, 4:52:20
>> AM EDT,
>> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi Tomas,
>> > >>
>> > >> I already had the max_allowed_packet
>> configured on the Mysql
>> > >> server-side (in /etc/my.cnf).
>> > >>
>> > >> So as I said, when testing with mysql
>> (client) I had to
>> > include
>> > >> "--max_allowed_packet=100M" on the
>> mysql (client) command
>> > line, e.g.:
>> > >>
>> > >> mysql -u root -p
>> --max_allowed_packet=100M
>> > >>
>> > >> If I did NOT include the
>> "--max_allowed_packet=100M" on
>> > that command
>> > >> line, then when I did a "select" on
>> the field base64Crl, I
>> > would get
>> > >> an error.
>> > >>
>> > >> I suspect that EJBCA is acting as a
>> Mysql client, so, in
>> > order to
>> > >> work with such large CRLs, there
>> needs to be some way to
>> > set the
>> > >> equivalent of the
>> "--max_allowed_packet=100M" on the EJBCA
>> > side.
>> > >>
>> > >> How can I do that?
>> > >>
>> > >> If that can't be done, then I think
>> that when EJBCA tries to
>> > >> retrieve the CRL from the Mysql, it
>> (EJBCA) will see the
>> > same error.
>> > >>
>> > >> Thanks,
>> > >> Jim
>> > >>
>> > >> On Saturday, June 29, 2019, 3:25:47
>> AM EDT, Tomas Gustavsson
>> > >> <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>> > <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>>> wrote:
>> > >>
>> > >>
>> > >>
>> > >> You configure max allowed packet size
>> in you MySQL
>> > configuration on the
>> > >> MySQL server.
>> > >>
>> > >> /Tomas
>> > >>
>> > >> On 2019-06-29 02:19, o haya via
>> Ejbca-develop wrote:
>> > >> > I am not 100% sure, but I think I
>> know what might be
>> > causing this
>> > >> problem.
>> > >> >
>> > >> > I did a test, where I just tried to
>> log into mysql and
>> > do the
>> > >> following
>> > >> > select:
>> > >> >
>> > >> > select base64Crl INTO OUTFILE
>> > '/var/lib/mysql-files/xx.out' from
>> > >> CRLData
>> > >> > where cRLNumber=3167;
>> > >> >
>> > >> > and I got an error saying that the
>> result was more
>> > >> than max_allowed_packet.
>> > >> >
>> > >> > So then I did the same thing, but
>> for the mysql command
>> > I used:
>> > >> >
>> > >> > mysql -u root -p
>> --max_allowed_packet=100M
>> > >> >
>> > >> > and then when I did that same
>> select, I was able to
>> > output the entire
>> > >> > CRL as a PEM.
>> > >> >
>> > >> > So, I am theorizing that that is
>> what is happening with
>> > EJBCA, i.e.,
>> > >> > when I run the "openssl ocsp"
>> command, EJBCA is querying the
>> > > base64CRL
>> > >> > but the result is too large and so
>> that is causing the
>> > query to fail.
>> > >> >
>> > >> > So the question now is: How can I
>> cause EJBCA OCSP
>> > Responder to run
>> > >> > with a larger max_allowed_packet so
>> that the query
>> > succeeds (and then
>> > >> > hopefully EJBCA OCSP Responder will
>> be able to check the
>> > revocation
>> > >> > properly)?
>> > >> >
>> > >> > Does anyone know how that can be done?
>> > >> >
>> > >> > Thanks,
>> > >> > Jim
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Friday, June 28, 2019, 07:16:39
>> PM UTC, o haya
>> > > <oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...
[truncated message content] |
|
From: Tomas G. <to...@pr...> - 2019-07-19 08:18:19
|
Ah, eap 7.2 corresponds to wildly 14 as base.
Yes, run that to convert, then you should be able to create the index.
Cheers,
Tomas
On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
> Hi Tomas,
>No, I am not using Wildfly, but rather JBOSS... specifically
>"jboss-eap-7.2".
>So I restored my snapshot from just after I had deleted all the rows in
>CertificateData that were created before.
>And it looks like all the tables in the "ejbca" database are MyISAM.
>So I found a one-liner that will convert them all to InnoDB:
>mysql -u root -p ejbca -e "show table status where Engine='MyISAM';" |
>awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql -u
>root -p ejbca
>Should I run that and convert all the tables in the "ejbca" to
>InnoDB???
>
>And then try to run the create indexes SQL script?
>
>Thanks for all your help!
>Jim
>
>
>
>
>On Friday, July 19, 2019, 8:01:15 AM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>
>Yes, MyISAM is horrible and can not be used.
>
>I'm not sure about the exact commands to change it. But I think there
>are good internet guides.
>
>The reason you have MyISAM is probably because you are using WildFly 14
>(are you?). There was a change in Hibernate that caused this issue with
>automatic database creation.
>
>One way to modify is to mysqldump the database, change enging in the
>sql
>file and restore it.
>
>Regards,
>Tomas
>
>On 2019-07-19 09:19, ohaya--- via Ejbca-develop wrote:
>> Hi,
>>
>> Mysql version: Server version: 5.7.26 MySQL Community Server (GPL)
>>
>>
>>
>>
>> mysql> show create table CertificateData;
>>
>>
>>
>> | CertificateData | CREATE TABLE `CertificateData` (
>> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `certificateProfileId` int(11) NOT NULL,
>> `endEntityProfileId` int(11) DEFAULT NULL,
>> `expireDate` bigint(20) NOT NULL,
>> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `notBefore` bigint(20) DEFAULT NULL,
>> `revocationDate` bigint(20) NOT NULL,
>> `revocationReason` int(11) NOT NULL,
>> `rowVersion` int(11) NOT NULL,
>> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `status` int(11) NOT NULL,
>> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
>NULL,
>> `type` int(11) NOT NULL,
>> `updateTime` bigint(20) NOT NULL,
>> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
>NULL,
>> `base64Cert` longtext,
>> `rowProtection` longtext,
>> PRIMARY KEY (`fingerprint`),
>> KEY `certificatedata_idx2` (`username`)
>> ) ENGINE=MyISAM DEFAULT CHARSET=utf8 |
>>
>>
>>
>> AH! Yes mine is showing as MyISAM and not InnoDB!
>>
>>
>>
>> Do I need to change the engine to InnoDB? And, if so, how do I do
>that?
>>
>> Do I need to do that for every table in the ejbca database?
>>
>>
>> Thanks,
>> Jim
>>
>>
>>
>>
>> On Friday, July 19, 2019, 6:20:23 AM UTC, Tomas Gustavsson
>> <to...@pr...> wrote:
>>
>>
>>
>> That looks about right. What version of database are you using? I
>just
>> tried the index command (on line 23 in the SQL file) myself and it
>works.
>>
>> CREATE INDEX certificatedata_idx4 ON CertificateData (subjectDN);
>>
>>
>> What does your database creation say?
>>
>>>show create table CertiifcateData;
>>
>> It should look something like this:
>>
>> | CertificateData | CREATE TABLE `CertificateData` (
>> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `certificateProfileId` int(11) NOT NULL,
>> `crlPartitionIndex` int(11) DEFAULT NULL,
>> `endEntityProfileId` int(11) DEFAULT NULL,
>> `expireDate` bigint(20) NOT NULL,
>> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `notBefore` bigint(20) DEFAULT NULL,
>> `revocationDate` bigint(20) NOT NULL,
>> `revocationReason` int(11) NOT NULL,
>> `rowVersion` int(11) NOT NULL,
>> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `status` int(11) NOT NULL,
>> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
>NULL,
>> `type` int(11) NOT NULL,
>> `updateTime` bigint(20) NOT NULL,
>> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
>NULL,
>> `base64Cert` longtext,
>> `certificateRequest` longtext,
>> `rowProtection` longtext,
>> PRIMARY KEY (`fingerprint`)
>> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
>>
>> Is the database engine wrong? I.e. not InnoDB?
>>
>> I am importing the CRL right now and processing about 27 records per
>> second on my laptop.
>>
>> Regards,
>> Tomas
>>
>>
>> On 2019-07-18 18:09, ohaya--- via Ejbca-develop wrote:
>>> According to my notes I used these commands:
>>>
>>> |*/CREATE DATABASE ejbca CHARACTER SET utf8 COLLATE
>utf8_general_ci;/*|
>>>
>>> |*/ /*|
>>>
>>> |*/GRANT ALL PRIVILEGES ON ejbca.* TO 'ejbca'@'localhost' IDENTIFIED
>BY
>>> 'ejbca';/*|
>>>
>>>
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 3:37:51 PM UTC, Tomas Gustavsson
>>> <to...@pr... <mailto:to...@pr...>> wrote:
>>>
>>>
>>> What command did you use to create the database?
>>>
>>> On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop
>>> <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi,
>>>
>>> This is really getting frustrating!!!
>>>
>>> [root@ejbca <mailto:root@ejbca> sql-scripts]# mysql -u root
>> -p'xxxxxx' ejbca <
>>> create-index-ejbca.sql
>>> mysql: [Warning] Using a password on the command line interface
>can
>>> be insecure.
>>> ERROR 1071 (42000) at line 23: Specified key was too long; max
>key
>>> length is 1000 bytes
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya...
>> <mailto:oh...@ya...>
>>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> The "machine" is a virtual machine under VirtualBox on a machine
>I
>>> have that has 64GB physical RAM. Drives are all 7200 RPM drives.
>>>
>>> I will restore the snapshot, then increase the memory and # of
>>> processors and try to run the create index again.
>>>
>>> Jim
>>>
>>> On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson
>>> <to...@pr... <mailto:to...@pr...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> Just run create index script.
>>>
>>> Create index should run pretty fast even with millions of records
>in
>>> the database. I suspect that your database is configured very
>slow,
>>> perhaps running everything on disl, without RAM, using slow
>disks?
>>>
>>> A standard installation of mysql/mariadb on a centos machine
>should
>>> run faster. Then you have pretty low amount of ram for running
>>> everything on one box, handing millions of records.
>>>
>>> Regards,
>>> Tomas
>>>
>>> On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop
>>> <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi,
>>>
>>> I had taken a snapshot of the machine before I tried to do
>the
>>> index earlier, so I restored that snapshot.
>>>
>>> Then, I tried running the "optimize-ejbca-mysql.sql" and I
>got a
>>> different/new error:
>>>
>>> ERROR 1091 (42000) at line 27: Can't DROP
>>> 'certificatedata_idx5'; check that column/key exists
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
>> <mailto:oh...@ya...>
>>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> Oh oh, I just got this from the mysql execution:
>>>
>>> ERROR 1071 (42000) at line 23: Specified key was too long;
>max
>>> key length is 1000 bytes
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
>> <mailto:oh...@ya...>
>>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> FYI, I am running the create-index-ejbca.sql script now, but
>>> that seems to be taking FOREVER?
>>>
>>> I had stopped the earlier import, and then deleted all the
>rows
>>> for that CA DN in CertificateData table, and then bounced the
>>> machine, then I ran the create-index-ejbca.sql.
>>>
>>> Why is it taking so long? Should I have done something in
>>> addition to deleting the rows? I am guessing that the reason
>>> the sql script is taking so long is that it's somehow still
>>> trying to index the rows were deleted or something like that?
>>>
>>> Thanks,
>>> Jim
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
>> <mailto:oh...@ya...>
>>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> I notice that there is:
>>>
>>> - create-index-ejbca.sql
>>> - optimize-ejbca-mysql.sql
>>>
>>> Should I run BOTH of them, or just the
>"create-index-ejbca.sql "?
>>>
>>> Thanks,
>>> Jim
>>>
>>>
>>> On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
>>> <to...@pr... <mailto:to...@pr...>> wrote:
>>>
>>>
>>> Yes, i could see from your output from top that mysql takes
>all
>>> CPU. This indicates that you don't have any indexes in your
>>> database, which makes it horribly slow as numbers grow.
>>>
>>> Apply database indexes from doc/sql-scripts.
>>>
>>> Regards,
>>> Tomas
>>>
>>>
>>> On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via
>>> Ejbca-develop <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi,
>>>
>>> So far, processed only 185706 entries, and it looks like
>the
>>> processing is going to a crawl (really slow now).
>>>
>>> Which kind of makes me wonder if maybe part of reason for
>>> the slowness is that it is having to search the
>certificate
>>> database for the serial number, and now, as there are
>more
>>> dummy certs in the database, that search is taking longer
>>> and longer?
>>>
>>> Jim
>>>
>>>
>>> On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via
>>> Ejbca-develop <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> This is what top is showing now (the import is still
>running):
>>>
>>> top - 13:56:19 up 1 day, 1:09, 3 users, load average:
>>> 1.18, 1.11, 1.07
>>> Tasks: 167 total, 2 running, 165 sleeping, 0
>stopped,
>>> 0 zombie
>>> %Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa,
>0.0
>>> hi, 0.3 si, 0.0 st
>>> KiB Mem : 4045880 total, 123192 free, 3053072 used,
>>> 869616 buff/cache
>>> KiB Swap: 3905532 total, 3753980 free, 151552 used.
>>> 688136 avail Mem
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU
>>> %MEM TIME+ COMMAND
>>> 1489 mysql 20 0 1580320 129624 4396 S 92.7
>3.2
>>> 1227:41 mysqld
>>> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2
>>> 139:05.67 java
>>> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3
>>> 176:26.68 java
>>> 22466 root 20 0 0 0 0 S 0.7
>0.0
>>> 0:00.73 kworker/u4:1
>>>
>>> That is running on 2-CPU machine with 4GB RAM.
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>> On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas
>>> Gustavsson <to...@pr... <mailto:to...@pr...>>
>> wrote:
>>>
>>>
>>> Yes, like top.
>>>
>>>
>>> On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via
>>> Ejbca-develop <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi Tomas,
>>>
>>> I'm sorry that I missed the message below, but I am
>not
>>> clear what you are asking me to check/monitor? Do
>you
>>> mean like running something like "top" command?
>>>
>>> Thanks,
>>> Jim
>>>
>>> On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas
>>> Gustavsson <to...@pr...
>> <mailto:to...@pr...>> wrote:
>>>
>>>
>>> To verify that the changes actually happen...
>>>
>>> Do you monitor CPU usage to see what processes use
>CPU
>>> and memory?
>>>
>>>
>>> On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via
>>> Ejbca-develop <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi,
>>>
>>> The import has been running for about an hour now
>>> and has imported 17836 entries.
>>>
>>> Per what I had said previously, this CRL file has
>>> 1002255 entries, that means it would take about
>56+
>>> hours to import this CRL.
>>>
>>> So, this import processing is still way too slow.
>>>
>>> I don't know what is involved, or even how much
>it
>>> would improve things, but I think that the next
>>> thing to try is to eliminated both the check for
>the
>>> serial number in the DB AND the dummy cert
>creation.
>>>
>>> Jim
>>>
>>>
>>> On Tuesday, July 16, 2019, 5:54:54 PM UTC,
>ohaya---
>>> via Ejbca-develop
>>> <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> Ok, I rebuilt the JAR using your Java class, and
>it
>>> is running now, and creating rows in the
>>> CertificateData, but it still SEEMS kind of
>slow.
>>>
>>> I am running now and will try to get a
>measurement
>>> of revoked entries/time and will post back.
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>> On Tuesday, July 16, 2019, 5:25:32 PM UTC,
>>> oh...@ya... <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> So I just tried what I said (emptying the "if")
>and
>>> you are right, it doesn't create the rows in the
>>> CeritificateData table :(...
>>>
>>> Ok, I will try your version...
>>>
>>> Jim
>>>
>>> On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas
>>> Gustavsson <to...@pr...
>> <mailto:to...@pr...>> wrote:
>>>
>>>
>>>
>>> Hi,
>>>
>>> I don't think so, no quite actually. In order to
>use
>>> the API to populate
>>> the database you need the certificate object it
>>> seems. Something like
>>> the attached file should work, it still creates a
>>> dummy certificate, but
>>> uses a static hard coded key, so no key
>generation.
>>>
>>> The file assumes you are on 6.15.x.
>>>
>>> Regards,
>>> Tomas
>>>
>>> On 2019-07-16 15:02, ohaya--- via Ejbca-develop
>wrote:
>>> > For testing (only...), I am thinking that if I
>>> modify this If clause in
>>> > CaImportCrlCommand.java:
>>> >
>>> > if (adaptive)
>>> > {
>>> > Date time = new Date();
>>> > KeyPair key_pair =
>>> KeyTools.genKeys("2048", "RSA");
>>> >
>>> > SubjectPublicKeyInfo pkinfo =
>>> >
>>>
>> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
>>> > X500Name dnName = new
>>> X500Name("CN=Dummy Missing in
>>> > Imported CRL, serialNumber=" + serialHex);
>>> > Date notAfter = new
>>> Date(time.getTime() + 315360000000L);
>>> > X509v3CertificateBuilder
>certbuilder
>>> = new
>>> >
>X509v3CertificateBuilder(X500Name.getInstance(cacert
>>> >
>>> .getSubjectX500Principal().getEncoded()),
>serialNr,
>>> > time, notAfter, dnName, pkinfo);
>>> > ContentSigner signer = new
>>> BufferingContentSigner(new
>>> >
>>>
>>
>JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
>>> > .getPrivate()), 20480);
>>> > X509CertificateHolder certHolder
>=
>>> certbuilder.build(signer);
>>> > X509Certificate certificate =
>>> >
>>>
>>
>(X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
>>> > X509Certificate.class);
>>> >
>>> > String fingerprint =
>>> > CertTools.getFingerprintAsString(certificate);
>>> >
>>> > EndEntityInformation
>>> missingUserEndEntityInformation =
>>> >
>>>
>>
>((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
>>> > missing_user_name);
>>> > if
>(missingUserEndEntityInformation
>>> == null)
>>> > {
>>> > log.debug("Loading/updating
>user "
>>> + missing_user_name);
>>> >
>>> > EndEntityInformation
>userdataNew
>> = new
>>> > EndEntityInformation(missing_user_name,
>>> > CertTools.getSubjectDN(certificate),
>>> cainfo.getCAId(), null, null, 10,
>>> > new EndEntityType(EndEntityTypes.ENDUSER), 1,
>1,
>>> null, null, 1, 0, null);
>>> >
>>> >
>userdataNew.setPassword("foo123");
>>> >
>>> >
>>>
>>
>((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
>>> > userdataNew, false);
>>> >
>>> > log.info("User '" +
>>> missing_user_name + "' has been
>>> > added.");
>>> >
>>> >
>>>
>>
>((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
>>> > missing_user_name, 50);
>>> >
>>> > log.info("User '" +
>>> missing_user_name + "' has been
>>> > updated.");
>>> > }
>>> >
>>> >
>>>
>>
>((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>>> >
>>> > EJBTools.wrap(certificate),
>>> missing_user_name,
>>> > fingerprint, 20, 1, 1, 0, null, new Date()
>>> >
>>> > .getTime());
>>> > log.info("Dummy certificate '" +
>>> serialHex + "' has been
>>> > stored.");
>>> > }
>>> >
>>> > to (for example):
>>> >
>>> > if (adaptive)
>>> > {
>>> >
>>> > log.info("*** SKIPPING ***
>Dummy
>>> certificate '" +
>>> > serialHex + "' has been stored.");
>>> > }
>>> >
>>> >
>>> > Would THAT work??
>>> >
>>> > Thanks,
>>> > Jim
>>> >
>>> >
>>> >
>>> >
>>> > On Tuesday, July 16, 2019, 10:59:31 AM UTC,
>>> ohaya--- via Ejbca-develop
>>> > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>>> >
>>> >
>>> > Hi Tomas (or anyone else),
>>> >
>>> > Sorry - I was typing my last msg on phone, and
>it
>>> wasn't as detailed or
>>> > clear as I wanted.
>>> >
>>> > Basically what I was saying is I was thinking
>>> that, as a quick sanity
>>> > test, if there was one or two places where the
>> code in
>>> > the CaImportCRLCommand.java was maybe making a
>>> method call to create the
>>> > dummy cert and put it into the database, then
>>> maybe I could decompile
>>> > and modify the CaImportCRLCommand.java, and
>>> comment out those method
>>> > calls, then, re-do my large CRL import test and
>>> time it?
>>> >
>>> > If so, can you let me know specifically where
>>> those calls are, and if I
>>> > just comment them out, would that work?
>>> >
>>> > Ironically, that (not doing the dummy cert
>>> creation and writing) was
>>> > what I *thought* the "LENIENT" option was
>doing).
>>> >
>>> > Thanks,
>>> > Jim
>>> >
>>> > P. S. I have time today to try this if I
>figure
>>> out what needs to be done.
>>> >
>>> >
>>> >
>>> > On Monday, July 15, 2019, 6:24:53 PM UTC,
>ohaya---
>>> via Ejbca-develop
>>> > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>>> >
>>> >
>>> >
>>> > Yes, we are definitely interested/willing to
>test
>>> patches. I actually
>>> > was wondering if it would be a quick test to
>mod
>>> the code somewhere to
>>> > avoid a call that creates the dummy cert, to
>get
>>> an idea if that would
>>> > reduce the import time enough??
>>> >
>>> >
>>> > Sent from Yahoo Mail on Android
>>> >
>>>
>>
><https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>>> >
>>> > On Mon, Jul 15, 2019 at 12:10 PM, Tomas
>Gustavsson
>>> > <to...@pr...
><mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>>> wrote:
>>> > Most large users, some very very large,
>>> typically uses on line
>>> > updates directly from the CA. Many CAs today
>>> don't accept only
>>> > importing CRLs, as they need to keep track
>of
>>> issued certificates as
>>> > well not only revoked (can't use "non
>existing
>>> is good").
>>> > Having said thst, it's interesting to be
>able
>>> to import large CRLs,
>>> > so improving it is of interest.
>>> >
>>> > Importing CRLs would actually not have to
>>> generate a dummy cert at
>>> > all, since a certificate object is not
>needed
>>> on the responder,
>>> > which should make it very fast.
>>> >
>>> > If you are willing to test patches, we can
>work
>>> with you on improving.
>>> >
>>> > Regards,
>>> > Tomas
>>> >
>>> >
>>> > On July 15, 2019 5:26:08 PM GMT+02:00,
>ohaya---
>>> via Ejbca-develop
>>> > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I am curious whether you have other
>users
>>> that import large CRLs
>>> > like this?
>>> >
>>> > Are we the only ones who would encounter
>>> these really long
>>> > import times, because the other users
>who
>>> might have large CRLs
>>> > use the EJBCA CA also, so they don't run
>>> into having the dummy
>>> > certs being created for every entry in
>the
>> CRL?
>>> >
>>> > Jim
>>> >
>>> > On Monday, July 15, 2019, 1:42:57 PM
>UTC,
>>> Tomas Gustavsson
>>> > <to...@pr...
><mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>>> wrote:
>>> >
>>> >
>>> >
>>> > I see...
>>> >
>>> > I can see that a new key pair is being
>>> generated for each dummy
>>> > cert.
>>> >
>>> > In CaImportCRLCommand.java:
>>> >
>>> > �� final KeyPair key_pair =
>>> KeyTools.genKeys("2048",
>>> > AlgorithmConstants.KEYALGORITHM_RSA);
>>> >
>>> > Chaning that to 1024 or 512 would be an
>>> easy improvement. A bit more
>>> > complex, but even larger speedup would
>be
>>> to use a "hard coded"
>>> > dummy
>>> > key, i.e. same key for all dummy certs.
>>> >
>>> > You can experiment by changing 2048 to
>512
>>> (not secure but it
>>> > doesn't
>>> > have to be for this dummy cert).
>>> >
>>> > Regards,
>>> > Tomas
>>> >
>>> > On 2019-07-11 14:19, ohaya--- via
>>> Ejbca-develop wrote:
>>> > > FYI, the is running, but it is REALLY
>>> slow. The CRL file has
>>> > 1002255
>>> > > entries in it, and the import is only
>up
>>> to 8899 now, after
>>> > about 40
>>> > > minutes... which would take about 75
>> hours!!
>>> > >
>>> > > In our production systems, we are
>>> required to update some CRLs
>>> > every
>>> > > hour, so that would be a problem.
>>> > >
>>> > > What can I do to speed that up??
>>> > >
>>> > > Thanks,
>>> > > Jim
>>> > >
>>> > >
>>> > > On Thursday, July 11, 2019, 11:09:33
>AM
>>> UTC, ohaya--- via
>>> > Ejbca-develop
>>> > > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>> wrote:
>>> > >
>>> > >
>>> > > AHHH... Ok, it is running now, and
>from
>>> checking the
>>> > CertificateData in
>>> > > the Mysql database, I can see that it
>is
>>> now populating the
>>> > > CertificateData!!
>>> > >
>>> > > Thanks!
>>> > >
>>> > > Jim
>>> > >
>>> > >
>>> > >
>>> > > On Thursday, July 11, 2019, 11:07:22
>AM
>>> UTC, ohaya--- via
>>> > Ejbca-develop
>>> > > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>> wrote:
>>> > >
>>> > >
>>> > > I wasn't sure if we should use
>ADAPTIVE,
>>> since we are not
>>> > using the CA part.
>>> > >
>>> > >
>>> > > I will try with "-o ADAPTIVE" now.
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Thursday, July 11, 2019, 7:42:14 AM
>>> UTC, Tomas Gustavsson
>>> > > <to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>>>> wrote:
>>> > >
>>> > >
>>> > >
>>> > > It should be ADAPTIVE.
>>> > >
>>> > > The documentation says:
>>> > > bin/ejbca.sh ca importcrl --help
>>> > > ---
>>> > > ADAPTIVE means that missing
>certficates
>>> will be replaced by dummy
>>> > > certificates to cater for proper CRLs
>for
>>> missing certificates.
>>> > > ---
>>> > >
>>> > > Regards,
>>> > > Tomas
>>> > >
>>> > > On 2019-07-11 08:05, ohaya--- via
>>> Ejbca-develop wrote:
>>> > >> Hi,
>>> > >>
>>> > >> I've been looking at the code in
>>> CaImportCRLCommand, and I am
>>> > wondering
>>> > >> if, when the command is run in
>LENIENT
>>> mode and it doesn't
>>> > find the cert
>>> > >> in the database (and outputs the
>>> "missing in database" error
>>> > message),
>>> > >> what is supposed to happen?
>>> > >>
>>> > >> Mind you, the only error messages I
>am
>>> seeing are the "missing in
>>> > >> database" error messages. I don't
>see
>>> any of the other error
>>> > messages
>>> > >> that are in/from
>CaImportCRLCommand.java.
>>> > >>
>>> > >> I am wondering if, when the command
>is
>>> run with "-o LENIENT",
>>> > and when
>>> > >> it outputs the "missing in database"
>>> error message, then the
>>> > ejbca.sh is
>>> > >> basically not doing anything (and not
>>> storing anything into the
>>> > >> CertificateData table)?
>>> > >>
>>> > >> From the lack of any other error
>>> messages (vs. what I see in the
>>> > >> CaImportCRLCommand.java code), that
>is
>>> what it looks like is
>>> > happening.
>>> > >>
>>> > >> So given I am using EJBCA for OCSP
>>> responder only, is running the
>>> > >> ejbca.sh in the "-o LENIENT" mode
>correct?
>>> > >>
>>> > >> Jim
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Thursday, July 11, 2019, 5:43:37
>AM
>>> UTC, ohaya--- via
>>> > Ejbca-develop
>>> > >> <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>
>>> > >
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> I've been doing a bunch of testing
>with
>>> trying to import the
>>> > large CRL,
>>> > >> and wanted to summarize:
>>> > >>
>>> > >> - The ejbca.sh runs fine - it
>completes
>>> without outputting an
>>> > errors,
>>> > >> except for outputting a bunch of the
>>> "cert xxxx missing in
>>> > database"
>>> > >> messages.
>>> > >>
>>> > >> - When the import completes, and I
>check
>>> the Mysql database,
>>> > the crlData
>>> > >> (the full PEM of the CRL) is in the
>>> database, but there is no
>>> > rows in
>>> > >> the CertificateData table for that CA
>at
>>> all.
>>> > >>
>>> > >> - I even figured out how to get the
>>> ejbca.sh to log using
>>> > >> log4j.properties, and it output a
>bunch
>>> of logging, mainly
>>> > the missing
>>> > >> in database errors, but other than
>that
>>> I didn't see any
>>> > obvious errors.
>>> > >>
>>> > >>
>>> > >> So, bottom line is that even though
>the
>>> ejbca.sh seems to run
>>> > without
>>> > >> errors (other than the missing in
>>> database errors), the import is
>>> > >> causing the CRL PEM to be stored in
>the
>>> database, but the
>>> > >> CertificateData table is not being
>>> populated at all for that CA.
>>> > >>
>>> > >>
>>> > >> Jim
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Wednesday, July 10, 2019, 7:40:16
>PM
>>> UTC, ohaya--- via
>>> > Ejbca-develop
>>> > >> <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>
>>> > >
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> Ok, I found Tomas' original
>message...
>>> it was in my Spam
>>> > folder....
>>> > >>
>>> > >> I also tried re-running the
>"./ejbca.sh
>>> ca importcrl" command
>>> > a bunch of
>>> > >> ways. I think it is running without
>>> outputting any errors,
>>> > but still,
>>> > >> no rows are appearing with that
>issuerDN
>>> at all.
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Tuesday, July 9, 2019, 10:45:01 AM
>>> UTC, Tomas Gustavsson
>>> > >> <to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
><mailto:to...@pr...>>>
>>> > <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> OCSP responses does not have anything
>to
>>> do with the CRL
>>> > itself. Once
>>> > >> the CRL has been imported each
>>> certificate status is in its
>>> > own record
>>> > >> in the CertificateData table. This is
>>> where status is read
>>> > when reponses
>>> > >> are created.
>>> > >> If there are issues, it is in that
>case
>>> related to the import
>>> > of the CRL.
>>> > >>
>>> > >> Verify that certificate records are
>>> created in
>>> > CertificateData when
>>> > >> importing the CRL.
>>> > >> Also, a debug log when making OCSP
>>> request will tell exact
>>> > details of
>>> > >> what the responder finds in the
>database
>>> or not.
>>> > >>
>>> > >> Regards,
>>> > >> Tomas
>>> > >>
>>> > >>
>>> > >> On July 8, 2019 4:59:06 PM GMT+02:00,
>>> ohaya--- via Ejbca-develop
>>> > >> <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>
>>> > >
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>>> wrote:
>>> > >>
>>> > >> [Re-sending, as this (handling of
>>> large CRLs) is still a
>>> > problem and
>>> > >> is still not working.]
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Monday, July 1, 2019, 12:47:23
>PM
>>> UTC, <oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>>> > <mailto:oh...@ya...
>> <mailto:oh...@ya...>
>>> <mailto:oh...@ya...
><mailto:oh...@ya...>>>
>>> > > <mailto:oh...@ya...
>> <mailto:oh...@ya...>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
>>> <mailto:oh...@ya...
>> <mailto:oh...@ya...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> I think that I've found how to
>>> set/change the
>>> > max_allowed_packet on
>>> > >> the client (i.e., EJBCA) side, by
>>> editing the Mysql URL in the
>>> > >>
>>> standalone/configuration/standalone.xml to:
>>> > >>
>>> > >>
>>> > >
>>> >
>>>
>>
><connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
>>> > >>
>>> > >> However, when I test the OCSP
>>> Responder with a serial
>>> > number (and CA
>>> > >> cert) from the CA with the large
>>> (>71MB) CRL file, I am still
>>> > >> getting "unknown" and EJBCA OCSP
>>> Responder is still NOT
>>> > FINDING the
>>> > >> cert serial number.
>>> > >>
>>> > >> NOTE, again, that I can test with
>>> certs (and CA cert) that
>>> > have
>>> > >> smaller CRL files, and those tests
>work.
>>> > >>
>>> > >> So it is only when testing with
>the
>>> large CRL file that is
>>> > failing
>>> > >> with the OCSP Responder.
>>> > >>
>>> > >> This seems like a potential BUG
>for
>>> EJBCA OCSP Responder?
>>> > >>
>>> > >> Please advise.
>>> > >>
>>> > >> Thanks,
>>> > >> Jim
>>> > >> On Saturday, June 29, 2019,
>3:36:18
>>> PM EDT,
>>> > <oh...@ya...
><mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>>> > > <mailto:oh...@ya...
>> <mailto:oh...@ya...>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
>>> <mailto:oh...@ya...
>> <mailto:oh...@ya...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> BTW, just to check, I have
>actually
>>> gone through the
>>> > process of
>>> > >> getting the crlData from the MySQL
>>> using mysql client (I
>>> > had to add
>>> > >> the --max_allowed-packet=100M to
>the
>>> mysql command) and I
>>> > was able
>>> > >> to actually get the PEM from the
>>> Mysql database that way.
>>> > >>
>>> > >> So, I KNOW that the entire CRL is
>in
>>> the Mysql database,
>>> > and yet
>>> > >> when I run an "openssl ocsp" test
>>> against the EJBCA OCSP
>>> > Responder,
>>> > >> it says it cannot find the
>revocation.
>>> > >>
>>> > >> OCSP Request Data:
>>> > >> Version: 1 (0x0)
>>> > >> Requestor List:
>>> > >> Certificate ID:
>>> > >> Hash Algorithm: sha1
>>> > >> Issuer Name Hash:
>>> 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
>>> > >> Issuer Key Hash:
>>> 3929A80864E28F83AA816147119133F85A37A6E4
>>> > >> Serial Number: 2FFFF4
>>> > >> Request Extensions:
>>> > >> OCSP Nonce:
>>> > >>
>04103054A60A55E389B239A55228033D8F62
>>> > >> Response verify OK
>>> > >> 0x2FFFF4: unknown
>>> > >> This Update: Jun 29 19:31:02 2019
>GMT
>>> > >>
>>> > >>
>>> > >> In the server log:
>>> > >>
>>> > >> 2019-06-29 15:31:02,150 DEBUG
>>> > >>
>>> >
>>>
>> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
>>> > >> (default task-1) Unable to find
>>> revocation information for
>>> > >> certificate with serial '2ffff4'
>from
>>> issuer 'CN=XXX....'
>>> > >>
>>> > >>
>>> > >> Honestly, I am no 100% sure that
>the
>>> problem is that
>>> > >> max_allowe_packet is too small on
>the
>>> mysql client side,
>>> > but that is
>>> > >> my best guess.
>>> > >>
>>> > >> What I AM sure is that the CRL is
>in
>>> mysql, but when I do
>>> > a test
>>> > >> with OCSP Responder, it cannot
>find
>>> the cert serial number
>>> > in the CRL.
>>> > >>
>>> > >> Jim
>>> > >>
>>> > >>
>>> > >> On Saturday, June 29, 2019,
>4:52:20
>>> AM EDT,
>>> > <oh...@ya...
><mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>>> > > <mailto:oh...@ya...
>> <mailto:oh...@ya...>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
>>> <mailto:oh...@ya...
>> <mailto:oh...@ya...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi Tomas,
>>> > >>
>>> > >> I already had the
>max_allowed_packet
>>> configured on the Mysql
>>> > >> server-side (in /etc/my.cnf).
>>> > >>
>>> > >> So as I said, when testing with
>mysql
>>> (client) I had to
>>> > include
>>> > >> "--max_allowed_packet=100M" on the
>>> mysql (client) command
>>> > line, e.g.:
>>> > >>
>>> > >> mysql -u root -p
>>> --max_allowed_packet=100M
>>> > >>
>>> > >> If I did NOT include the
>>> "--max_allowed_packet=100M" on
>>> > that command
>>> > >> line, then when I did a "select"
>on
>>> the field base64Crl, I
>>> > would get
>>> > >> an error.
>>> > >>
>>> > >> I suspect that EJBCA is acting as
>a
>>> Mysql client, so, in
>>> > order to
>>> > >> work with such large CRLs, there
>>> needs to be some way to
>>> > set the
>>> > >> equivalent of the
>>> "--max_allowed_packet=100M" on the EJBCA
>>> > side.
>>> > >>
>>> > >> How can I do that?
>>> > >>
>>> > >> If that can't be done, then I
>think
>>> that when EJBCA tries to
>>> > >> retrieve the CRL from the Mysql,
>it
>>> (EJBCA) will see the
>>> > same error.
>>> > >>
>>> > >> Thanks,
>>> > >> Jim
>>> > >>
>>> > >> On Saturday, June 29, 2019,
>3:25:47
>>> AM EDT, Tomas Gustavsson
>>> > >> <to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
><mailto:to...@pr...>>>
>>> > <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >>
>>> > >> You configure max allowed packet
>size
>>> in you MySQL
>>> > configuration on the
>>> > >> MySQL server.
>>> > >>
>>> > >> /Tomas
>>> > >>
>>> > >> On 2019-06-29 02:19, o haya via
>>> Ejbca-develop wrote:
>>> > >> > I am not 100% sure, but I think
>I
>>> know what might be
>>> > causing this
>>> > >> problem.
>>> > >> >
>>> > >> > I did a test, where I just tried
>to
>>> log into mysql and
>>> > do the
>>> > >> following
>>> > >> > select:
>>> > >> >
>>> > >> > select base64Crl INTO OUTFILE
>>> > '/var/lib/mysql-files/xx.out' from
>>> > >> CRLData
>>> > >> > where cRLNumber=3167;
>>> > >> >
>>> > >> > and I got an error saying that
>the
>>> result was more
>>> > >> than max_allowed_packet.
>>> > >> >
>>> > >> > So then I did the same thing,
>but
>>> for the mysql command
>>> > I used:
>>> > >> >
>>> > >> > mysql -u root -p
>>> --max_allowed_packet=100M
>>> > >> >
>>> > >> > and then when I did that same
>>> select, I was able to
>>> > output the entire
>>> > >> > CRL as a PEM.
>>> > >> >
>>> > >> > So, I am theorizing that that is
>>> what is happening with
>>> > EJBCA, i.e.,
>>> > >> > when I run the "openssl ocsp"
>>> command, EJBCA is querying the
>>> > > base64CRL
>>> > >> > but the result is too large and
>so
>>> that is causing the
>>> > query to fail.
>>> > >> >
>>> > >> > So the question now is: How can
>I
>>> cause EJBCA OCSP
>>> > Responder to run
>>> > >> > with a larger max_allowed_packet
>so
>>> that the query
>>> > ...
[truncated message content] |
|
From: <oh...@ya...> - 2019-07-19 08:07:23
|
Hi Tomas,
No, I am not using Wildfly, but rather JBOSS... specifically "jboss-eap-7.2".
So I restored my snapshot from just after I had deleted all the rows in CertificateData that were created before.
And it looks like all the tables in the "ejbca" database are MyISAM.
So I found a one-liner that will convert them all to InnoDB:
mysql -u root -p ejbca -e "show table status where Engine='MyISAM';" | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql -u root -p ejbca
Should I run that and convert all the tables in the "ejbca" to InnoDB???
And then try to run the create indexes SQL script?
Thanks for all your help!
Jim
On Friday, July 19, 2019, 8:01:15 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, MyISAM is horrible and can not be used.
I'm not sure about the exact commands to change it. But I think there
are good internet guides.
The reason you have MyISAM is probably because you are using WildFly 14
(are you?). There was a change in Hibernate that caused this issue with
automatic database creation.
One way to modify is to mysqldump the database, change enging in the sql
file and restore it.
Regards,
Tomas
On 2019-07-19 09:19, ohaya--- via Ejbca-develop wrote:
> Hi,
>
> Mysql version: Server version: 5.7.26 MySQL Community Server (GPL)
>
>
>
>
> mysql> show create table CertificateData;
>
>
>
> | CertificateData | CREATE TABLE `CertificateData` (
> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `certificateProfileId` int(11) NOT NULL,
> `endEntityProfileId` int(11) DEFAULT NULL,
> `expireDate` bigint(20) NOT NULL,
> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `notBefore` bigint(20) DEFAULT NULL,
> `revocationDate` bigint(20) NOT NULL,
> `revocationReason` int(11) NOT NULL,
> `rowVersion` int(11) NOT NULL,
> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `status` int(11) NOT NULL,
> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `type` int(11) NOT NULL,
> `updateTime` bigint(20) NOT NULL,
> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `base64Cert` longtext,
> `rowProtection` longtext,
> PRIMARY KEY (`fingerprint`),
> KEY `certificatedata_idx2` (`username`)
> ) ENGINE=MyISAM DEFAULT CHARSET=utf8 |
>
>
>
> AH! Yes mine is showing as MyISAM and not InnoDB!
>
>
>
> Do I need to change the engine to InnoDB? And, if so, how do I do that?
>
> Do I need to do that for every table in the ejbca database?
>
>
> Thanks,
> Jim
>
>
>
>
> On Friday, July 19, 2019, 6:20:23 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> That looks about right. What version of database are you using? I just
> tried the index command (on line 23 in the SQL file) myself and it works.
>
> CREATE INDEX certificatedata_idx4 ON CertificateData (subjectDN);
>
>
> What does your database creation say?
>
>>show create table CertiifcateData;
>
> It should look something like this:
>
> | CertificateData | CREATE TABLE `CertificateData` (
> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `certificateProfileId` int(11) NOT NULL,
> `crlPartitionIndex` int(11) DEFAULT NULL,
> `endEntityProfileId` int(11) DEFAULT NULL,
> `expireDate` bigint(20) NOT NULL,
> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `notBefore` bigint(20) DEFAULT NULL,
> `revocationDate` bigint(20) NOT NULL,
> `revocationReason` int(11) NOT NULL,
> `rowVersion` int(11) NOT NULL,
> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `status` int(11) NOT NULL,
> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `type` int(11) NOT NULL,
> `updateTime` bigint(20) NOT NULL,
> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `base64Cert` longtext,
> `certificateRequest` longtext,
> `rowProtection` longtext,
> PRIMARY KEY (`fingerprint`)
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
>
> Is the database engine wrong? I.e. not InnoDB?
>
> I am importing the CRL right now and processing about 27 records per
> second on my laptop.
>
> Regards,
> Tomas
>
>
> On 2019-07-18 18:09, ohaya--- via Ejbca-develop wrote:
>> According to my notes I used these commands:
>>
>> |*/CREATE DATABASE ejbca CHARACTER SET utf8 COLLATE utf8_general_ci;/*|
>>
>> |*/ /*|
>>
>> |*/GRANT ALL PRIVILEGES ON ejbca.* TO 'ejbca'@'localhost' IDENTIFIED BY
>> 'ejbca';/*|
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 3:37:51 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> What command did you use to create the database?
>>
>> On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> This is really getting frustrating!!!
>>
>> [root@ejbca <mailto:root@ejbca> sql-scripts]# mysql -u root
> -p'xxxxxx' ejbca <
>> create-index-ejbca.sql
>> mysql: [Warning] Using a password on the command line interface can
>> be insecure.
>> ERROR 1071 (42000) at line 23: Specified key was too long; max key
>> length is 1000 bytes
>>
>> Jim
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> The "machine" is a virtual machine under VirtualBox on a machine I
>> have that has 64GB physical RAM. Drives are all 7200 RPM drives.
>>
>> I will restore the snapshot, then increase the memory and # of
>> processors and try to run the create index again.
>>
>> Jim
>>
>> On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Hi,
>>
>> Just run create index script.
>>
>> Create index should run pretty fast even with millions of records in
>> the database. I suspect that your database is configured very slow,
>> perhaps running everything on disl, without RAM, using slow disks?
>>
>> A standard installation of mysql/mariadb on a centos machine should
>> run faster. Then you have pretty low amount of ram for running
>> everything on one box, handing millions of records.
>>
>> Regards,
>> Tomas
>>
>> On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> I had taken a snapshot of the machine before I tried to do the
>> index earlier, so I restored that snapshot.
>>
>> Then, I tried running the "optimize-ejbca-mysql.sql" and I got a
>> different/new error:
>>
>> ERROR 1091 (42000) at line 27: Can't DROP
>> 'certificatedata_idx5'; check that column/key exists
>>
>> Jim
>>
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> Oh oh, I just got this from the mysql execution:
>>
>> ERROR 1071 (42000) at line 23: Specified key was too long; max
>> key length is 1000 bytes
>>
>> Jim
>>
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> FYI, I am running the create-index-ejbca.sql script now, but
>> that seems to be taking FOREVER?
>>
>> I had stopped the earlier import, and then deleted all the rows
>> for that CA DN in CertificateData table, and then bounced the
>> machine, then I ran the create-index-ejbca.sql.
>>
>> Why is it taking so long? Should I have done something in
>> addition to deleting the rows? I am guessing that the reason
>> the sql script is taking so long is that it's somehow still
>> trying to index the rows were deleted or something like that?
>>
>> Thanks,
>> Jim
>>
>>
>>
>> On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> I notice that there is:
>>
>> - create-index-ejbca.sql
>> - optimize-ejbca-mysql.sql
>>
>> Should I run BOTH of them, or just the "create-index-ejbca.sql "?
>>
>> Thanks,
>> Jim
>>
>>
>> On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Yes, i could see from your output from top that mysql takes all
>> CPU. This indicates that you don't have any indexes in your
>> database, which makes it horribly slow as numbers grow.
>>
>> Apply database indexes from doc/sql-scripts.
>>
>> Regards,
>> Tomas
>>
>>
>> On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> So far, processed only 185706 entries, and it looks like the
>> processing is going to a crawl (really slow now).
>>
>> Which kind of makes me wonder if maybe part of reason for
>> the slowness is that it is having to search the certificate
>> database for the serial number, and now, as there are more
>> dummy certs in the database, that search is taking longer
>> and longer?
>>
>> Jim
>>
>>
>> On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> This is what top is showing now (the import is still running):
>>
>> top - 13:56:19 up 1 day, 1:09, 3 users, load average:
>> 1.18, 1.11, 1.07
>> Tasks: 167 total, 2 running, 165 sleeping, 0 stopped,
>> 0 zombie
>> %Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0
>> hi, 0.3 si, 0.0 st
>> KiB Mem : 4045880 total, 123192 free, 3053072 used,
>> 869616 buff/cache
>> KiB Swap: 3905532 total, 3753980 free, 151552 used.
>> 688136 avail Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU
>> %MEM TIME+ COMMAND
>> 1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2
>> 1227:41 mysqld
>> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2
>> 139:05.67 java
>> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3
>> 176:26.68 java
>> 22466 root 20 0 0 0 0 S 0.7 0.0
>> 0:00.73 kworker/u4:1
>>
>> That is running on 2-CPU machine with 4GB RAM.
>>
>> Jim
>>
>>
>>
>>
>> On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas
>> Gustavsson <to...@pr... <mailto:to...@pr...>>
> wrote:
>>
>>
>> Yes, like top.
>>
>>
>> On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi Tomas,
>>
>> I'm sorry that I missed the message below, but I am not
>> clear what you are asking me to check/monitor? Do you
>> mean like running something like "top" command?
>>
>> Thanks,
>> Jim
>>
>> On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas
>> Gustavsson <to...@pr...
> <mailto:to...@pr...>> wrote:
>>
>>
>> To verify that the changes actually happen...
>>
>> Do you monitor CPU usage to see what processes use CPU
>> and memory?
>>
>>
>> On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> The import has been running for about an hour now
>> and has imported 17836 entries.
>>
>> Per what I had said previously, this CRL file has
>> 1002255 entries, that means it would take about 56+
>> hours to import this CRL.
>>
>> So, this import processing is still way too slow.
>>
>> I don't know what is involved, or even how much it
>> would improve things, but I think that the next
>> thing to try is to eliminated both the check for the
>> serial number in the DB AND the dummy cert creation.
>>
>> Jim
>>
>>
>> On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya---
>> via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> Ok, I rebuilt the JAR using your Java class, and it
>> is running now, and creating rows in the
>> CertificateData, but it still SEEMS kind of slow.
>>
>> I am running now and will try to get a measurement
>> of revoked entries/time and will post back.
>>
>> Jim
>>
>>
>>
>>
>> On Tuesday, July 16, 2019, 5:25:32 PM UTC,
>> oh...@ya... <mailto:oh...@ya...>
> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> So I just tried what I said (emptying the "if") and
>> you are right, it doesn't create the rows in the
>> CeritificateData table :(...
>>
>> Ok, I will try your version...
>>
>> Jim
>>
>> On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas
>> Gustavsson <to...@pr...
> <mailto:to...@pr...>> wrote:
>>
>>
>>
>> Hi,
>>
>> I don't think so, no quite actually. In order to use
>> the API to populate
>> the database you need the certificate object it
>> seems. Something like
>> the attached file should work, it still creates a
>> dummy certificate, but
>> uses a static hard coded key, so no key generation.
>>
>> The file assumes you are on 6.15.x.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
>> > For testing (only...), I am thinking that if I
>> modify this If clause in
>> > CaImportCrlCommand.java:
>> >
>> > if (adaptive)
>> > {
>> > Date time = new Date();
>> > KeyPair key_pair =
>> KeyTools.genKeys("2048", "RSA");
>> >
>> > SubjectPublicKeyInfo pkinfo =
>> >
>>
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
>> > X500Name dnName = new
>> X500Name("CN=Dummy Missing in
>> > Imported CRL, serialNumber=" + serialHex);
>> > Date notAfter = new
>> Date(time.getTime() + 315360000000L);
>> > X509v3CertificateBuilder certbuilder
>> = new
>> > X509v3CertificateBuilder(X500Name.getInstance(cacert
>> >
>> .getSubjectX500Principal().getEncoded()), serialNr,
>> > time, notAfter, dnName, pkinfo);
>> > ContentSigner signer = new
>> BufferingContentSigner(new
>> >
>>
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
>> > .getPrivate()), 20480);
>> > X509CertificateHolder certHolder =
>> certbuilder.build(signer);
>> > X509Certificate certificate =
>> >
>>
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
>> > X509Certificate.class);
>> >
>> > String fingerprint =
>> > CertTools.getFingerprintAsString(certificate);
>> >
>> > EndEntityInformation
>> missingUserEndEntityInformation =
>> >
>>
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
>> > missing_user_name);
>> > if (missingUserEndEntityInformation
>> == null)
>> > {
>> > log.debug("Loading/updating user "
>> + missing_user_name);
>> >
>> > EndEntityInformation userdataNew
> = new
>> > EndEntityInformation(missing_user_name,
>> > CertTools.getSubjectDN(certificate),
>> cainfo.getCAId(), null, null, 10,
>> > new EndEntityType(EndEntityTypes.ENDUSER), 1, 1,
>> null, null, 1, 0, null);
>> >
>> > userdataNew.setPassword("foo123");
>> >
>> >
>>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
>> > userdataNew, false);
>> >
>> > log.info("User '" +
>> missing_user_name + "' has been
>> > added.");
>> >
>> >
>>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
>> > missing_user_name, 50);
>> >
>> > log.info("User '" +
>> missing_user_name + "' has been
>> > updated.");
>> > }
>> >
>> >
>>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>> >
>> > EJBTools.wrap(certificate),
>> missing_user_name,
>> > fingerprint, 20, 1, 1, 0, null, new Date()
>> >
>> > .getTime());
>> > log.info("Dummy certificate '" +
>> serialHex + "' has been
>> > stored.");
>> > }
>> >
>> > to (for example):
>> >
>> > if (adaptive)
>> > {
>> >
>> > log.info("*** SKIPPING *** Dummy
>> certificate '" +
>> > serialHex + "' has been stored.");
>> > }
>> >
>> >
>> > Would THAT work??
>> >
>> > Thanks,
>> > Jim
>> >
>> >
>> >
>> >
>> > On Tuesday, July 16, 2019, 10:59:31 AM UTC,
>> ohaya--- via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> >
>> > Hi Tomas (or anyone else),
>> >
>> > Sorry - I was typing my last msg on phone, and it
>> wasn't as detailed or
>> > clear as I wanted.
>> >
>> > Basically what I was saying is I was thinking
>> that, as a quick sanity
>> > test, if there was one or two places where the
> code in
>> > the CaImportCRLCommand.java was maybe making a
>> method call to create the
>> > dummy cert and put it into the database, then
>> maybe I could decompile
>> > and modify the CaImportCRLCommand.java, and
>> comment out those method
>> > calls, then, re-do my large CRL import test and
>> time it?
>> >
>> > If so, can you let me know specifically where
>> those calls are, and if I
>> > just comment them out, would that work?
>> >
>> > Ironically, that (not doing the dummy cert
>> creation and writing) was
>> > what I *thought* the "LENIENT" option was doing).
>> >
>> > Thanks,
>> > Jim
>> >
>> > P. S. I have time today to try this if I figure
>> out what needs to be done.
>> >
>> >
>> >
>> > On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya---
>> via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> >
>> >
>> > Yes, we are definitely interested/willing to test
>> patches. I actually
>> > was wondering if it would be a quick test to mod
>> the code somewhere to
>> > avoid a call that creates the dummy cert, to get
>> an idea if that would
>> > reduce the import time enough??
>> >
>> >
>> > Sent from Yahoo Mail on Android
>> >
>>
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>> >
>> > On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>>
>> wrote:
>> > Most large users, some very very large,
>> typically uses on line
>> > updates directly from the CA. Many CAs today
>> don't accept only
>> > importing CRLs, as they need to keep track of
>> issued certificates as
>> > well not only revoked (can't use "non existing
>> is good").
>> > Having said thst, it's interesting to be able
>> to import large CRLs,
>> > so improving it is of interest.
>> >
>> > Importing CRLs would actually not have to
>> generate a dummy cert at
>> > all, since a certificate object is not needed
>> on the responder,
>> > which should make it very fast.
>> >
>> > If you are willing to test patches, we can work
>> with you on improving.
>> >
>> > Regards,
>> > Tomas
>> >
>> >
>> > On July 15, 2019 5:26:08 PM GMT+02:00, ohaya---
>> via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> > Hi,
>> >
>> > I am curious whether you have other users
>> that import large CRLs
>> > like this?
>> >
>> > Are we the only ones who would encounter
>> these really long
>> > import times, because the other users who
>> might have large CRLs
>> > use the EJBCA CA also, so they don't run
>> into having the dummy
>> > certs being created for every entry in the
> CRL?
>> >
>> > Jim
>> >
>> > On Monday, July 15, 2019, 1:42:57 PM UTC,
>> Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>> wrote:
>> >
>> >
>> >
>> > I see...
>> >
>> > I can see that a new key pair is being
>> generated for each dummy
>> > cert.
>> >
>> > In CaImportCRLCommand.java:
>> >
>> > �� final KeyPair key_pair =
>> KeyTools.genKeys("2048",
>> > AlgorithmConstants.KEYALGORITHM_RSA);
>> >
>> > Chaning that to 1024 or 512 would be an
>> easy improvement. A bit more
>> > complex, but even larger speedup would be
>> to use a "hard coded"
>> > dummy
>> > key, i.e. same key for all dummy certs.
>> >
>> > You can experiment by changing 2048 to 512
>> (not secure but it
>> > doesn't
>> > have to be for this dummy cert).
>> >
>> > Regards,
>> > Tomas
>> >
>> > On 2019-07-11 14:19, ohaya--- via
>> Ejbca-develop wrote:
>> > > FYI, the is running, but it is REALLY
>> slow. The CRL file has
>> > 1002255
>> > > entries in it, and the import is only up
>> to 8899 now, after
>> > about 40
>> > > minutes... which would take about 75
> hours!!
>> > >
>> > > In our production systems, we are
>> required to update some CRLs
>> > every
>> > > hour, so that would be a problem.
>> > >
>> > > What can I do to speed that up??
>> > >
>> > > Thanks,
>> > > Jim
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 11:09:33 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
>> > >
>> > >
>> > > AHHH... Ok, it is running now, and from
>> checking the
>> > CertificateData in
>> > > the Mysql database, I can see that it is
>> now populating the
>> > > CertificateData!!
>> > >
>> > > Thanks!
>> > >
>> > > Jim
>> > >
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 11:07:22 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
>> > >
>> > >
>> > > I wasn't sure if we should use ADAPTIVE,
>> since we are not
>> > using the CA part.
>> > >
>> > >
>> > > I will try with "-o ADAPTIVE" now.
>> > >
>> > >
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 7:42:14 AM
>> UTC, Tomas Gustavsson
>> > > <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>> wrote:
>> > >
>> > >
>> > >
>> > > It should be ADAPTIVE.
>> > >
>> > > The documentation says:
>> > > bin/ejbca.sh ca importcrl --help
>> > > ---
>> > > ADAPTIVE means that missing certficates
>> will be replaced by dummy
>> > > certificates to cater for proper CRLs for
>> missing certificates.
>> > > ---
>> > >
>> > > Regards,
>> > > Tomas
>> > >
>> > > On 2019-07-11 08:05, ohaya--- via
>> Ejbca-develop wrote:
>> > >> Hi,
>> > >>
>> > >> I've been looking at the code in
>> CaImportCRLCommand, and I am
>> > wondering
>> > >> if, when the command is run in LENIENT
>> mode and it doesn't
>> > find the cert
>> > >> in the database (and outputs the
>> "missing in database" error
>> > message),
>> > >> what is supposed to happen?
>> > >>
>> > >> Mind you, the only error messages I am
>> seeing are the "missing in
>> > >> database" error messages. I don't see
>> any of the other error
>> > messages
>> > >> that are in/from CaImportCRLCommand.java.
>> > >>
>> > >> I am wondering if, when the command is
>> run with "-o LENIENT",
>> > and when
>> > >> it outputs the "missing in database"
>> error message, then the
>> > ejbca.sh is
>> > >> basically not doing anything (and not
>> storing anything into the
>> > >> CertificateData table)?
>> > >>
>> > >> From the lack of any other error
>> messages (vs. what I see in the
>> > >> CaImportCRLCommand.java code), that is
>> what it looks like is
>> > happening.
>> > >>
>> > >> So given I am using EJBCA for OCSP
>> responder only, is running the
>> > >> ejbca.sh in the "-o LENIENT" mode correct?
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >>
>> > >> On Thursday, July 11, 2019, 5:43:37 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> I've been doing a bunch of testing with
>> trying to import the
>> > large CRL,
>> > >> and wanted to summarize:
>> > >>
>> > >> - The ejbca.sh runs fine - it completes
>> without outputting an
>> > errors,
>> > >> except for outputting a bunch of the
>> "cert xxxx missing in
>> > database"
>> > >> messages.
>> > >>
>> > >> - When the import completes, and I check
>> the Mysql database,
>> > the crlData
>> > >> (the full PEM of the CRL) is in the
>> database, but there is no
>> > rows in
>> > >> the CertificateData table for that CA at
>> all.
>> > >>
>> > >> - I even figured out how to get the
>> ejbca.sh to log using
>> > >> log4j.properties, and it output a bunch
>> of logging, mainly
>> > the missing
>> > >> in database errors, but other than that
>> I didn't see any
>> > obvious errors.
>> > >>
>> > >>
>> > >> So, bottom line is that even though the
>> ejbca.sh seems to run
>> > without
>> > >> errors (other than the missing in
>> database errors), the import is
>> > >> causing the CRL PEM to be stored in the
>> database, but the
>> > >> CertificateData table is not being
>> populated at all for that CA.
>> > >>
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >>
>> > >> On Wednesday, July 10, 2019, 7:40:16 PM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> Ok, I found Tomas' original message...
>> it was in my Spam
>> > folder....
>> > >>
>> > >> I also tried re-running the "./ejbca.sh
>> ca importcrl" command
>> > a bunch of
>> > >> ways. I think it is running without
>> outputting any errors,
>> > but still,
>> > >> no rows are appearing with that issuerDN
>> at all.
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Tuesday, July 9, 2019, 10:45:01 AM
>> UTC, Tomas Gustavsson
>> > >> <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>> > <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> OCSP responses does not have anything to
>> do with the CRL
>> > itself. Once
>> > >> the CRL has been imported each
>> certificate status is in its
>> > own record
>> > >> in the CertificateData table. This is
>> where status is read
>> > when reponses
>> > >> are created.
>> > >> If there are issues, it is in that case
>> related to the import
>> > of the CRL.
>> > >>
>> > >> Verify that certificate records are
>> created in
>> > CertificateData when
>> > >> importing the CRL.
>> > >> Also, a debug log when making OCSP
>> request will tell exact
>> > details of
>> > >> what the responder finds in the database
>> or not.
>> > >>
>> > >> Regards,
>> > >> Tomas
>> > >>
>> > >>
>> > >> On July 8, 2019 4:59:06 PM GMT+02:00,
>> ohaya--- via Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >> [Re-sending, as this (handling of
>> large CRLs) is still a
>> > problem and
>> > >> is still not working.]
>> > >>
>> > >>
>> > >>
>> > >> On Monday, July 1, 2019, 12:47:23 PM
>> UTC, <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> I think that I've found how to
>> set/change the
>> > max_allowed_packet on
>> > >> the client (i.e., EJBCA) side, by
>> editing the Mysql URL in the
>> > >>
>> standalone/configuration/standalone.xml to:
>> > >>
>> > >>
>> > >
>> >
>>
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
>> > >>
>> > >> However, when I test the OCSP
>> Responder with a serial
>> > number (and CA
>> > >> cert) from the CA with the large
>> (>71MB) CRL file, I am still
>> > >> getting "unknown" and EJBCA OCSP
>> Responder is still NOT
>> > FINDING the
>> > >> cert serial number.
>> > >>
>> > >> NOTE, again, that I can test with
>> certs (and CA cert) that
>> > have
>> > >> smaller CRL files, and those tests work.
>> > >>
>> > >> So it is only when testing with the
>> large CRL file that is
>> > failing
>> > >> with the OCSP Responder.
>> > >>
>> > >> This seems like a potential BUG for
>> EJBCA OCSP Responder?
>> > >>
>> > >> Please advise.
>> > >>
>> > >> Thanks,
>> > >> Jim
>> > >> On Saturday, June 29, 2019, 3:36:18
>> PM EDT,
>> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> BTW, just to check, I have actually
>> gone through the
>> > process of
>> > >> getting the crlData from the MySQL
>> using mysql client (I
>> > had to add
>> > >> the --max_allowed-packet=100M to the
>> mysql command) and I
>> > was able
>> > >> to actually get the PEM from the
>> Mysql database that way.
>> > >>
>> > >> So, I KNOW that the entire CRL is in
>> the Mysql database,
>> > and yet
>> > >> when I run an "openssl ocsp" test
>> against the EJBCA OCSP
>> > Responder,
>> > >> it says it cannot find the revocation.
>> > >>
>> > >> OCSP Request Data:
>> > >> Version: 1 (0x0)
>> > >> Requestor List:
>> > >> Certificate ID:
>> > >> Hash Algorithm: sha1
>> > >> Issuer Name Hash:
>> 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
>> > >> Issuer Key Hash:
>> 3929A80864E28F83AA816147119133F85A37A6E4
>> > >> Serial Number: 2FFFF4
>> > >> Request Extensions:
>> > >> OCSP Nonce:
>> > >> 04103054A60A55E389B239A55228033D8F62
>> > >> Response verify OK
>> > >> 0x2FFFF4: unknown
>> > >> This Update: Jun 29 19:31:02 2019 GMT
>> > >>
>> > >>
>> > >> In the server log:
>> > >>
>> > >> 2019-06-29 15:31:02,150 DEBUG
>> > >>
>> >
>>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
>> > >> (default task-1) Unable to find
>> revocation information for
>> > >> certificate with serial '2ffff4' from
>> issuer 'CN=XXX....'
>> > >>
>> > >>
>> > >> Honestly, I am no 100% sure that the
>> problem is that
>> > >> max_allowe_packet is too small on the
>> mysql client side,
>> > but that is
>> > >> my best guess.
>> > >>
>> > >> What I AM sure is that the CRL is in
>> mysql, but when I do
>> > a test
>> > >> with OCSP Responder, it cannot find
>> the cert serial number
>> > in the CRL.
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >> On Saturday, June 29, 2019, 4:52:20
>> AM EDT,
>> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi Tomas,
>> > >>
>> > >> I already had the max_allowed_packet
>> configured on the Mysql
>> > >> server-side (in /etc/my.cnf).
>> > >>
>> > >> So as I said, when testing with mysql
>> (client) I had to
>> > include
>> > >> "--max_allowed_packet=100M" on the
>> mysql (client) command
>> > line, e.g.:
>> > >>
>> > >> mysql -u root -p
>> --max_allowed_packet=100M
>> > >>
>> > >> If I did NOT include the
>> "--max_allowed_packet=100M" on
>> > that command
>> > >> line, then when I did a "select" on
>> the field base64Crl, I
>> > would get
>> > >> an error.
>> > >>
>> > >> I suspect that EJBCA is acting as a
>> Mysql client, so, in
>> > order to
>> > >> work with such large CRLs, there
>> needs to be some way to
>> > set the
>> > >> equivalent of the
>> "--max_allowed_packet=100M" on the EJBCA
>> > side.
>> > >>
>> > >> How can I do that?
>> > >>
>> > >> If that can't be done, then I think
>> that when EJBCA tries to
>> > >> retrieve the CRL from the Mysql, it
>> (EJBCA) will see the
>> > same error.
>> > >>
>> > >> Thanks,
>> > >> Jim
>> > >>
>> > >> On Saturday, June 29, 2019, 3:25:47
>> AM EDT, Tomas Gustavsson
>> > >> <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>> > <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>>> wrote:
>> > >>
>> > >>
>> > >>
>> > >> You configure max allowed packet size
>> in you MySQL
>> > configuration on the
>> > >> MySQL server.
>> > >>
>> > >> /Tomas
>> > >>
>> > >> On 2019-06-29 02:19, o haya via
>> Ejbca-develop wrote:
>> > >> > I am not 100% sure, but I think I
>> know what might be
>> > causing this
>> > >> problem.
>> > >> >
>> > >> > I did a test, where I just tried to
>> log into mysql and
>> > do the
>> > >> following
>> > >> > select:
>> > >> >
>> > >> > select base64Crl INTO OUTFILE
>> > '/var/lib/mysql-files/xx.out' from
>> > >> CRLData
>> > >> > where cRLNumber=3167;
>> > >> >
>> > >> > and I got an error saying that the
>> result was more
>> > >> than max_allowed_packet.
>> > >> >
>> > >> > So then I did the same thing, but
>> for the mysql command
>> > I used:
>> > >> >
>> > >> > mysql -u root -p
>> --max_allowed_packet=100M
>> > >> >
>> > >> > and then when I did that same
>> select, I was able to
>> > output the entire
>> > >> > CRL as a PEM.
>> > >> >
>> > >> > So, I am theorizing that that is
>> what is happening with
>> > EJBCA, i.e.,
>> > >> > when I run the "openssl ocsp"
>> command, EJBCA is querying the
>> > > base64CRL
>> > >> > but the result is too large and so
>> that is causing the
>> > query to fail.
>> > >> >
>> > >> > So the question now is: How can I
>> cause EJBCA OCSP
>> > Responder to run
>> > >> > with a larger max_allowed_packet so
>> that the query
>> > succeeds (and then
>> > >> > hopefully EJBCA OCSP Responder will
>> be able to check the
>> > revocation
>> > >> > properly)?
>> > >> >
>> > >> > Does anyone know how that can be done?
>> > >> >
>> > >> > Thanks,
>> > >> > Jim
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Friday, June 28, 2019, 07:16:39
>> PM UTC, o haya
>> > > <oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>>
>> > >> <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <m...
[truncated message content] |
|
From: Tomas G. <to...@pr...> - 2019-07-19 08:00:50
|
Yes, MyISAM is horrible and can not be used.
I'm not sure about the exact commands to change it. But I think there
are good internet guides.
The reason you have MyISAM is probably because you are using WildFly 14
(are you?). There was a change in Hibernate that caused this issue with
automatic database creation.
One way to modify is to mysqldump the database, change enging in the sql
file and restore it.
Regards,
Tomas
On 2019-07-19 09:19, ohaya--- via Ejbca-develop wrote:
> Hi,
>
> Mysql version: Server version: 5.7.26 MySQL Community Server (GPL)
>
>
>
>
> mysql> show create table CertificateData;
>
>
>
> | CertificateData | CREATE TABLE `CertificateData` (
> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `certificateProfileId` int(11) NOT NULL,
> `endEntityProfileId` int(11) DEFAULT NULL,
> `expireDate` bigint(20) NOT NULL,
> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `notBefore` bigint(20) DEFAULT NULL,
> `revocationDate` bigint(20) NOT NULL,
> `revocationReason` int(11) NOT NULL,
> `rowVersion` int(11) NOT NULL,
> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `status` int(11) NOT NULL,
> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `type` int(11) NOT NULL,
> `updateTime` bigint(20) NOT NULL,
> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `base64Cert` longtext,
> `rowProtection` longtext,
> PRIMARY KEY (`fingerprint`),
> KEY `certificatedata_idx2` (`username`)
> ) ENGINE=MyISAM DEFAULT CHARSET=utf8 |
>
>
>
> AH! Yes mine is showing as MyISAM and not InnoDB!
>
>
>
> Do I need to change the engine to InnoDB? And, if so, how do I do that?
>
> Do I need to do that for every table in the ejbca database?
>
>
> Thanks,
> Jim
>
>
>
>
> On Friday, July 19, 2019, 6:20:23 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> That looks about right. What version of database are you using? I just
> tried the index command (on line 23 in the SQL file) myself and it works.
>
> CREATE INDEX certificatedata_idx4 ON CertificateData (subjectDN);
>
>
> What does your database creation say?
>
>>show create table CertiifcateData;
>
> It should look something like this:
>
> | CertificateData | CREATE TABLE `CertificateData` (
> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `certificateProfileId` int(11) NOT NULL,
> `crlPartitionIndex` int(11) DEFAULT NULL,
> `endEntityProfileId` int(11) DEFAULT NULL,
> `expireDate` bigint(20) NOT NULL,
> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `notBefore` bigint(20) DEFAULT NULL,
> `revocationDate` bigint(20) NOT NULL,
> `revocationReason` int(11) NOT NULL,
> `rowVersion` int(11) NOT NULL,
> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `status` int(11) NOT NULL,
> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `type` int(11) NOT NULL,
> `updateTime` bigint(20) NOT NULL,
> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `base64Cert` longtext,
> `certificateRequest` longtext,
> `rowProtection` longtext,
> PRIMARY KEY (`fingerprint`)
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
>
> Is the database engine wrong? I.e. not InnoDB?
>
> I am importing the CRL right now and processing about 27 records per
> second on my laptop.
>
> Regards,
> Tomas
>
>
> On 2019-07-18 18:09, ohaya--- via Ejbca-develop wrote:
>> According to my notes I used these commands:
>>
>> |*/CREATE DATABASE ejbca CHARACTER SET utf8 COLLATE utf8_general_ci;/*|
>>
>> |*/ /*|
>>
>> |*/GRANT ALL PRIVILEGES ON ejbca.* TO 'ejbca'@'localhost' IDENTIFIED BY
>> 'ejbca';/*|
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 3:37:51 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> What command did you use to create the database?
>>
>> On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> This is really getting frustrating!!!
>>
>> [root@ejbca <mailto:root@ejbca> sql-scripts]# mysql -u root
> -p'xxxxxx' ejbca <
>> create-index-ejbca.sql
>> mysql: [Warning] Using a password on the command line interface can
>> be insecure.
>> ERROR 1071 (42000) at line 23: Specified key was too long; max key
>> length is 1000 bytes
>>
>> Jim
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> The "machine" is a virtual machine under VirtualBox on a machine I
>> have that has 64GB physical RAM. Drives are all 7200 RPM drives.
>>
>> I will restore the snapshot, then increase the memory and # of
>> processors and try to run the create index again.
>>
>> Jim
>>
>> On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Hi,
>>
>> Just run create index script.
>>
>> Create index should run pretty fast even with millions of records in
>> the database. I suspect that your database is configured very slow,
>> perhaps running everything on disl, without RAM, using slow disks?
>>
>> A standard installation of mysql/mariadb on a centos machine should
>> run faster. Then you have pretty low amount of ram for running
>> everything on one box, handing millions of records.
>>
>> Regards,
>> Tomas
>>
>> On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> I had taken a snapshot of the machine before I tried to do the
>> index earlier, so I restored that snapshot.
>>
>> Then, I tried running the "optimize-ejbca-mysql.sql" and I got a
>> different/new error:
>>
>> ERROR 1091 (42000) at line 27: Can't DROP
>> 'certificatedata_idx5'; check that column/key exists
>>
>> Jim
>>
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> Oh oh, I just got this from the mysql execution:
>>
>> ERROR 1071 (42000) at line 23: Specified key was too long; max
>> key length is 1000 bytes
>>
>> Jim
>>
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> FYI, I am running the create-index-ejbca.sql script now, but
>> that seems to be taking FOREVER?
>>
>> I had stopped the earlier import, and then deleted all the rows
>> for that CA DN in CertificateData table, and then bounced the
>> machine, then I ran the create-index-ejbca.sql.
>>
>> Why is it taking so long? Should I have done something in
>> addition to deleting the rows? I am guessing that the reason
>> the sql script is taking so long is that it's somehow still
>> trying to index the rows were deleted or something like that?
>>
>> Thanks,
>> Jim
>>
>>
>>
>> On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> I notice that there is:
>>
>> - create-index-ejbca.sql
>> - optimize-ejbca-mysql.sql
>>
>> Should I run BOTH of them, or just the "create-index-ejbca.sql "?
>>
>> Thanks,
>> Jim
>>
>>
>> On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Yes, i could see from your output from top that mysql takes all
>> CPU. This indicates that you don't have any indexes in your
>> database, which makes it horribly slow as numbers grow.
>>
>> Apply database indexes from doc/sql-scripts.
>>
>> Regards,
>> Tomas
>>
>>
>> On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> So far, processed only 185706 entries, and it looks like the
>> processing is going to a crawl (really slow now).
>>
>> Which kind of makes me wonder if maybe part of reason for
>> the slowness is that it is having to search the certificate
>> database for the serial number, and now, as there are more
>> dummy certs in the database, that search is taking longer
>> and longer?
>>
>> Jim
>>
>>
>> On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> This is what top is showing now (the import is still running):
>>
>> top - 13:56:19 up 1 day, 1:09, 3 users, load average:
>> 1.18, 1.11, 1.07
>> Tasks: 167 total, 2 running, 165 sleeping, 0 stopped,
>> 0 zombie
>> %Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0
>> hi, 0.3 si, 0.0 st
>> KiB Mem : 4045880 total, 123192 free, 3053072 used,
>> 869616 buff/cache
>> KiB Swap: 3905532 total, 3753980 free, 151552 used.
>> 688136 avail Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU
>> %MEM TIME+ COMMAND
>> 1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2
>> 1227:41 mysqld
>> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2
>> 139:05.67 java
>> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3
>> 176:26.68 java
>> 22466 root 20 0 0 0 0 S 0.7 0.0
>> 0:00.73 kworker/u4:1
>>
>> That is running on 2-CPU machine with 4GB RAM.
>>
>> Jim
>>
>>
>>
>>
>> On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas
>> Gustavsson <to...@pr... <mailto:to...@pr...>>
> wrote:
>>
>>
>> Yes, like top.
>>
>>
>> On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi Tomas,
>>
>> I'm sorry that I missed the message below, but I am not
>> clear what you are asking me to check/monitor? Do you
>> mean like running something like "top" command?
>>
>> Thanks,
>> Jim
>>
>> On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas
>> Gustavsson <to...@pr...
> <mailto:to...@pr...>> wrote:
>>
>>
>> To verify that the changes actually happen...
>>
>> Do you monitor CPU usage to see what processes use CPU
>> and memory?
>>
>>
>> On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> The import has been running for about an hour now
>> and has imported 17836 entries.
>>
>> Per what I had said previously, this CRL file has
>> 1002255 entries, that means it would take about 56+
>> hours to import this CRL.
>>
>> So, this import processing is still way too slow.
>>
>> I don't know what is involved, or even how much it
>> would improve things, but I think that the next
>> thing to try is to eliminated both the check for the
>> serial number in the DB AND the dummy cert creation.
>>
>> Jim
>>
>>
>> On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya---
>> via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> Ok, I rebuilt the JAR using your Java class, and it
>> is running now, and creating rows in the
>> CertificateData, but it still SEEMS kind of slow.
>>
>> I am running now and will try to get a measurement
>> of revoked entries/time and will post back.
>>
>> Jim
>>
>>
>>
>>
>> On Tuesday, July 16, 2019, 5:25:32 PM UTC,
>> oh...@ya... <mailto:oh...@ya...>
> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> So I just tried what I said (emptying the "if") and
>> you are right, it doesn't create the rows in the
>> CeritificateData table :(...
>>
>> Ok, I will try your version...
>>
>> Jim
>>
>> On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas
>> Gustavsson <to...@pr...
> <mailto:to...@pr...>> wrote:
>>
>>
>>
>> Hi,
>>
>> I don't think so, no quite actually. In order to use
>> the API to populate
>> the database you need the certificate object it
>> seems. Something like
>> the attached file should work, it still creates a
>> dummy certificate, but
>> uses a static hard coded key, so no key generation.
>>
>> The file assumes you are on 6.15.x.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
>> > For testing (only...), I am thinking that if I
>> modify this If clause in
>> > CaImportCrlCommand.java:
>> >
>> > if (adaptive)
>> > {
>> > Date time = new Date();
>> > KeyPair key_pair =
>> KeyTools.genKeys("2048", "RSA");
>> >
>> > SubjectPublicKeyInfo pkinfo =
>> >
>>
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
>> > X500Name dnName = new
>> X500Name("CN=Dummy Missing in
>> > Imported CRL, serialNumber=" + serialHex);
>> > Date notAfter = new
>> Date(time.getTime() + 315360000000L);
>> > X509v3CertificateBuilder certbuilder
>> = new
>> > X509v3CertificateBuilder(X500Name.getInstance(cacert
>> >
>> .getSubjectX500Principal().getEncoded()), serialNr,
>> > time, notAfter, dnName, pkinfo);
>> > ContentSigner signer = new
>> BufferingContentSigner(new
>> >
>>
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
>> > .getPrivate()), 20480);
>> > X509CertificateHolder certHolder =
>> certbuilder.build(signer);
>> > X509Certificate certificate =
>> >
>>
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
>> > X509Certificate.class);
>> >
>> > String fingerprint =
>> > CertTools.getFingerprintAsString(certificate);
>> >
>> > EndEntityInformation
>> missingUserEndEntityInformation =
>> >
>>
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
>> > missing_user_name);
>> > if (missingUserEndEntityInformation
>> == null)
>> > {
>> > log.debug("Loading/updating user "
>> + missing_user_name);
>> >
>> > EndEntityInformation userdataNew
> = new
>> > EndEntityInformation(missing_user_name,
>> > CertTools.getSubjectDN(certificate),
>> cainfo.getCAId(), null, null, 10,
>> > new EndEntityType(EndEntityTypes.ENDUSER), 1, 1,
>> null, null, 1, 0, null);
>> >
>> > userdataNew.setPassword("foo123");
>> >
>> >
>>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
>> > userdataNew, false);
>> >
>> > log.info("User '" +
>> missing_user_name + "' has been
>> > added.");
>> >
>> >
>>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
>> > missing_user_name, 50);
>> >
>> > log.info("User '" +
>> missing_user_name + "' has been
>> > updated.");
>> > }
>> >
>> >
>>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>> >
>> > EJBTools.wrap(certificate),
>> missing_user_name,
>> > fingerprint, 20, 1, 1, 0, null, new Date()
>> >
>> > .getTime());
>> > log.info("Dummy certificate '" +
>> serialHex + "' has been
>> > stored.");
>> > }
>> >
>> > to (for example):
>> >
>> > if (adaptive)
>> > {
>> >
>> > log.info("*** SKIPPING *** Dummy
>> certificate '" +
>> > serialHex + "' has been stored.");
>> > }
>> >
>> >
>> > Would THAT work??
>> >
>> > Thanks,
>> > Jim
>> >
>> >
>> >
>> >
>> > On Tuesday, July 16, 2019, 10:59:31 AM UTC,
>> ohaya--- via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> >
>> > Hi Tomas (or anyone else),
>> >
>> > Sorry - I was typing my last msg on phone, and it
>> wasn't as detailed or
>> > clear as I wanted.
>> >
>> > Basically what I was saying is I was thinking
>> that, as a quick sanity
>> > test, if there was one or two places where the
> code in
>> > the CaImportCRLCommand.java was maybe making a
>> method call to create the
>> > dummy cert and put it into the database, then
>> maybe I could decompile
>> > and modify the CaImportCRLCommand.java, and
>> comment out those method
>> > calls, then, re-do my large CRL import test and
>> time it?
>> >
>> > If so, can you let me know specifically where
>> those calls are, and if I
>> > just comment them out, would that work?
>> >
>> > Ironically, that (not doing the dummy cert
>> creation and writing) was
>> > what I *thought* the "LENIENT" option was doing).
>> >
>> > Thanks,
>> > Jim
>> >
>> > P. S. I have time today to try this if I figure
>> out what needs to be done.
>> >
>> >
>> >
>> > On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya---
>> via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> >
>> >
>> > Yes, we are definitely interested/willing to test
>> patches. I actually
>> > was wondering if it would be a quick test to mod
>> the code somewhere to
>> > avoid a call that creates the dummy cert, to get
>> an idea if that would
>> > reduce the import time enough??
>> >
>> >
>> > Sent from Yahoo Mail on Android
>> >
>>
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>> >
>> > On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>>
>> wrote:
>> > Most large users, some very very large,
>> typically uses on line
>> > updates directly from the CA. Many CAs today
>> don't accept only
>> > importing CRLs, as they need to keep track of
>> issued certificates as
>> > well not only revoked (can't use "non existing
>> is good").
>> > Having said thst, it's interesting to be able
>> to import large CRLs,
>> > so improving it is of interest.
>> >
>> > Importing CRLs would actually not have to
>> generate a dummy cert at
>> > all, since a certificate object is not needed
>> on the responder,
>> > which should make it very fast.
>> >
>> > If you are willing to test patches, we can work
>> with you on improving.
>> >
>> > Regards,
>> > Tomas
>> >
>> >
>> > On July 15, 2019 5:26:08 PM GMT+02:00, ohaya---
>> via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> > Hi,
>> >
>> > I am curious whether you have other users
>> that import large CRLs
>> > like this?
>> >
>> > Are we the only ones who would encounter
>> these really long
>> > import times, because the other users who
>> might have large CRLs
>> > use the EJBCA CA also, so they don't run
>> into having the dummy
>> > certs being created for every entry in the
> CRL?
>> >
>> > Jim
>> >
>> > On Monday, July 15, 2019, 1:42:57 PM UTC,
>> Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>> wrote:
>> >
>> >
>> >
>> > I see...
>> >
>> > I can see that a new key pair is being
>> generated for each dummy
>> > cert.
>> >
>> > In CaImportCRLCommand.java:
>> >
>> > �� final KeyPair key_pair =
>> KeyTools.genKeys("2048",
>> > AlgorithmConstants.KEYALGORITHM_RSA);
>> >
>> > Chaning that to 1024 or 512 would be an
>> easy improvement. A bit more
>> > complex, but even larger speedup would be
>> to use a "hard coded"
>> > dummy
>> > key, i.e. same key for all dummy certs.
>> >
>> > You can experiment by changing 2048 to 512
>> (not secure but it
>> > doesn't
>> > have to be for this dummy cert).
>> >
>> > Regards,
>> > Tomas
>> >
>> > On 2019-07-11 14:19, ohaya--- via
>> Ejbca-develop wrote:
>> > > FYI, the is running, but it is REALLY
>> slow. The CRL file has
>> > 1002255
>> > > entries in it, and the import is only up
>> to 8899 now, after
>> > about 40
>> > > minutes... which would take about 75
> hours!!
>> > >
>> > > In our production systems, we are
>> required to update some CRLs
>> > every
>> > > hour, so that would be a problem.
>> > >
>> > > What can I do to speed that up??
>> > >
>> > > Thanks,
>> > > Jim
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 11:09:33 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
>> > >
>> > >
>> > > AHHH... Ok, it is running now, and from
>> checking the
>> > CertificateData in
>> > > the Mysql database, I can see that it is
>> now populating the
>> > > CertificateData!!
>> > >
>> > > Thanks!
>> > >
>> > > Jim
>> > >
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 11:07:22 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
>> > >
>> > >
>> > > I wasn't sure if we should use ADAPTIVE,
>> since we are not
>> > using the CA part.
>> > >
>> > >
>> > > I will try with "-o ADAPTIVE" now.
>> > >
>> > >
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 7:42:14 AM
>> UTC, Tomas Gustavsson
>> > > <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>> wrote:
>> > >
>> > >
>> > >
>> > > It should be ADAPTIVE.
>> > >
>> > > The documentation says:
>> > > bin/ejbca.sh ca importcrl --help
>> > > ---
>> > > ADAPTIVE means that missing certficates
>> will be replaced by dummy
>> > > certificates to cater for proper CRLs for
>> missing certificates.
>> > > ---
>> > >
>> > > Regards,
>> > > Tomas
>> > >
>> > > On 2019-07-11 08:05, ohaya--- via
>> Ejbca-develop wrote:
>> > >> Hi,
>> > >>
>> > >> I've been looking at the code in
>> CaImportCRLCommand, and I am
>> > wondering
>> > >> if, when the command is run in LENIENT
>> mode and it doesn't
>> > find the cert
>> > >> in the database (and outputs the
>> "missing in database" error
>> > message),
>> > >> what is supposed to happen?
>> > >>
>> > >> Mind you, the only error messages I am
>> seeing are the "missing in
>> > >> database" error messages. I don't see
>> any of the other error
>> > messages
>> > >> that are in/from CaImportCRLCommand.java.
>> > >>
>> > >> I am wondering if, when the command is
>> run with "-o LENIENT",
>> > and when
>> > >> it outputs the "missing in database"
>> error message, then the
>> > ejbca.sh is
>> > >> basically not doing anything (and not
>> storing anything into the
>> > >> CertificateData table)?
>> > >>
>> > >> From the lack of any other error
>> messages (vs. what I see in the
>> > >> CaImportCRLCommand.java code), that is
>> what it looks like is
>> > happening.
>> > >>
>> > >> So given I am using EJBCA for OCSP
>> responder only, is running the
>> > >> ejbca.sh in the "-o LENIENT" mode correct?
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >>
>> > >> On Thursday, July 11, 2019, 5:43:37 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> I've been doing a bunch of testing with
>> trying to import the
>> > large CRL,
>> > >> and wanted to summarize:
>> > >>
>> > >> - The ejbca.sh runs fine - it completes
>> without outputting an
>> > errors,
>> > >> except for outputting a bunch of the
>> "cert xxxx missing in
>> > database"
>> > >> messages.
>> > >>
>> > >> - When the import completes, and I check
>> the Mysql database,
>> > the crlData
>> > >> (the full PEM of the CRL) is in the
>> database, but there is no
>> > rows in
>> > >> the CertificateData table for that CA at
>> all.
>> > >>
>> > >> - I even figured out how to get the
>> ejbca.sh to log using
>> > >> log4j.properties, and it output a bunch
>> of logging, mainly
>> > the missing
>> > >> in database errors, but other than that
>> I didn't see any
>> > obvious errors.
>> > >>
>> > >>
>> > >> So, bottom line is that even though the
>> ejbca.sh seems to run
>> > without
>> > >> errors (other than the missing in
>> database errors), the import is
>> > >> causing the CRL PEM to be stored in the
>> database, but the
>> > >> CertificateData table is not being
>> populated at all for that CA.
>> > >>
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >>
>> > >> On Wednesday, July 10, 2019, 7:40:16 PM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> Ok, I found Tomas' original message...
>> it was in my Spam
>> > folder....
>> > >>
>> > >> I also tried re-running the "./ejbca.sh
>> ca importcrl" command
>> > a bunch of
>> > >> ways. I think it is running without
>> outputting any errors,
>> > but still,
>> > >> no rows are appearing with that issuerDN
>> at all.
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Tuesday, July 9, 2019, 10:45:01 AM
>> UTC, Tomas Gustavsson
>> > >> <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>> > <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> OCSP responses does not have anything to
>> do with the CRL
>> > itself. Once
>> > >> the CRL has been imported each
>> certificate status is in its
>> > own record
>> > >> in the CertificateData table. This is
>> where status is read
>> > when reponses
>> > >> are created.
>> > >> If there are issues, it is in that case
>> related to the import
>> > of the CRL.
>> > >>
>> > >> Verify that certificate records are
>> created in
>> > CertificateData when
>> > >> importing the CRL.
>> > >> Also, a debug log when making OCSP
>> request will tell exact
>> > details of
>> > >> what the responder finds in the database
>> or not.
>> > >>
>> > >> Regards,
>> > >> Tomas
>> > >>
>> > >>
>> > >> On July 8, 2019 4:59:06 PM GMT+02:00,
>> ohaya--- via Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >> [Re-sending, as this (handling of
>> large CRLs) is still a
>> > problem and
>> > >> is still not working.]
>> > >>
>> > >>
>> > >>
>> > >> On Monday, July 1, 2019, 12:47:23 PM
>> UTC, <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> I think that I've found how to
>> set/change the
>> > max_allowed_packet on
>> > >> the client (i.e., EJBCA) side, by
>> editing the Mysql URL in the
>> > >>
>> standalone/configuration/standalone.xml to:
>> > >>
>> > >>
>> > >
>> >
>>
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
>> > >>
>> > >> However, when I test the OCSP
>> Responder with a serial
>> > number (and CA
>> > >> cert) from the CA with the large
>> (>71MB) CRL file, I am still
>> > >> getting "unknown" and EJBCA OCSP
>> Responder is still NOT
>> > FINDING the
>> > >> cert serial number.
>> > >>
>> > >> NOTE, again, that I can test with
>> certs (and CA cert) that
>> > have
>> > >> smaller CRL files, and those tests work.
>> > >>
>> > >> So it is only when testing with the
>> large CRL file that is
>> > failing
>> > >> with the OCSP Responder.
>> > >>
>> > >> This seems like a potential BUG for
>> EJBCA OCSP Responder?
>> > >>
>> > >> Please advise.
>> > >>
>> > >> Thanks,
>> > >> Jim
>> > >> On Saturday, June 29, 2019, 3:36:18
>> PM EDT,
>> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> BTW, just to check, I have actually
>> gone through the
>> > process of
>> > >> getting the crlData from the MySQL
>> using mysql client (I
>> > had to add
>> > >> the --max_allowed-packet=100M to the
>> mysql command) and I
>> > was able
>> > >> to actually get the PEM from the
>> Mysql database that way.
>> > >>
>> > >> So, I KNOW that the entire CRL is in
>> the Mysql database,
>> > and yet
>> > >> when I run an "openssl ocsp" test
>> against the EJBCA OCSP
>> > Responder,
>> > >> it says it cannot find the revocation.
>> > >>
>> > >> OCSP Request Data:
>> > >> Version: 1 (0x0)
>> > >> Requestor List:
>> > >> Certificate ID:
>> > >> Hash Algorithm: sha1
>> > >> Issuer Name Hash:
>> 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
>> > >> Issuer Key Hash:
>> 3929A80864E28F83AA816147119133F85A37A6E4
>> > >> Serial Number: 2FFFF4
>> > >> Request Extensions:
>> > >> OCSP Nonce:
>> > >> 04103054A60A55E389B239A55228033D8F62
>> > >> Response verify OK
>> > >> 0x2FFFF4: unknown
>> > >> This Update: Jun 29 19:31:02 2019 GMT
>> > >>
>> > >>
>> > >> In the server log:
>> > >>
>> > >> 2019-06-29 15:31:02,150 DEBUG
>> > >>
>> >
>>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
>> > >> (default task-1) Unable to find
>> revocation information for
>> > >> certificate with serial '2ffff4' from
>> issuer 'CN=XXX....'
>> > >>
>> > >>
>> > >> Honestly, I am no 100% sure that the
>> problem is that
>> > >> max_allowe_packet is too small on the
>> mysql client side,
>> > but that is
>> > >> my best guess.
>> > >>
>> > >> What I AM sure is that the CRL is in
>> mysql, but when I do
>> > a test
>> > >> with OCSP Responder, it cannot find
>> the cert serial number
>> > in the CRL.
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >> On Saturday, June 29, 2019, 4:52:20
>> AM EDT,
>> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi Tomas,
>> > >>
>> > >> I already had the max_allowed_packet
>> configured on the Mysql
>> > >> server-side (in /etc/my.cnf).
>> > >>
>> > >> So as I said, when testing with mysql
>> (client) I had to
>> > include
>> > >> "--max_allowed_packet=100M" on the
>> mysql (client) command
>> > line, e.g.:
>> > >>
>> > >> mysql -u root -p
>> --max_allowed_packet=100M
>> > >>
>> > >> If I did NOT include the
>> "--max_allowed_packet=100M" on
>> > that command
>> > >> line, then when I did a "select" on
>> the field base64Crl, I
>> > would get
>> > >> an error.
>> > >>
>> > >> I suspect that EJBCA is acting as a
>> Mysql client, so, in
>> > order to
>> > >> work with such large CRLs, there
>> needs to be some way to
>> > set the
>> > >> equivalent of the
>> "--max_allowed_packet=100M" on the EJBCA
>> > side.
>> > >>
>> > >> How can I do that?
>> > >>
>> > >> If that can't be done, then I think
>> that when EJBCA tries to
>> > >> retrieve the CRL from the Mysql, it
>> (EJBCA) will see the
>> > same error.
>> > >>
>> > >> Thanks,
>> > >> Jim
>> > >>
>> > >> On Saturday, June 29, 2019, 3:25:47
>> AM EDT, Tomas Gustavsson
>> > >> <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>> > <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>>> wrote:
>> > >>
>> > >>
>> > >>
>> > >> You configure max allowed packet size
>> in you MySQL
>> > configuration on the
>> > >> MySQL server.
>> > >>
>> > >> /Tomas
>> > >>
>> > >> On 2019-06-29 02:19, o haya via
>> Ejbca-develop wrote:
>> > >> > I am not 100% sure, but I think I
>> know what might be
>> > causing this
>> > >> problem.
>> > >> >
>> > >> > I did a test, where I just tried to
>> log into mysql and
>> > do the
>> > >> following
>> > >> > select:
>> > >> >
>> > >> > select base64Crl INTO OUTFILE
>> > '/var/lib/mysql-files/xx.out' from
>> > >> CRLData
>> > >> > where cRLNumber=3167;
>> > >> >
>> > >> > and I got an error saying that the
>> result was more
>> > >> than max_allowed_packet.
>> > >> >
>> > >> > So then I did the same thing, but
>> for the mysql command
>> > I used:
>> > >> >
>> > >> > mysql -u root -p
>> --max_allowed_packet=100M
>> > >> >
>> > >> > and then when I did that same
>> select, I was able to
>> > output the entire
>> > >> > CRL as a PEM.
>> > >> >
>> > >> > So, I am theorizing that that is
>> what is happening with
>> > EJBCA, i.e.,
>> > >> > when I run the "openssl ocsp"
>> command, EJBCA is querying the
>> > > base64CRL
>> > >> > but the result is too large and so
>> that is causing the
>> > query to fail.
>> > >> >
>> > >> > So the question now is: How can I
>> cause EJBCA OCSP
>> > Responder to run
>> > >> > with a larger max_allowed_packet so
>> that the query
>> > succeeds (and then
>> > >> > hopefully EJBCA OCSP Responder will
>> be able to check the
>> > revocation
>> > >> > properly)?
>> > >> >
>> > >> > Does anyone know how that can be done?
>> > >> >
>> > >> > Thanks,
>> > >> > Jim
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Friday, June 28, 2019, 07:16:39
>> PM UTC, o haya
>> > > <oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>>
>> > >> <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>>> wrote:
>> > >> >
>> > >> >
>> > >> > Hi,
>> > >> >
>> > >> > As discussed in an earlier thread,
>> it looks like I was
>> > able to
>> > >> import a
>> > >> > large (> 71MB) CRL file into EJBCA
>> (as an OCSP
>> > Responder), so now,
>> > >> ...
[truncated message content] |
|
From: <oh...@ya...> - 2019-07-19 07:20:19
|
Hi,
Mysql version: Server version: 5.7.26 MySQL Community Server (GPL)
mysql> show create table CertificateData;
| CertificateData | CREATE TABLE `CertificateData` (
`fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`certificateProfileId` int(11) NOT NULL,
`endEntityProfileId` int(11) DEFAULT NULL,
`expireDate` bigint(20) NOT NULL,
`issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`notBefore` bigint(20) DEFAULT NULL,
`revocationDate` bigint(20) NOT NULL,
`revocationReason` int(11) NOT NULL,
`rowVersion` int(11) NOT NULL,
`serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`status` int(11) NOT NULL,
`subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`type` int(11) NOT NULL,
`updateTime` bigint(20) NOT NULL,
`username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`base64Cert` longtext,
`rowProtection` longtext,
PRIMARY KEY (`fingerprint`),
KEY `certificatedata_idx2` (`username`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 |
AH! Yes mine is showing as MyISAM and not InnoDB!
Do I need to change the engine to InnoDB? And, if so, how do I do that?
Do I need to do that for every table in the ejbca database?
Thanks,Jim
On Friday, July 19, 2019, 6:20:23 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
That looks about right. What version of database are you using? I just
tried the index command (on line 23 in the SQL file) myself and it works.
CREATE INDEX certificatedata_idx4 ON CertificateData (subjectDN);
What does your database creation say?
>show create table CertiifcateData;
It should look something like this:
| CertificateData | CREATE TABLE `CertificateData` (
`fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
DEFAULT NULL,
`certificateProfileId` int(11) NOT NULL,
`crlPartitionIndex` int(11) DEFAULT NULL,
`endEntityProfileId` int(11) DEFAULT NULL,
`expireDate` bigint(20) NOT NULL,
`issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`notBefore` bigint(20) DEFAULT NULL,
`revocationDate` bigint(20) NOT NULL,
`revocationReason` int(11) NOT NULL,
`rowVersion` int(11) NOT NULL,
`serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`status` int(11) NOT NULL,
`subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
DEFAULT NULL,
`subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
DEFAULT NULL,
`tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`type` int(11) NOT NULL,
`updateTime` bigint(20) NOT NULL,
`username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`base64Cert` longtext,
`certificateRequest` longtext,
`rowProtection` longtext,
PRIMARY KEY (`fingerprint`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
Is the database engine wrong? I.e. not InnoDB?
I am importing the CRL right now and processing about 27 records per
second on my laptop.
Regards,
Tomas
On 2019-07-18 18:09, ohaya--- via Ejbca-develop wrote:
> According to my notes I used these commands:
>
> |*/CREATE DATABASE ejbca CHARACTER SET utf8 COLLATE utf8_general_ci;/*|
>
> |*/ /*|
>
> |*/GRANT ALL PRIVILEGES ON ejbca.* TO 'ejbca'@'localhost' IDENTIFIED BY
> 'ejbca';/*|
>
>
>
>
>
> On Thursday, July 18, 2019, 3:37:51 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> What command did you use to create the database?
>
> On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> This is really getting frustrating!!!
>
> [root@ejbca sql-scripts]# mysql -u root -p'xxxxxx' ejbca <
> create-index-ejbca.sql
> mysql: [Warning] Using a password on the command line interface can
> be insecure.
> ERROR 1071 (42000) at line 23: Specified key was too long; max key
> length is 1000 bytes
>
> Jim
>
>
>
>
>
> On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya...
> <oh...@ya...> wrote:
>
>
> Hi,
>
> The "machine" is a virtual machine under VirtualBox on a machine I
> have that has 64GB physical RAM. Drives are all 7200 RPM drives.
>
> I will restore the snapshot, then increase the memory and # of
> processors and try to run the create index again.
>
> Jim
>
> On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Hi,
>
> Just run create index script.
>
> Create index should run pretty fast even with millions of records in
> the database. I suspect that your database is configured very slow,
> perhaps running everything on disl, without RAM, using slow disks?
>
> A standard installation of mysql/mariadb on a centos machine should
> run faster. Then you have pretty low amount of ram for running
> everything on one box, handing millions of records.
>
> Regards,
> Tomas
>
> On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> I had taken a snapshot of the machine before I tried to do the
> index earlier, so I restored that snapshot.
>
> Then, I tried running the "optimize-ejbca-mysql.sql" and I got a
> different/new error:
>
> ERROR 1091 (42000) at line 27: Can't DROP
> 'certificatedata_idx5'; check that column/key exists
>
> Jim
>
>
>
>
>
>
> On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
> <oh...@ya...> wrote:
>
>
> Hi,
>
> Oh oh, I just got this from the mysql execution:
>
> ERROR 1071 (42000) at line 23: Specified key was too long; max
> key length is 1000 bytes
>
> Jim
>
>
>
>
>
>
> On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
> <oh...@ya...> wrote:
>
>
> Hi,
>
> FYI, I am running the create-index-ejbca.sql script now, but
> that seems to be taking FOREVER?
>
> I had stopped the earlier import, and then deleted all the rows
> for that CA DN in CertificateData table, and then bounced the
> machine, then I ran the create-index-ejbca.sql.
>
> Why is it taking so long? Should I have done something in
> addition to deleting the rows? I am guessing that the reason
> the sql script is taking so long is that it's somehow still
> trying to index the rows were deleted or something like that?
>
> Thanks,
> Jim
>
>
>
> On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
> <oh...@ya...> wrote:
>
>
> Hi,
>
> I notice that there is:
>
> - create-index-ejbca.sql
> - optimize-ejbca-mysql.sql
>
> Should I run BOTH of them, or just the "create-index-ejbca.sql "?
>
> Thanks,
> Jim
>
>
> On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Yes, i could see from your output from top that mysql takes all
> CPU. This indicates that you don't have any indexes in your
> database, which makes it horribly slow as numbers grow.
>
> Apply database indexes from doc/sql-scripts.
>
> Regards,
> Tomas
>
>
> On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via
> Ejbca-develop <ejb...@li...> wrote:
>
> Hi,
>
> So far, processed only 185706 entries, and it looks like the
> processing is going to a crawl (really slow now).
>
> Which kind of makes me wonder if maybe part of reason for
> the slowness is that it is having to search the certificate
> database for the serial number, and now, as there are more
> dummy certs in the database, that search is taking longer
> and longer?
>
> Jim
>
>
> On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via
> Ejbca-develop <ejb...@li...> wrote:
>
>
> Hi,
>
> This is what top is showing now (the import is still running):
>
> top - 13:56:19 up 1 day, 1:09, 3 users, load average:
> 1.18, 1.11, 1.07
> Tasks: 167 total, 2 running, 165 sleeping, 0 stopped,
> 0 zombie
> %Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0
> hi, 0.3 si, 0.0 st
> KiB Mem : 4045880 total, 123192 free, 3053072 used,
> 869616 buff/cache
> KiB Swap: 3905532 total, 3753980 free, 151552 used.
> 688136 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU
> %MEM TIME+ COMMAND
> 1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2
> 1227:41 mysqld
> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2
> 139:05.67 java
> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3
> 176:26.68 java
> 22466 root 20 0 0 0 0 S 0.7 0.0
> 0:00.73 kworker/u4:1
>
> That is running on 2-CPU machine with 4GB RAM.
>
> Jim
>
>
>
>
> On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas
> Gustavsson <to...@pr...> wrote:
>
>
> Yes, like top.
>
>
> On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via
> Ejbca-develop <ejb...@li...> wrote:
>
> Hi Tomas,
>
> I'm sorry that I missed the message below, but I am not
> clear what you are asking me to check/monitor? Do you
> mean like running something like "top" command?
>
> Thanks,
> Jim
>
> On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas
> Gustavsson <to...@pr...> wrote:
>
>
> To verify that the changes actually happen...
>
> Do you monitor CPU usage to see what processes use CPU
> and memory?
>
>
> On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via
> Ejbca-develop <ejb...@li...> wrote:
>
> Hi,
>
> The import has been running for about an hour now
> and has imported 17836 entries.
>
> Per what I had said previously, this CRL file has
> 1002255 entries, that means it would take about 56+
> hours to import this CRL.
>
> So, this import processing is still way too slow.
>
> I don't know what is involved, or even how much it
> would improve things, but I think that the next
> thing to try is to eliminated both the check for the
> serial number in the DB AND the dummy cert creation.
>
> Jim
>
>
> On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya---
> via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> Ok, I rebuilt the JAR using your Java class, and it
> is running now, and creating rows in the
> CertificateData, but it still SEEMS kind of slow.
>
> I am running now and will try to get a measurement
> of revoked entries/time and will post back.
>
> Jim
>
>
>
>
> On Tuesday, July 16, 2019, 5:25:32 PM UTC,
> oh...@ya... <oh...@ya...> wrote:
>
>
> Hi,
>
> So I just tried what I said (emptying the "if") and
> you are right, it doesn't create the rows in the
> CeritificateData table :(...
>
> Ok, I will try your version...
>
> Jim
>
> On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas
> Gustavsson <to...@pr...> wrote:
>
>
>
> Hi,
>
> I don't think so, no quite actually. In order to use
> the API to populate
> the database you need the certificate object it
> seems. Something like
> the attached file should work, it still creates a
> dummy certificate, but
> uses a static hard coded key, so no key generation.
>
> The file assumes you are on 6.15.x.
>
> Regards,
> Tomas
>
> On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
> > For testing (only...), I am thinking that if I
> modify this If clause in
> > CaImportCrlCommand.java:
> >
> > if (adaptive)
> > {
> > Date time = new Date();
> > KeyPair key_pair =
> KeyTools.genKeys("2048", "RSA");
> >
> > SubjectPublicKeyInfo pkinfo =
> >
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
> > X500Name dnName = new
> X500Name("CN=Dummy Missing in
> > Imported CRL, serialNumber=" + serialHex);
> > Date notAfter = new
> Date(time.getTime() + 315360000000L);
> > X509v3CertificateBuilder certbuilder
> = new
> > X509v3CertificateBuilder(X500Name.getInstance(cacert
> >
> .getSubjectX500Principal().getEncoded()), serialNr,
> > time, notAfter, dnName, pkinfo);
> > ContentSigner signer = new
> BufferingContentSigner(new
> >
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
> > .getPrivate()), 20480);
> > X509CertificateHolder certHolder =
> certbuilder.build(signer);
> > X509Certificate certificate =
> >
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
> > X509Certificate.class);
> >
> > String fingerprint =
> > CertTools.getFingerprintAsString(certificate);
> >
> > EndEntityInformation
> missingUserEndEntityInformation =
> >
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
> > missing_user_name);
> > if (missingUserEndEntityInformation
> == null)
> > {
> > log.debug("Loading/updating user "
> + missing_user_name);
> >
> > EndEntityInformation userdataNew = new
> > EndEntityInformation(missing_user_name,
> > CertTools.getSubjectDN(certificate),
> cainfo.getCAId(), null, null, 10,
> > new EndEntityType(EndEntityTypes.ENDUSER), 1, 1,
> null, null, 1, 0, null);
> >
> > userdataNew.setPassword("foo123");
> >
> >
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
> > userdataNew, false);
> >
> > log.info("User '" +
> missing_user_name + "' has been
> > added.");
> >
> >
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
> > missing_user_name, 50);
> >
> > log.info("User '" +
> missing_user_name + "' has been
> > updated.");
> > }
> >
> >
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
> >
> > EJBTools.wrap(certificate),
> missing_user_name,
> > fingerprint, 20, 1, 1, 0, null, new Date()
> >
> > .getTime());
> > log.info("Dummy certificate '" +
> serialHex + "' has been
> > stored.");
> > }
> >
> > to (for example):
> >
> > if (adaptive)
> > {
> >
> > log.info("*** SKIPPING *** Dummy
> certificate '" +
> > serialHex + "' has been stored.");
> > }
> >
> >
> > Would THAT work??
> >
> > Thanks,
> > Jim
> >
> >
> >
> >
> > On Tuesday, July 16, 2019, 10:59:31 AM UTC,
> ohaya--- via Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > Hi Tomas (or anyone else),
> >
> > Sorry - I was typing my last msg on phone, and it
> wasn't as detailed or
> > clear as I wanted.
> >
> > Basically what I was saying is I was thinking
> that, as a quick sanity
> > test, if there was one or two places where the code in
> > the CaImportCRLCommand.java was maybe making a
> method call to create the
> > dummy cert and put it into the database, then
> maybe I could decompile
> > and modify the CaImportCRLCommand.java, and
> comment out those method
> > calls, then, re-do my large CRL import test and
> time it?
> >
> > If so, can you let me know specifically where
> those calls are, and if I
> > just comment them out, would that work?
> >
> > Ironically, that (not doing the dummy cert
> creation and writing) was
> > what I *thought* the "LENIENT" option was doing).
> >
> > Thanks,
> > Jim
> >
> > P. S. I have time today to try this if I figure
> out what needs to be done.
> >
> >
> >
> > On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya---
> via Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> >
> > Yes, we are definitely interested/willing to test
> patches. I actually
> > was wondering if it would be a quick test to mod
> the code somewhere to
> > avoid a call that creates the dummy cert, to get
> an idea if that would
> > reduce the import time enough??
> >
> >
> > Sent from Yahoo Mail on Android
> >
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
> >
> > On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
> > <to...@pr... <mailto:to...@pr...>>
> wrote:
> > Most large users, some very very large,
> typically uses on line
> > updates directly from the CA. Many CAs today
> don't accept only
> > importing CRLs, as they need to keep track of
> issued certificates as
> > well not only revoked (can't use "non existing
> is good").
> > Having said thst, it's interesting to be able
> to import large CRLs,
> > so improving it is of interest.
> >
> > Importing CRLs would actually not have to
> generate a dummy cert at
> > all, since a certificate object is not needed
> on the responder,
> > which should make it very fast.
> >
> > If you are willing to test patches, we can work
> with you on improving.
> >
> > Regards,
> > Tomas
> >
> >
> > On July 15, 2019 5:26:08 PM GMT+02:00, ohaya---
> via Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> > Hi,
> >
> > I am curious whether you have other users
> that import large CRLs
> > like this?
> >
> > Are we the only ones who would encounter
> these really long
> > import times, because the other users who
> might have large CRLs
> > use the EJBCA CA also, so they don't run
> into having the dummy
> > certs being created for every entry in the CRL?
> >
> > Jim
> >
> > On Monday, July 15, 2019, 1:42:57 PM UTC,
> Tomas Gustavsson
> > <to...@pr...
> <mailto:to...@pr...>> wrote:
> >
> >
> >
> > I see...
> >
> > I can see that a new key pair is being
> generated for each dummy
> > cert.
> >
> > In CaImportCRLCommand.java:
> >
> > �� final KeyPair key_pair =
> KeyTools.genKeys("2048",
> > AlgorithmConstants.KEYALGORITHM_RSA);
> >
> > Chaning that to 1024 or 512 would be an
> easy improvement. A bit more
> > complex, but even larger speedup would be
> to use a "hard coded"
> > dummy
> > key, i.e. same key for all dummy certs.
> >
> > You can experiment by changing 2048 to 512
> (not secure but it
> > doesn't
> > have to be for this dummy cert).
> >
> > Regards,
> > Tomas
> >
> > On 2019-07-11 14:19, ohaya--- via
> Ejbca-develop wrote:
> > > FYI, the is running, but it is REALLY
> slow. The CRL file has
> > 1002255
> > > entries in it, and the import is only up
> to 8899 now, after
> > about 40
> > > minutes... which would take about 75 hours!!
> > >
> > > In our production systems, we are
> required to update some CRLs
> > every
> > > hour, so that would be a problem.
> > >
> > > What can I do to speed that up??
> > >
> > > Thanks,
> > > Jim
> > >
> > >
> > > On Thursday, July 11, 2019, 11:09:33 AM
> UTC, ohaya--- via
> > Ejbca-develop
> > > <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> > >
> > >
> > > AHHH... Ok, it is running now, and from
> checking the
> > CertificateData in
> > > the Mysql database, I can see that it is
> now populating the
> > > CertificateData!!
> > >
> > > Thanks!
> > >
> > > Jim
> > >
> > >
> > >
> > > On Thursday, July 11, 2019, 11:07:22 AM
> UTC, ohaya--- via
> > Ejbca-develop
> > > <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> > >
> > >
> > > I wasn't sure if we should use ADAPTIVE,
> since we are not
> > using the CA part.
> > >
> > >
> > > I will try with "-o ADAPTIVE" now.
> > >
> > >
> > >
> > >
> > > On Thursday, July 11, 2019, 7:42:14 AM
> UTC, Tomas Gustavsson
> > > <to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>> wrote:
> > >
> > >
> > >
> > > It should be ADAPTIVE.
> > >
> > > The documentation says:
> > > bin/ejbca.sh ca importcrl --help
> > > ---
> > > ADAPTIVE means that missing certficates
> will be replaced by dummy
> > > certificates to cater for proper CRLs for
> missing certificates.
> > > ---
> > >
> > > Regards,
> > > Tomas
> > >
> > > On 2019-07-11 08:05, ohaya--- via
> Ejbca-develop wrote:
> > >> Hi,
> > >>
> > >> I've been looking at the code in
> CaImportCRLCommand, and I am
> > wondering
> > >> if, when the command is run in LENIENT
> mode and it doesn't
> > find the cert
> > >> in the database (and outputs the
> "missing in database" error
> > message),
> > >> what is supposed to happen?
> > >>
> > >> Mind you, the only error messages I am
> seeing are the "missing in
> > >> database" error messages. I don't see
> any of the other error
> > messages
> > >> that are in/from CaImportCRLCommand.java.
> > >>
> > >> I am wondering if, when the command is
> run with "-o LENIENT",
> > and when
> > >> it outputs the "missing in database"
> error message, then the
> > ejbca.sh is
> > >> basically not doing anything (and not
> storing anything into the
> > >> CertificateData table)?
> > >>
> > >> From the lack of any other error
> messages (vs. what I see in the
> > >> CaImportCRLCommand.java code), that is
> what it looks like is
> > happening.
> > >>
> > >> So given I am using EJBCA for OCSP
> responder only, is running the
> > >> ejbca.sh in the "-o LENIENT" mode correct?
> > >>
> > >> Jim
> > >>
> > >>
> > >>
> > >> On Thursday, July 11, 2019, 5:43:37 AM
> UTC, ohaya--- via
> > Ejbca-develop
> > >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>
> > >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> I've been doing a bunch of testing with
> trying to import the
> > large CRL,
> > >> and wanted to summarize:
> > >>
> > >> - The ejbca.sh runs fine - it completes
> without outputting an
> > errors,
> > >> except for outputting a bunch of the
> "cert xxxx missing in
> > database"
> > >> messages.
> > >>
> > >> - When the import completes, and I check
> the Mysql database,
> > the crlData
> > >> (the full PEM of the CRL) is in the
> database, but there is no
> > rows in
> > >> the CertificateData table for that CA at
> all.
> > >>
> > >> - I even figured out how to get the
> ejbca.sh to log using
> > >> log4j.properties, and it output a bunch
> of logging, mainly
> > the missing
> > >> in database errors, but other than that
> I didn't see any
> > obvious errors.
> > >>
> > >>
> > >> So, bottom line is that even though the
> ejbca.sh seems to run
> > without
> > >> errors (other than the missing in
> database errors), the import is
> > >> causing the CRL PEM to be stored in the
> database, but the
> > >> CertificateData table is not being
> populated at all for that CA.
> > >>
> > >>
> > >> Jim
> > >>
> > >>
> > >>
> > >> On Wednesday, July 10, 2019, 7:40:16 PM
> UTC, ohaya--- via
> > Ejbca-develop
> > >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>
> > >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> Ok, I found Tomas' original message...
> it was in my Spam
> > folder....
> > >>
> > >> I also tried re-running the "./ejbca.sh
> ca importcrl" command
> > a bunch of
> > >> ways. I think it is running without
> outputting any errors,
> > but still,
> > >> no rows are appearing with that issuerDN
> at all.
> > >>
> > >>
> > >>
> > >>
> > >> On Tuesday, July 9, 2019, 10:45:01 AM
> UTC, Tomas Gustavsson
> > >> <to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>
> > <mailto:to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> OCSP responses does not have anything to
> do with the CRL
> > itself. Once
> > >> the CRL has been imported each
> certificate status is in its
> > own record
> > >> in the CertificateData table. This is
> where status is read
> > when reponses
> > >> are created.
> > >> If there are issues, it is in that case
> related to the import
> > of the CRL.
> > >>
> > >> Verify that certificate records are
> created in
> > CertificateData when
> > >> importing the CRL.
> > >> Also, a debug log when making OCSP
> request will tell exact
> > details of
> > >> what the responder finds in the database
> or not.
> > >>
> > >> Regards,
> > >> Tomas
> > >>
> > >>
> > >> On July 8, 2019 4:59:06 PM GMT+02:00,
> ohaya--- via Ejbca-develop
> > >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>
> > >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
> > >>
> > >> [Re-sending, as this (handling of
> large CRLs) is still a
> > problem and
> > >> is still not working.]
> > >>
> > >>
> > >>
> > >> On Monday, July 1, 2019, 12:47:23 PM
> UTC, <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya...
> <mailto:oh...@ya...>>
> > > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> I think that I've found how to
> set/change the
> > max_allowed_packet on
> > >> the client (i.e., EJBCA) side, by
> editing the Mysql URL in the
> > >>
> standalone/configuration/standalone.xml to:
> > >>
> > >>
> > >
> >
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
> > >>
> > >> However, when I test the OCSP
> Responder with a serial
> > number (and CA
> > >> cert) from the CA with the large
> (>71MB) CRL file, I am still
> > >> getting "unknown" and EJBCA OCSP
> Responder is still NOT
> > FINDING the
> > >> cert serial number.
> > >>
> > >> NOTE, again, that I can test with
> certs (and CA cert) that
> > have
> > >> smaller CRL files, and those tests work.
> > >>
> > >> So it is only when testing with the
> large CRL file that is
> > failing
> > >> with the OCSP Responder.
> > >>
> > >> This seems like a potential BUG for
> EJBCA OCSP Responder?
> > >>
> > >> Please advise.
> > >>
> > >> Thanks,
> > >> Jim
> > >> On Saturday, June 29, 2019, 3:36:18
> PM EDT,
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> > > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> BTW, just to check, I have actually
> gone through the
> > process of
> > >> getting the crlData from the MySQL
> using mysql client (I
> > had to add
> > >> the --max_allowed-packet=100M to the
> mysql command) and I
> > was able
> > >> to actually get the PEM from the
> Mysql database that way.
> > >>
> > >> So, I KNOW that the entire CRL is in
> the Mysql database,
> > and yet
> > >> when I run an "openssl ocsp" test
> against the EJBCA OCSP
> > Responder,
> > >> it says it cannot find the revocation.
> > >>
> > >> OCSP Request Data:
> > >> Version: 1 (0x0)
> > >> Requestor List:
> > >> Certificate ID:
> > >> Hash Algorithm: sha1
> > >> Issuer Name Hash:
> 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
> > >> Issuer Key Hash:
> 3929A80864E28F83AA816147119133F85A37A6E4
> > >> Serial Number: 2FFFF4
> > >> Request Extensions:
> > >> OCSP Nonce:
> > >> 04103054A60A55E389B239A55228033D8F62
> > >> Response verify OK
> > >> 0x2FFFF4: unknown
> > >> This Update: Jun 29 19:31:02 2019 GMT
> > >>
> > >>
> > >> In the server log:
> > >>
> > >> 2019-06-29 15:31:02,150 DEBUG
> > >>
> >
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
> > >> (default task-1) Unable to find
> revocation information for
> > >> certificate with serial '2ffff4' from
> issuer 'CN=XXX....'
> > >>
> > >>
> > >> Honestly, I am no 100% sure that the
> problem is that
> > >> max_allowe_packet is too small on the
> mysql client side,
> > but that is
> > >> my best guess.
> > >>
> > >> What I AM sure is that the CRL is in
> mysql, but when I do
> > a test
> > >> with OCSP Responder, it cannot find
> the cert serial number
> > in the CRL.
> > >>
> > >> Jim
> > >>
> > >>
> > >> On Saturday, June 29, 2019, 4:52:20
> AM EDT,
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> > > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>> wrote:
> > >>
> > >>
> > >> Hi Tomas,
> > >>
> > >> I already had the max_allowed_packet
> configured on the Mysql
> > >> server-side (in /etc/my.cnf).
> > >>
> > >> So as I said, when testing with mysql
> (client) I had to
> > include
> > >> "--max_allowed_packet=100M" on the
> mysql (client) command
> > line, e.g.:
> > >>
> > >> mysql -u root -p
> --max_allowed_packet=100M
> > >>
> > >> If I did NOT include the
> "--max_allowed_packet=100M" on
> > that command
> > >> line, then when I did a "select" on
> the field base64Crl, I
> > would get
> > >> an error.
> > >>
> > >> I suspect that EJBCA is acting as a
> Mysql client, so, in
> > order to
> > >> work with such large CRLs, there
> needs to be some way to
> > set the
> > >> equivalent of the
> "--max_allowed_packet=100M" on the EJBCA
> > side.
> > >>
> > >> How can I do that?
> > >>
> > >> If that can't be done, then I think
> that when EJBCA tries to
> > >> retrieve the CRL from the Mysql, it
> (EJBCA) will see the
> > same error.
> > >>
> > >> Thanks,
> > >> Jim
> > >>
> > >> On Saturday, June 29, 2019, 3:25:47
> AM EDT, Tomas Gustavsson
> > >> <to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>
> > <mailto:to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>>> wrote:
> > >>
> > >>
> > >>
> > >> You configure max allowed packet size
> in you MySQL
> > configuration on the
> > >> MySQL server.
> > >>
> > >> /Tomas
> > >>
> > >> On 2019-06-29 02:19, o haya via
> Ejbca-develop wrote:
> > >> > I am not 100% sure, but I think I
> know what might be
> > causing this
> > >> problem.
> > >> >
> > >> > I did a test, where I just tried to
> log into mysql and
> > do the
> > >> following
> > >> > select:
> > >> >
> > >> > select base64Crl INTO OUTFILE
> > '/var/lib/mysql-files/xx.out' from
> > >> CRLData
> > >> > where cRLNumber=3167;
> > >> >
> > >> > and I got an error saying that the
> result was more
> > >> than max_allowed_packet.
> > >> >
> > >> > So then I did the same thing, but
> for the mysql command
> > I used:
> > >> >
> > >> > mysql -u root -p
> --max_allowed_packet=100M
> > >> >
> > >> > and then when I did that same
> select, I was able to
> > output the entire
> > >> > CRL as a PEM.
> > >> >
> > >> > So, I am theorizing that that is
> what is happening with
> > EJBCA, i.e.,
> > >> > when I run the "openssl ocsp"
> command, EJBCA is querying the
> > > base64CRL
> > >> > but the result is too large and so
> that is causing the
> > query to fail.
> > >> >
> > >> > So the question now is: How can I
> cause EJBCA OCSP
> > Responder to run
> > >> > with a larger max_allowed_packet so
> that the query
> > succeeds (and then
> > >> > hopefully EJBCA OCSP Responder will
> be able to check the
> > revocation
> > >> > properly)?
> > >> >
> > >> > Does anyone know how that can be done?
> > >> >
> > >> > Thanks,
> > >> > Jim
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Friday, June 28, 2019, 07:16:39
> PM UTC, o haya
> > > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>
> > >> <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>
> > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
> > >> >
> > >> >
> > >> > Hi,
> > >> >
> > >> > As discussed in an earlier thread,
> it looks like I was
> > able to
> > >> import a
> > >> > large (> 71MB) CRL file into EJBCA
> (as an OCSP
> > Responder), so now,
> > >> I am
> > >> > doing testing, using "openssl ocsp"
> to send requests
> > into the
> > >> EJBCA OCSP
> > >> > Responder:
> > >> >
> > >> > openssl ocsp -issuer <CA_CERT>
> -VAfile <SIGNING_CERT>
> > >> -req_text -serial
> > >> > 0x<CERT_SERIAL> -host <HOST:PORT>
> > >> >
> > >> > But it looks like no matter what
> serial number I use,
> > whether the
> > >> serial
> > >> > number is in the CRL or not,
> returns a response like:
> > >> >
> > >> > OCSP Request Data:
> > >> > Version: 1 (0x0)
> > >> > Requestor List:
> > >> > Certificate ID:
> > >> > Hash Algorithm: sha1
> > >> > Issuer Name Hash: 37...00
> > >> > Issuer Key Hash: 39...E4
> > >> > Serial Number: 2FFFF9
> > >> > Request Extensions:
> > >> > OCSP Nonce:
> > >> >
> 041078B1311984A110B5E19C8DC4895F485B
> > >> > Response verify OK
> > >> > 0x2FFFF9: unknown
> > >> > This Update: Jun 28
> 18:56:20 2019 GMT
> > >> >
> > >> > i.e., it looks like the EJBCA OCSP
> Responder is not able
> > to actually
> > >> > find any revoked certs in the CRL?
> > >> >
> > >> >
> > >> > FYI, I *KNOW* that the EJBCA OCSP
> Responder is functioning
> > >> properly, in
> > >> > general, because I can do the same
> test against a
> > smaller CRL in the
> > >> > same EJBCA OCSP Responder, for CA
> simpleca, and I can
> > get an actual
> > >> > "Revoked" response:
> > >> >
> > >> > [root@ejbca <mailto:root@ejbca>
> <mailto:root@ejbca <mailto:root@ejbca>>
> <mailto:root@ejbca <mailto:root@ejbca>
> > <mailto:root@ejbca <mailto:root@ejbca>>>
> <mailto:root@ejbca <mailto:root@ejbca>
> <mailto:root@ejbca <mailto:root@ejbca>>
> > > <mailto:root@ejbca <mailto:root@ejbca>
> <mailto:root@ejbca <mailto:root@ejbca>>>>
> testocspresponder]#
> > openssl ocsp
> > >> -issuer simpleca.crt
> > >> > -VAfile simplecabinding2.crt -serial
> > 0x008486394c03e1f5d9 -req_text
> > >> > -host 192.168.0.28:80
> > >> > OCSP Request Data:
> > >> > Version: 1 (0x0)
> > >> > Requestor List:
> > >> > Certificate ID:
> > >> > Hash Algorithm: sha1
> > >> > Issuer Name Hash:
> > 0C16107310427EA4ADB3C6436915CE44A15FFE55
> > >> > Issuer Key Hash:
> > E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
> > >> > Serial Number:
> 8486394C03E1F5D9
> > >> > Request Extensions:
> > >> > OCSP Nonce:
> > >> >
> 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
> > >> > Response verify OK
> > >> > 0x008486394c03e1f5d9: revoked
> > >> > This Update: Jun 28
> 19:12:48 2019 GMT
> > >> > Reason: unspecified
> > >> > Revocation Time: May 26
> 12:30:44 2019 GMT
> > >> >
> > >> > So it seems like the EJBCA OCSP
> Responder is either not
> > > processing the
> > >> > large CRL properly, or maybe the
> import didn't import
> > the large
> > >> CRL into
> > >> > the OCSP Responder correctly.
> > >> >
> > >> > Can anyone tell me how do I
> diagnose this problem?
> > >> >
> > >> > Thanks,
> > >> > Jim
> > >>
> > >> >
> > >> >
> > >> >
> _______________________________________________
> > >> > Ejbca-develop mailing list
> > >> > Ejb...@li.......
[truncated message content] |
|
From: Tomas G. <to...@pr...> - 2019-07-19 06:19:58
|
That looks about right. What version of database are you using? I just
tried the index command (on line 23 in the SQL file) myself and it works.
CREATE INDEX certificatedata_idx4 ON CertificateData (subjectDN);
What does your database creation say?
>show create table CertiifcateData;
It should look something like this:
| CertificateData | CREATE TABLE `CertificateData` (
`fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
DEFAULT NULL,
`certificateProfileId` int(11) NOT NULL,
`crlPartitionIndex` int(11) DEFAULT NULL,
`endEntityProfileId` int(11) DEFAULT NULL,
`expireDate` bigint(20) NOT NULL,
`issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`notBefore` bigint(20) DEFAULT NULL,
`revocationDate` bigint(20) NOT NULL,
`revocationReason` int(11) NOT NULL,
`rowVersion` int(11) NOT NULL,
`serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`status` int(11) NOT NULL,
`subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
DEFAULT NULL,
`subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
DEFAULT NULL,
`tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`type` int(11) NOT NULL,
`updateTime` bigint(20) NOT NULL,
`username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`base64Cert` longtext,
`certificateRequest` longtext,
`rowProtection` longtext,
PRIMARY KEY (`fingerprint`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
Is the database engine wrong? I.e. not InnoDB?
I am importing the CRL right now and processing about 27 records per
second on my laptop.
Regards,
Tomas
On 2019-07-18 18:09, ohaya--- via Ejbca-develop wrote:
> According to my notes I used these commands:
>
> |*/CREATE DATABASE ejbca CHARACTER SET utf8 COLLATE utf8_general_ci;/*|
>
> |*/ /*|
>
> |*/GRANT ALL PRIVILEGES ON ejbca.* TO 'ejbca'@'localhost' IDENTIFIED BY
> 'ejbca';/*|
>
>
>
>
>
> On Thursday, July 18, 2019, 3:37:51 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> What command did you use to create the database?
>
> On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> This is really getting frustrating!!!
>
> [root@ejbca sql-scripts]# mysql -u root -p'xxxxxx' ejbca <
> create-index-ejbca.sql
> mysql: [Warning] Using a password on the command line interface can
> be insecure.
> ERROR 1071 (42000) at line 23: Specified key was too long; max key
> length is 1000 bytes
>
> Jim
>
>
>
>
>
> On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya...
> <oh...@ya...> wrote:
>
>
> Hi,
>
> The "machine" is a virtual machine under VirtualBox on a machine I
> have that has 64GB physical RAM. Drives are all 7200 RPM drives.
>
> I will restore the snapshot, then increase the memory and # of
> processors and try to run the create index again.
>
> Jim
>
> On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Hi,
>
> Just run create index script.
>
> Create index should run pretty fast even with millions of records in
> the database. I suspect that your database is configured very slow,
> perhaps running everything on disl, without RAM, using slow disks?
>
> A standard installation of mysql/mariadb on a centos machine should
> run faster. Then you have pretty low amount of ram for running
> everything on one box, handing millions of records.
>
> Regards,
> Tomas
>
> On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> I had taken a snapshot of the machine before I tried to do the
> index earlier, so I restored that snapshot.
>
> Then, I tried running the "optimize-ejbca-mysql.sql" and I got a
> different/new error:
>
> ERROR 1091 (42000) at line 27: Can't DROP
> 'certificatedata_idx5'; check that column/key exists
>
> Jim
>
>
>
>
>
>
> On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
> <oh...@ya...> wrote:
>
>
> Hi,
>
> Oh oh, I just got this from the mysql execution:
>
> ERROR 1071 (42000) at line 23: Specified key was too long; max
> key length is 1000 bytes
>
> Jim
>
>
>
>
>
>
> On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
> <oh...@ya...> wrote:
>
>
> Hi,
>
> FYI, I am running the create-index-ejbca.sql script now, but
> that seems to be taking FOREVER?
>
> I had stopped the earlier import, and then deleted all the rows
> for that CA DN in CertificateData table, and then bounced the
> machine, then I ran the create-index-ejbca.sql.
>
> Why is it taking so long? Should I have done something in
> addition to deleting the rows? I am guessing that the reason
> the sql script is taking so long is that it's somehow still
> trying to index the rows were deleted or something like that?
>
> Thanks,
> Jim
>
>
>
> On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
> <oh...@ya...> wrote:
>
>
> Hi,
>
> I notice that there is:
>
> - create-index-ejbca.sql
> - optimize-ejbca-mysql.sql
>
> Should I run BOTH of them, or just the "create-index-ejbca.sql "?
>
> Thanks,
> Jim
>
>
> On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
> Yes, i could see from your output from top that mysql takes all
> CPU. This indicates that you don't have any indexes in your
> database, which makes it horribly slow as numbers grow.
>
> Apply database indexes from doc/sql-scripts.
>
> Regards,
> Tomas
>
>
> On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via
> Ejbca-develop <ejb...@li...> wrote:
>
> Hi,
>
> So far, processed only 185706 entries, and it looks like the
> processing is going to a crawl (really slow now).
>
> Which kind of makes me wonder if maybe part of reason for
> the slowness is that it is having to search the certificate
> database for the serial number, and now, as there are more
> dummy certs in the database, that search is taking longer
> and longer?
>
> Jim
>
>
> On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via
> Ejbca-develop <ejb...@li...> wrote:
>
>
> Hi,
>
> This is what top is showing now (the import is still running):
>
> top - 13:56:19 up 1 day, 1:09, 3 users, load average:
> 1.18, 1.11, 1.07
> Tasks: 167 total, 2 running, 165 sleeping, 0 stopped,
> 0 zombie
> %Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0
> hi, 0.3 si, 0.0 st
> KiB Mem : 4045880 total, 123192 free, 3053072 used,
> 869616 buff/cache
> KiB Swap: 3905532 total, 3753980 free, 151552 used.
> 688136 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU
> %MEM TIME+ COMMAND
> 1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2
> 1227:41 mysqld
> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2
> 139:05.67 java
> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3
> 176:26.68 java
> 22466 root 20 0 0 0 0 S 0.7 0.0
> 0:00.73 kworker/u4:1
>
> That is running on 2-CPU machine with 4GB RAM.
>
> Jim
>
>
>
>
> On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas
> Gustavsson <to...@pr...> wrote:
>
>
> Yes, like top.
>
>
> On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via
> Ejbca-develop <ejb...@li...> wrote:
>
> Hi Tomas,
>
> I'm sorry that I missed the message below, but I am not
> clear what you are asking me to check/monitor? Do you
> mean like running something like "top" command?
>
> Thanks,
> Jim
>
> On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas
> Gustavsson <to...@pr...> wrote:
>
>
> To verify that the changes actually happen...
>
> Do you monitor CPU usage to see what processes use CPU
> and memory?
>
>
> On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via
> Ejbca-develop <ejb...@li...> wrote:
>
> Hi,
>
> The import has been running for about an hour now
> and has imported 17836 entries.
>
> Per what I had said previously, this CRL file has
> 1002255 entries, that means it would take about 56+
> hours to import this CRL.
>
> So, this import processing is still way too slow.
>
> I don't know what is involved, or even how much it
> would improve things, but I think that the next
> thing to try is to eliminated both the check for the
> serial number in the DB AND the dummy cert creation.
>
> Jim
>
>
> On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya---
> via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi,
>
> Ok, I rebuilt the JAR using your Java class, and it
> is running now, and creating rows in the
> CertificateData, but it still SEEMS kind of slow.
>
> I am running now and will try to get a measurement
> of revoked entries/time and will post back.
>
> Jim
>
>
>
>
> On Tuesday, July 16, 2019, 5:25:32 PM UTC,
> oh...@ya... <oh...@ya...> wrote:
>
>
> Hi,
>
> So I just tried what I said (emptying the "if") and
> you are right, it doesn't create the rows in the
> CeritificateData table :(...
>
> Ok, I will try your version...
>
> Jim
>
> On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas
> Gustavsson <to...@pr...> wrote:
>
>
>
> Hi,
>
> I don't think so, no quite actually. In order to use
> the API to populate
> the database you need the certificate object it
> seems. Something like
> the attached file should work, it still creates a
> dummy certificate, but
> uses a static hard coded key, so no key generation.
>
> The file assumes you are on 6.15.x.
>
> Regards,
> Tomas
>
> On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
> > For testing (only...), I am thinking that if I
> modify this If clause in
> > CaImportCrlCommand.java:
> >
> > if (adaptive)
> > {
> > Date time = new Date();
> > KeyPair key_pair =
> KeyTools.genKeys("2048", "RSA");
> >
> > SubjectPublicKeyInfo pkinfo =
> >
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
> > X500Name dnName = new
> X500Name("CN=Dummy Missing in
> > Imported CRL, serialNumber=" + serialHex);
> > Date notAfter = new
> Date(time.getTime() + 315360000000L);
> > X509v3CertificateBuilder certbuilder
> = new
> > X509v3CertificateBuilder(X500Name.getInstance(cacert
> >
> .getSubjectX500Principal().getEncoded()), serialNr,
> > time, notAfter, dnName, pkinfo);
> > ContentSigner signer = new
> BufferingContentSigner(new
> >
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
> > .getPrivate()), 20480);
> > X509CertificateHolder certHolder =
> certbuilder.build(signer);
> > X509Certificate certificate =
> >
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
> > X509Certificate.class);
> >
> > String fingerprint =
> > CertTools.getFingerprintAsString(certificate);
> >
> > EndEntityInformation
> missingUserEndEntityInformation =
> >
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
> > missing_user_name);
> > if (missingUserEndEntityInformation
> == null)
> > {
> > log.debug("Loading/updating user "
> + missing_user_name);
> >
> > EndEntityInformation userdataNew = new
> > EndEntityInformation(missing_user_name,
> > CertTools.getSubjectDN(certificate),
> cainfo.getCAId(), null, null, 10,
> > new EndEntityType(EndEntityTypes.ENDUSER), 1, 1,
> null, null, 1, 0, null);
> >
> > userdataNew.setPassword("foo123");
> >
> >
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
> > userdataNew, false);
> >
> > log.info("User '" +
> missing_user_name + "' has been
> > added.");
> >
> >
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
> > missing_user_name, 50);
> >
> > log.info("User '" +
> missing_user_name + "' has been
> > updated.");
> > }
> >
> >
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
> >
> > EJBTools.wrap(certificate),
> missing_user_name,
> > fingerprint, 20, 1, 1, 0, null, new Date()
> >
> > .getTime());
> > log.info("Dummy certificate '" +
> serialHex + "' has been
> > stored.");
> > }
> >
> > to (for example):
> >
> > if (adaptive)
> > {
> >
> > log.info("*** SKIPPING *** Dummy
> certificate '" +
> > serialHex + "' has been stored.");
> > }
> >
> >
> > Would THAT work??
> >
> > Thanks,
> > Jim
> >
> >
> >
> >
> > On Tuesday, July 16, 2019, 10:59:31 AM UTC,
> ohaya--- via Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > Hi Tomas (or anyone else),
> >
> > Sorry - I was typing my last msg on phone, and it
> wasn't as detailed or
> > clear as I wanted.
> >
> > Basically what I was saying is I was thinking
> that, as a quick sanity
> > test, if there was one or two places where the code in
> > the CaImportCRLCommand.java was maybe making a
> method call to create the
> > dummy cert and put it into the database, then
> maybe I could decompile
> > and modify the CaImportCRLCommand.java, and
> comment out those method
> > calls, then, re-do my large CRL import test and
> time it?
> >
> > If so, can you let me know specifically where
> those calls are, and if I
> > just comment them out, would that work?
> >
> > Ironically, that (not doing the dummy cert
> creation and writing) was
> > what I *thought* the "LENIENT" option was doing).
> >
> > Thanks,
> > Jim
> >
> > P. S. I have time today to try this if I figure
> out what needs to be done.
> >
> >
> >
> > On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya---
> via Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> >
> > Yes, we are definitely interested/willing to test
> patches. I actually
> > was wondering if it would be a quick test to mod
> the code somewhere to
> > avoid a call that creates the dummy cert, to get
> an idea if that would
> > reduce the import time enough??
> >
> >
> > Sent from Yahoo Mail on Android
> >
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
> >
> > On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
> > <to...@pr... <mailto:to...@pr...>>
> wrote:
> > Most large users, some very very large,
> typically uses on line
> > updates directly from the CA. Many CAs today
> don't accept only
> > importing CRLs, as they need to keep track of
> issued certificates as
> > well not only revoked (can't use "non existing
> is good").
> > Having said thst, it's interesting to be able
> to import large CRLs,
> > so improving it is of interest.
> >
> > Importing CRLs would actually not have to
> generate a dummy cert at
> > all, since a certificate object is not needed
> on the responder,
> > which should make it very fast.
> >
> > If you are willing to test patches, we can work
> with you on improving.
> >
> > Regards,
> > Tomas
> >
> >
> > On July 15, 2019 5:26:08 PM GMT+02:00, ohaya---
> via Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> > Hi,
> >
> > I am curious whether you have other users
> that import large CRLs
> > like this?
> >
> > Are we the only ones who would encounter
> these really long
> > import times, because the other users who
> might have large CRLs
> > use the EJBCA CA also, so they don't run
> into having the dummy
> > certs being created for every entry in the CRL?
> >
> > Jim
> >
> > On Monday, July 15, 2019, 1:42:57 PM UTC,
> Tomas Gustavsson
> > <to...@pr...
> <mailto:to...@pr...>> wrote:
> >
> >
> >
> > I see...
> >
> > I can see that a new key pair is being
> generated for each dummy
> > cert.
> >
> > In CaImportCRLCommand.java:
> >
> > �� final KeyPair key_pair =
> KeyTools.genKeys("2048",
> > AlgorithmConstants.KEYALGORITHM_RSA);
> >
> > Chaning that to 1024 or 512 would be an
> easy improvement. A bit more
> > complex, but even larger speedup would be
> to use a "hard coded"
> > dummy
> > key, i.e. same key for all dummy certs.
> >
> > You can experiment by changing 2048 to 512
> (not secure but it
> > doesn't
> > have to be for this dummy cert).
> >
> > Regards,
> > Tomas
> >
> > On 2019-07-11 14:19, ohaya--- via
> Ejbca-develop wrote:
> > > FYI, the is running, but it is REALLY
> slow. The CRL file has
> > 1002255
> > > entries in it, and the import is only up
> to 8899 now, after
> > about 40
> > > minutes... which would take about 75 hours!!
> > >
> > > In our production systems, we are
> required to update some CRLs
> > every
> > > hour, so that would be a problem.
> > >
> > > What can I do to speed that up??
> > >
> > > Thanks,
> > > Jim
> > >
> > >
> > > On Thursday, July 11, 2019, 11:09:33 AM
> UTC, ohaya--- via
> > Ejbca-develop
> > > <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> > >
> > >
> > > AHHH... Ok, it is running now, and from
> checking the
> > CertificateData in
> > > the Mysql database, I can see that it is
> now populating the
> > > CertificateData!!
> > >
> > > Thanks!
> > >
> > > Jim
> > >
> > >
> > >
> > > On Thursday, July 11, 2019, 11:07:22 AM
> UTC, ohaya--- via
> > Ejbca-develop
> > > <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> > >
> > >
> > > I wasn't sure if we should use ADAPTIVE,
> since we are not
> > using the CA part.
> > >
> > >
> > > I will try with "-o ADAPTIVE" now.
> > >
> > >
> > >
> > >
> > > On Thursday, July 11, 2019, 7:42:14 AM
> UTC, Tomas Gustavsson
> > > <to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>> wrote:
> > >
> > >
> > >
> > > It should be ADAPTIVE.
> > >
> > > The documentation says:
> > > bin/ejbca.sh ca importcrl --help
> > > ---
> > > ADAPTIVE means that missing certficates
> will be replaced by dummy
> > > certificates to cater for proper CRLs for
> missing certificates.
> > > ---
> > >
> > > Regards,
> > > Tomas
> > >
> > > On 2019-07-11 08:05, ohaya--- via
> Ejbca-develop wrote:
> > >> Hi,
> > >>
> > >> I've been looking at the code in
> CaImportCRLCommand, and I am
> > wondering
> > >> if, when the command is run in LENIENT
> mode and it doesn't
> > find the cert
> > >> in the database (and outputs the
> "missing in database" error
> > message),
> > >> what is supposed to happen?
> > >>
> > >> Mind you, the only error messages I am
> seeing are the "missing in
> > >> database" error messages. I don't see
> any of the other error
> > messages
> > >> that are in/from CaImportCRLCommand.java.
> > >>
> > >> I am wondering if, when the command is
> run with "-o LENIENT",
> > and when
> > >> it outputs the "missing in database"
> error message, then the
> > ejbca.sh is
> > >> basically not doing anything (and not
> storing anything into the
> > >> CertificateData table)?
> > >>
> > >> From the lack of any other error
> messages (vs. what I see in the
> > >> CaImportCRLCommand.java code), that is
> what it looks like is
> > happening.
> > >>
> > >> So given I am using EJBCA for OCSP
> responder only, is running the
> > >> ejbca.sh in the "-o LENIENT" mode correct?
> > >>
> > >> Jim
> > >>
> > >>
> > >>
> > >> On Thursday, July 11, 2019, 5:43:37 AM
> UTC, ohaya--- via
> > Ejbca-develop
> > >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>
> > >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> I've been doing a bunch of testing with
> trying to import the
> > large CRL,
> > >> and wanted to summarize:
> > >>
> > >> - The ejbca.sh runs fine - it completes
> without outputting an
> > errors,
> > >> except for outputting a bunch of the
> "cert xxxx missing in
> > database"
> > >> messages.
> > >>
> > >> - When the import completes, and I check
> the Mysql database,
> > the crlData
> > >> (the full PEM of the CRL) is in the
> database, but there is no
> > rows in
> > >> the CertificateData table for that CA at
> all.
> > >>
> > >> - I even figured out how to get the
> ejbca.sh to log using
> > >> log4j.properties, and it output a bunch
> of logging, mainly
> > the missing
> > >> in database errors, but other than that
> I didn't see any
> > obvious errors.
> > >>
> > >>
> > >> So, bottom line is that even though the
> ejbca.sh seems to run
> > without
> > >> errors (other than the missing in
> database errors), the import is
> > >> causing the CRL PEM to be stored in the
> database, but the
> > >> CertificateData table is not being
> populated at all for that CA.
> > >>
> > >>
> > >> Jim
> > >>
> > >>
> > >>
> > >> On Wednesday, July 10, 2019, 7:40:16 PM
> UTC, ohaya--- via
> > Ejbca-develop
> > >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>
> > >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> Ok, I found Tomas' original message...
> it was in my Spam
> > folder....
> > >>
> > >> I also tried re-running the "./ejbca.sh
> ca importcrl" command
> > a bunch of
> > >> ways. I think it is running without
> outputting any errors,
> > but still,
> > >> no rows are appearing with that issuerDN
> at all.
> > >>
> > >>
> > >>
> > >>
> > >> On Tuesday, July 9, 2019, 10:45:01 AM
> UTC, Tomas Gustavsson
> > >> <to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>
> > <mailto:to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> OCSP responses does not have anything to
> do with the CRL
> > itself. Once
> > >> the CRL has been imported each
> certificate status is in its
> > own record
> > >> in the CertificateData table. This is
> where status is read
> > when reponses
> > >> are created.
> > >> If there are issues, it is in that case
> related to the import
> > of the CRL.
> > >>
> > >> Verify that certificate records are
> created in
> > CertificateData when
> > >> importing the CRL.
> > >> Also, a debug log when making OCSP
> request will tell exact
> > details of
> > >> what the responder finds in the database
> or not.
> > >>
> > >> Regards,
> > >> Tomas
> > >>
> > >>
> > >> On July 8, 2019 4:59:06 PM GMT+02:00,
> ohaya--- via Ejbca-develop
> > >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>
> > >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
> > >>
> > >> [Re-sending, as this (handling of
> large CRLs) is still a
> > problem and
> > >> is still not working.]
> > >>
> > >>
> > >>
> > >> On Monday, July 1, 2019, 12:47:23 PM
> UTC, <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya...
> <mailto:oh...@ya...>>
> > > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> I think that I've found how to
> set/change the
> > max_allowed_packet on
> > >> the client (i.e., EJBCA) side, by
> editing the Mysql URL in the
> > >>
> standalone/configuration/standalone.xml to:
> > >>
> > >>
> > >
> >
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
> > >>
> > >> However, when I test the OCSP
> Responder with a serial
> > number (and CA
> > >> cert) from the CA with the large
> (>71MB) CRL file, I am still
> > >> getting "unknown" and EJBCA OCSP
> Responder is still NOT
> > FINDING the
> > >> cert serial number.
> > >>
> > >> NOTE, again, that I can test with
> certs (and CA cert) that
> > have
> > >> smaller CRL files, and those tests work.
> > >>
> > >> So it is only when testing with the
> large CRL file that is
> > failing
> > >> with the OCSP Responder.
> > >>
> > >> This seems like a potential BUG for
> EJBCA OCSP Responder?
> > >>
> > >> Please advise.
> > >>
> > >> Thanks,
> > >> Jim
> > >> On Saturday, June 29, 2019, 3:36:18
> PM EDT,
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> > > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> BTW, just to check, I have actually
> gone through the
> > process of
> > >> getting the crlData from the MySQL
> using mysql client (I
> > had to add
> > >> the --max_allowed-packet=100M to the
> mysql command) and I
> > was able
> > >> to actually get the PEM from the
> Mysql database that way.
> > >>
> > >> So, I KNOW that the entire CRL is in
> the Mysql database,
> > and yet
> > >> when I run an "openssl ocsp" test
> against the EJBCA OCSP
> > Responder,
> > >> it says it cannot find the revocation.
> > >>
> > >> OCSP Request Data:
> > >> Version: 1 (0x0)
> > >> Requestor List:
> > >> Certificate ID:
> > >> Hash Algorithm: sha1
> > >> Issuer Name Hash:
> 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
> > >> Issuer Key Hash:
> 3929A80864E28F83AA816147119133F85A37A6E4
> > >> Serial Number: 2FFFF4
> > >> Request Extensions:
> > >> OCSP Nonce:
> > >> 04103054A60A55E389B239A55228033D8F62
> > >> Response verify OK
> > >> 0x2FFFF4: unknown
> > >> This Update: Jun 29 19:31:02 2019 GMT
> > >>
> > >>
> > >> In the server log:
> > >>
> > >> 2019-06-29 15:31:02,150 DEBUG
> > >>
> >
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
> > >> (default task-1) Unable to find
> revocation information for
> > >> certificate with serial '2ffff4' from
> issuer 'CN=XXX....'
> > >>
> > >>
> > >> Honestly, I am no 100% sure that the
> problem is that
> > >> max_allowe_packet is too small on the
> mysql client side,
> > but that is
> > >> my best guess.
> > >>
> > >> What I AM sure is that the CRL is in
> mysql, but when I do
> > a test
> > >> with OCSP Responder, it cannot find
> the cert serial number
> > in the CRL.
> > >>
> > >> Jim
> > >>
> > >>
> > >> On Saturday, June 29, 2019, 4:52:20
> AM EDT,
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> > > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>> wrote:
> > >>
> > >>
> > >> Hi Tomas,
> > >>
> > >> I already had the max_allowed_packet
> configured on the Mysql
> > >> server-side (in /etc/my.cnf).
> > >>
> > >> So as I said, when testing with mysql
> (client) I had to
> > include
> > >> "--max_allowed_packet=100M" on the
> mysql (client) command
> > line, e.g.:
> > >>
> > >> mysql -u root -p
> --max_allowed_packet=100M
> > >>
> > >> If I did NOT include the
> "--max_allowed_packet=100M" on
> > that command
> > >> line, then when I did a "select" on
> the field base64Crl, I
> > would get
> > >> an error.
> > >>
> > >> I suspect that EJBCA is acting as a
> Mysql client, so, in
> > order to
> > >> work with such large CRLs, there
> needs to be some way to
> > set the
> > >> equivalent of the
> "--max_allowed_packet=100M" on the EJBCA
> > side.
> > >>
> > >> How can I do that?
> > >>
> > >> If that can't be done, then I think
> that when EJBCA tries to
> > >> retrieve the CRL from the Mysql, it
> (EJBCA) will see the
> > same error.
> > >>
> > >> Thanks,
> > >> Jim
> > >>
> > >> On Saturday, June 29, 2019, 3:25:47
> AM EDT, Tomas Gustavsson
> > >> <to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>
> > <mailto:to...@pr...
> <mailto:to...@pr...> <mailto:to...@pr...
> <mailto:to...@pr...>>>> wrote:
> > >>
> > >>
> > >>
> > >> You configure max allowed packet size
> in you MySQL
> > configuration on the
> > >> MySQL server.
> > >>
> > >> /Tomas
> > >>
> > >> On 2019-06-29 02:19, o haya via
> Ejbca-develop wrote:
> > >> > I am not 100% sure, but I think I
> know what might be
> > causing this
> > >> problem.
> > >> >
> > >> > I did a test, where I just tried to
> log into mysql and
> > do the
> > >> following
> > >> > select:
> > >> >
> > >> > select base64Crl INTO OUTFILE
> > '/var/lib/mysql-files/xx.out' from
> > >> CRLData
> > >> > where cRLNumber=3167;
> > >> >
> > >> > and I got an error saying that the
> result was more
> > >> than max_allowed_packet.
> > >> >
> > >> > So then I did the same thing, but
> for the mysql command
> > I used:
> > >> >
> > >> > mysql -u root -p
> --max_allowed_packet=100M
> > >> >
> > >> > and then when I did that same
> select, I was able to
> > output the entire
> > >> > CRL as a PEM.
> > >> >
> > >> > So, I am theorizing that that is
> what is happening with
> > EJBCA, i.e.,
> > >> > when I run the "openssl ocsp"
> command, EJBCA is querying the
> > > base64CRL
> > >> > but the result is too large and so
> that is causing the
> > query to fail.
> > >> >
> > >> > So the question now is: How can I
> cause EJBCA OCSP
> > Responder to run
> > >> > with a larger max_allowed_packet so
> that the query
> > succeeds (and then
> > >> > hopefully EJBCA OCSP Responder will
> be able to check the
> > revocation
> > >> > properly)?
> > >> >
> > >> > Does anyone know how that can be done?
> > >> >
> > >> > Thanks,
> > >> > Jim
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Friday, June 28, 2019, 07:16:39
> PM UTC, o haya
> > > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>
> > >> <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>
> > <mailto:oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
> > >> >
> > >> >
> > >> > Hi,
> > >> >
> > >> > As discussed in an earlier thread,
> it looks like I was
> > able to
> > >> import a
> > >> > large (> 71MB) CRL file into EJBCA
> (as an OCSP
> > Responder), so now,
> > >> I am
> > >> > doing testing, using "openssl ocsp"
> to send requests
> > into the
> > >> EJBCA OCSP
> > >> > Responder:
> > >> >
> > >> > openssl ocsp -issuer <CA_CERT>
> -VAfile <SIGNING_CERT>
> > >> -req_text -serial
> > >> > 0x<CERT_SERIAL> -host <HOST:PORT>
> > >> >
> > >> > But it looks like no matter what
> serial number I use,
> > whether the
> > >> serial
> > >> > number is in the CRL or not,
> returns a response like:
> > >> >
> > >> > OCSP Request Data:
> > >> > Version: 1 (0x0)
> > >> > Requestor List:
> > >> > Certificate ID:
> > >> > Hash Algorithm: sha1
> > >> > Issuer Name Hash: 37...00
> > >> > Issuer Key Hash: 39...E4
> > >> > Serial Number: 2FFFF9
> > >> > Request Extensions:
> > >> > OCSP Nonce:
> > >> >
> 041078B1311984A110B5E19C8DC4895F485B
> > >> > Response verify OK
> > >> > 0x2FFFF9: unknown
> > >> > This Update: Jun 28
> 18:56:20 2019 GMT
> > >> >
> > >> > i.e., it looks like the EJBCA OCSP
> Responder is not able
> > to actually
> > >> > find any revoked certs in the CRL?
> > >> >
> > >> >
> > >> > FYI, I *KNOW* that the EJBCA OCSP
> Responder is functioning
> > >> properly, in
> > >> > general, because I can do the same
> test against a
> > smaller CRL in the
> > >> > same EJBCA OCSP Responder, for CA
> simpleca, and I can
> > get an actual
> > >> > "Revoked" response:
> > >> >
> > >> > [root@ejbca <mailto:root@ejbca>
> <mailto:root@ejbca <mailto:root@ejbca>>
> <mailto:root@ejbca <mailto:root@ejbca>
> > <mailto:root@ejbca <mailto:root@ejbca>>>
> <mailto:root@ejbca <mailto:root@ejbca>
> <mailto:root@ejbca <mailto:root@ejbca>>
> > > <mailto:root@ejbca <mailto:root@ejbca>
> <mailto:root@ejbca <mailto:root@ejbca>>>>
> testocspresponder]#
> > openssl ocsp
> > >> -issuer simpleca.crt
> > >> > -VAfile simplecabinding2.crt -serial
> > 0x008486394c03e1f5d9 -req_text
> > >> > -host 192.168.0.28:80
> > >> > OCSP Request Data:
> > >> > Version: 1 (0x0)
> > >> > Requestor List:
> > >> > Certificate ID:
> > >> > Hash Algorithm: sha1
> > >> > Issuer Name Hash:
> > 0C16107310427EA4ADB3C6436915CE44A15FFE55
> > >> > Issuer Key Hash:
> > E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
> > >> > Serial Number:
> 8486394C03E1F5D9
> > >> > Request Extensions:
> > >> > OCSP Nonce:
> > >> >
> 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
> > >> > Response verify OK
> > >> > 0x008486394c03e1f5d9: revoked
> > >> > This Update: Jun 28
> 19:12:48 2019 GMT
> > >> > Reason: unspecified
> > >> > Revocation Time: May 26
> 12:30:44 2019 GMT
> > >> >
> > >> > So it seems like the EJBCA OCSP
> Responder is either not
> > > processing the
> > >> > large CRL properly, or maybe the
> import didn't import
> > the large
> > >> CRL into
> > >> > the OCSP Responder correctly.
> > >> >
> > >> > Can anyone tell me how do I
> diagnose this problem?
> > >> >
> > >> > Thanks,
> > >> > Jim
> > >>
> > >> >
> > >> >
> > >> >
> _______________________________________________
> > >> > Ejbca-develop mailing list
> > >> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > >
> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> > >>
> <...
[truncated message content] |
|
From: <oh...@ya...> - 2019-07-18 16:09:18
|
According to my notes I used these commands:
CREATE DATABASE ejbca CHARACTER SETutf8 COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON ejbca.* TO 'ejbca'@'localhost' IDENTIFIED BY'ejbca';
On Thursday, July 18, 2019, 3:37:51 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
What command did you use to create the database?
On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
This is really getting frustrating!!!
[root@ejbca sql-scripts]# mysql -u root -p'xxxxxx' ejbca < create-index-ejbca.sql
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1071 (42000) at line 23: Specified key was too long; max key length is 1000 bytes
Jim
On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
The "machine" is a virtual machine under VirtualBox on a machine I have that has 64GB physical RAM. Drives are all 7200 RPM drives.
I will restore the snapshot, then increase the memory and # of processors and try to run the create index again.
Jim
On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
Just run create index script.
Create index should run pretty fast even with millions of records in the database. I suspect that your database is configured very slow, perhaps running everything on disl, without RAM, using slow disks?
A standard installation of mysql/mariadb on a centos machine should run faster. Then you have pretty low amount of ram for running everything on one box, handing millions of records.
Regards,
Tomas
On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I had taken a snapshot of the machine before I tried to do the index earlier, so I restored that snapshot.
Then, I tried running the "optimize-ejbca-mysql.sql" and I got a different/new error:
ERROR 1091 (42000) at line 27: Can't DROP 'certificatedata_idx5'; check that column/key exists
Jim
On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
Oh oh, I just got this from the mysql execution:
ERROR 1071 (42000) at line 23: Specified key was too long; max key length is 1000 bytes
Jim
On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
FYI, I am running the create-index-ejbca.sql script now, but that seems to be taking FOREVER?
I had stopped the earlier import, and then deleted all the rows for that CA DN in CertificateData table, and then bounced the machine, then I ran the create-index-ejbca.sql.
Why is it taking so long? Should I have done something in addition to deleting the rows? I am guessing that the reason the sql script is taking so long is that it's somehow still trying to index the rows were deleted or something like that?
Thanks,Jim
On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
I notice that there is:
- create-index-ejbca.sql
- optimize-ejbca-mysql.sql
Should I run BOTH of them, or just the "create-index-ejbca.sql "?
Thanks,Jim
On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, i could see from your output from top that mysql takes all CPU. This indicates that you don't have any indexes in your database, which makes it horribly slow as numbers grow.
Apply database indexes from doc/sql-scripts.
Regards,
Tomas
On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
So far, processed only 185706 entries, and it looks like the processing is going to a crawl (really slow now).
Which kind of makes me wonder if maybe part of reason for the slowness is that it is having to search the certificate database for the serial number, and now, as there are more dummy certs in the database, that search is taking longer and longer?
Jim
On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
This is what top is showing now (the import is still running):
top - 13:56:19 up 1 day, 1:09, 3 users, load average: 1.18, 1.11, 1.07
Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
%Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 4045880 total, 123192 free, 3053072 used, 869616 buff/cache
KiB Swap: 3905532 total, 3753980 free, 151552 used. 688136 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2 1227:41 mysqld
3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2 139:05.67 java
3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3 176:26.68 java
22466 root 20 0 0 0 0 S 0.7 0.0 0:00.73 kworker/u4:1
That is running on 2-CPU machine with 4GB RAM.
Jim
On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, like top.
On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi Tomas,
I'm sorry that I missed the message below, but I am not clear what you are asking me to check/monitor? Do you mean like running something like "top" command?
Thanks,Jim
On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
To verify that the changes actually happen...
Do you monitor CPU usage to see what processes use CPU and memory?
On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
The import has been running for about an hour now and has imported 17836 entries.
Per what I had said previously, this CRL file has 1002255entries, that means it would take about 56+ hours to import this CRL.
So, this import processing is still way too slow.
I don't know what is involved, or even how much it would improve things, but I think that the next thing to try is to eliminated both the check for the serial number in the DB AND the dummy cert creation.
Jim
On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
Ok, I rebuilt the JAR using your Java class, and it is running now, and creating rows in the CertificateData, but it still SEEMS kind of slow.
I am running now and will try to get a measurement of revoked entries/time and will post back.
Jim
On Tuesday, July 16, 2019, 5:25:32 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
So I just tried what I said (emptying the "if") and you are right, it doesn't create the rows in the CeritificateData table :(...
Ok, I will try your version...
Jim
On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
I don't think so, no quite actually. In order to use the API to populate
the database you need the certificate object it seems. Something like
the attached file should work, it still creates a dummy certificate, but
uses a static hard coded key, so no key generation.
The file assumes you are on 6.15.x.
Regards,
Tomas
On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
> For testing (only...), I am thinking that if I modify this If clause in
> CaImportCrlCommand.java:
>
> if (adaptive)
> {
> Date time = new Date();
> KeyPair key_pair = KeyTools.genKeys("2048", "RSA");
>
> SubjectPublicKeyInfo pkinfo =
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
> X500Name dnName = new X500Name("CN=Dummy Missing in
> Imported CRL, serialNumber=" + serialHex);
> Date notAfter = new Date(time.getTime() + 315360000000L);
> X509v3CertificateBuilder certbuilder = new
> X509v3CertificateBuilder(X500Name.getInstance(cacert
> .getSubjectX500Principal().getEncoded()), serialNr,
> time, notAfter, dnName, pkinfo);
> ContentSigner signer = new BufferingContentSigner(new
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
> .getPrivate()), 20480);
> X509CertificateHolder certHolder = certbuilder.build(signer);
> X509Certificate certificate =
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
> X509Certificate.class);
>
> String fingerprint =
> CertTools.getFingerprintAsString(certificate);
>
> EndEntityInformation missingUserEndEntityInformation =
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
> missing_user_name);
> if (missingUserEndEntityInformation == null)
> {
> log.debug("Loading/updating user " + missing_user_name);
>
> EndEntityInformation userdataNew = new
> EndEntityInformation(missing_user_name,
> CertTools.getSubjectDN(certificate), cainfo.getCAId(), null, null, 10,
> new EndEntityType(EndEntityTypes.ENDUSER), 1, 1, null, null, 1, 0, null);
>
> userdataNew.setPassword("foo123");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
> userdataNew, false);
>
> log.info("User '" + missing_user_name + "' has been
> added.");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
> missing_user_name, 50);
>
> log.info("User '" + missing_user_name + "' has been
> updated.");
> }
>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>
> EJBTools.wrap(certificate), missing_user_name,
> fingerprint, 20, 1, 1, 0, null, new Date()
>
> .getTime());
> log.info("Dummy certificate '" + serialHex + "' has been
> stored.");
> }
>
> to (for example):
>
> if (adaptive)
> {
>
> log.info("*** SKIPPING *** Dummy certificate '" +
> serialHex + "' has been stored.");
> }
>
>
> Would THAT work??
>
> Thanks,
> Jim
>
>
>
>
> On Tuesday, July 16, 2019, 10:59:31 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi Tomas (or anyone else),
>
> Sorry - I was typing my last msg on phone, and it wasn't as detailed or
> clear as I wanted.
>
> Basically what I was saying is I was thinking that, as a quick sanity
> test, if there was one or two places where the code in
> the CaImportCRLCommand.java was maybe making a method call to create the
> dummy cert and put it into the database, then maybe I could decompile
> and modify the CaImportCRLCommand.java, and comment out those method
> calls, then, re-do my large CRL import test and time it?
>
> If so, can you let me know specifically where those calls are, and if I
> just comment them out, would that work?
>
> Ironically, that (not doing the dummy cert creation and writing) was
> what I *thought* the "LENIENT" option was doing).
>
> Thanks,
> Jim
>
> P. S. I have time today to try this if I figure out what needs to be done.
>
>
>
> On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
>
> Yes, we are definitely interested/willing to test patches. I actually
> was wondering if it would be a quick test to mod the code somewhere to
> avoid a call that creates the dummy cert, to get an idea if that would
> reduce the import time enough??
>
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>
> On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
> <to...@pr...> wrote:
> Most large users, some very very large, typically uses on line
> updates directly from the CA. Many CAs today don't accept only
> importing CRLs, as they need to keep track of issued certificates as
> well not only revoked (can't use "non existing is good").
> Having said thst, it's interesting to be able to import large CRLs,
> so improving it is of interest.
>
> Importing CRLs would actually not have to generate a dummy cert at
> all, since a certificate object is not needed on the responder,
> which should make it very fast.
>
> If you are willing to test patches, we can work with you on improving.
>
> Regards,
> Tomas
>
>
> On July 15, 2019 5:26:08 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> I am curious whether you have other users that import large CRLs
> like this?
>
> Are we the only ones who would encounter these really long
> import times, because the other users who might have large CRLs
> use the EJBCA CA also, so they don't run into having the dummy
> certs being created for every entry in the CRL?
>
> Jim
>
> On Monday, July 15, 2019, 1:42:57 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> I see...
>
> I can see that a new key pair is being generated for each dummy
> cert.
>
> In CaImportCRLCommand.java:
>
> final KeyPair key_pair = KeyTools.genKeys("2048",
> AlgorithmConstants.KEYALGORITHM_RSA);
>
> Chaning that to 1024 or 512 would be an easy improvement. A bit more
> complex, but even larger speedup would be to use a "hard coded"
> dummy
> key, i.e. same key for all dummy certs.
>
> You can experiment by changing 2048 to 512 (not secure but it
> doesn't
> have to be for this dummy cert).
>
> Regards,
> Tomas
>
> On 2019-07-11 14:19, ohaya--- via Ejbca-develop wrote:
> > FYI, the is running, but it is REALLY slow. The CRL file has
> 1002255
> > entries in it, and the import is only up to 8899 now, after
> about 40
> > minutes... which would take about 75 hours!!
> >
> > In our production systems, we are required to update some CRLs
> every
> > hour, so that would be a problem.
> >
> > What can I do to speed that up??
> >
> > Thanks,
> > Jim
> >
> >
> > On Thursday, July 11, 2019, 11:09:33 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > AHHH... Ok, it is running now, and from checking the
> CertificateData in
> > the Mysql database, I can see that it is now populating the
> > CertificateData!!
> >
> > Thanks!
> >
> > Jim
> >
> >
> >
> > On Thursday, July 11, 2019, 11:07:22 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > I wasn't sure if we should use ADAPTIVE, since we are not
> using the CA part.
> >
> >
> > I will try with "-o ADAPTIVE" now.
> >
> >
> >
> >
> > On Thursday, July 11, 2019, 7:42:14 AM UTC, Tomas Gustavsson
> > <to...@pr... <mailto:to...@pr...>> wrote:
> >
> >
> >
> > It should be ADAPTIVE.
> >
> > The documentation says:
> > bin/ejbca.sh ca importcrl --help
> > ---
> > ADAPTIVE means that missing certficates will be replaced by dummy
> > certificates to cater for proper CRLs for missing certificates.
> > ---
> >
> > Regards,
> > Tomas
> >
> > On 2019-07-11 08:05, ohaya--- via Ejbca-develop wrote:
> >> Hi,
> >>
> >> I've been looking at the code in CaImportCRLCommand, and I am
> wondering
> >> if, when the command is run in LENIENT mode and it doesn't
> find the cert
> >> in the database (and outputs the "missing in database" error
> message),
> >> what is supposed to happen?
> >>
> >> Mind you, the only error messages I am seeing are the "missing in
> >> database" error messages. I don't see any of the other error
> messages
> >> that are in/from CaImportCRLCommand.java.
> >>
> >> I am wondering if, when the command is run with "-o LENIENT",
> and when
> >> it outputs the "missing in database" error message, then the
> ejbca.sh is
> >> basically not doing anything (and not storing anything into the
> >> CertificateData table)?
> >>
> >> From the lack of any other error messages (vs. what I see in the
> >> CaImportCRLCommand.java code), that is what it looks like is
> happening.
> >>
> >> So given I am using EJBCA for OCSP responder only, is running the
> >> ejbca.sh in the "-o LENIENT" mode correct?
> >>
> >> Jim
> >>
> >>
> >>
> >> On Thursday, July 11, 2019, 5:43:37 AM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I've been doing a bunch of testing with trying to import the
> large CRL,
> >> and wanted to summarize:
> >>
> >> - The ejbca.sh runs fine - it completes without outputting an
> errors,
> >> except for outputting a bunch of the "cert xxxx missing in
> database"
> >> messages.
> >>
> >> - When the import completes, and I check the Mysql database,
> the crlData
> >> (the full PEM of the CRL) is in the database, but there is no
> rows in
> >> the CertificateData table for that CA at all.
> >>
> >> - I even figured out how to get the ejbca.sh to log using
> >> log4j.properties, and it output a bunch of logging, mainly
> the missing
> >> in database errors, but other than that I didn't see any
> obvious errors.
> >>
> >>
> >> So, bottom line is that even though the ejbca.sh seems to run
> without
> >> errors (other than the missing in database errors), the import is
> >> causing the CRL PEM to be stored in the database, but the
> >> CertificateData table is not being populated at all for that CA.
> >>
> >>
> >> Jim
> >>
> >>
> >>
> >> On Wednesday, July 10, 2019, 7:40:16 PM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> Ok, I found Tomas' original message... it was in my Spam
> folder....
> >>
> >> I also tried re-running the "./ejbca.sh ca importcrl" command
> a bunch of
> >> ways. I think it is running without outputting any errors,
> but still,
> >> no rows are appearing with that issuerDN at all.
> >>
> >>
> >>
> >>
> >> On Tuesday, July 9, 2019, 10:45:01 AM UTC, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> OCSP responses does not have anything to do with the CRL
> itself. Once
> >> the CRL has been imported each certificate status is in its
> own record
> >> in the CertificateData table. This is where status is read
> when reponses
> >> are created.
> >> If there are issues, it is in that case related to the import
> of the CRL.
> >>
> >> Verify that certificate records are created in
> CertificateData when
> >> importing the CRL.
> >> Also, a debug log when making OCSP request will tell exact
> details of
> >> what the responder finds in the database or not.
> >>
> >> Regards,
> >> Tomas
> >>
> >>
> >> On July 8, 2019 4:59:06 PM GMT+02:00, ohaya--- via Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >> [Re-sending, as this (handling of large CRLs) is still a
> problem and
> >> is still not working.]
> >>
> >>
> >>
> >> On Monday, July 1, 2019, 12:47:23 PM UTC, <oh...@ya...
> <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I think that I've found how to set/change the
> max_allowed_packet on
> >> the client (i.e., EJBCA) side, by editing the Mysql URL in the
> >> standalone/configuration/standalone.xml to:
> >>
> >>
> >
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
> >>
> >> However, when I test the OCSP Responder with a serial
> number (and CA
> >> cert) from the CA with the large (>71MB) CRL file, I am still
> >> getting "unknown" and EJBCA OCSP Responder is still NOT
> FINDING the
> >> cert serial number.
> >>
> >> NOTE, again, that I can test with certs (and CA cert) that
> have
> >> smaller CRL files, and those tests work.
> >>
> >> So it is only when testing with the large CRL file that is
> failing
> >> with the OCSP Responder.
> >>
> >> This seems like a potential BUG for EJBCA OCSP Responder?
> >>
> >> Please advise.
> >>
> >> Thanks,
> >> Jim
> >> On Saturday, June 29, 2019, 3:36:18 PM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> BTW, just to check, I have actually gone through the
> process of
> >> getting the crlData from the MySQL using mysql client (I
> had to add
> >> the --max_allowed-packet=100M to the mysql command) and I
> was able
> >> to actually get the PEM from the Mysql database that way.
> >>
> >> So, I KNOW that the entire CRL is in the Mysql database,
> and yet
> >> when I run an "openssl ocsp" test against the EJBCA OCSP
> Responder,
> >> it says it cannot find the revocation.
> >>
> >> OCSP Request Data:
> >> Version: 1 (0x0)
> >> Requestor List:
> >> Certificate ID:
> >> Hash Algorithm: sha1
> >> Issuer Name Hash: 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
> >> Issuer Key Hash: 3929A80864E28F83AA816147119133F85A37A6E4
> >> Serial Number: 2FFFF4
> >> Request Extensions:
> >> OCSP Nonce:
> >> 04103054A60A55E389B239A55228033D8F62
> >> Response verify OK
> >> 0x2FFFF4: unknown
> >> This Update: Jun 29 19:31:02 2019 GMT
> >>
> >>
> >> In the server log:
> >>
> >> 2019-06-29 15:31:02,150 DEBUG
> >>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
> >> (default task-1) Unable to find revocation information for
> >> certificate with serial '2ffff4' from issuer 'CN=XXX....'
> >>
> >>
> >> Honestly, I am no 100% sure that the problem is that
> >> max_allowe_packet is too small on the mysql client side,
> but that is
> >> my best guess.
> >>
> >> What I AM sure is that the CRL is in mysql, but when I do
> a test
> >> with OCSP Responder, it cannot find the cert serial number
> in the CRL.
> >>
> >> Jim
> >>
> >>
> >> On Saturday, June 29, 2019, 4:52:20 AM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi Tomas,
> >>
> >> I already had the max_allowed_packet configured on the Mysql
> >> server-side (in /etc/my.cnf).
> >>
> >> So as I said, when testing with mysql (client) I had to
> include
> >> "--max_allowed_packet=100M" on the mysql (client) command
> line, e.g.:
> >>
> >> mysql -u root -p --max_allowed_packet=100M
> >>
> >> If I did NOT include the "--max_allowed_packet=100M" on
> that command
> >> line, then when I did a "select" on the field base64Crl, I
> would get
> >> an error.
> >>
> >> I suspect that EJBCA is acting as a Mysql client, so, in
> order to
> >> work with such large CRLs, there needs to be some way to
> set the
> >> equivalent of the "--max_allowed_packet=100M" on the EJBCA
> side.
> >>
> >> How can I do that?
> >>
> >> If that can't be done, then I think that when EJBCA tries to
> >> retrieve the CRL from the Mysql, it (EJBCA) will see the
> same error.
> >>
> >> Thanks,
> >> Jim
> >>
> >> On Saturday, June 29, 2019, 3:25:47 AM EDT, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >>
> >> You configure max allowed packet size in you MySQL
> configuration on the
> >> MySQL server.
> >>
> >> /Tomas
> >>
> >> On 2019-06-29 02:19, o haya via Ejbca-develop wrote:
> >> > I am not 100% sure, but I think I know what might be
> causing this
> >> problem.
> >> >
> >> > I did a test, where I just tried to log into mysql and
> do the
> >> following
> >> > select:
> >> >
> >> > select base64Crl INTO OUTFILE
> '/var/lib/mysql-files/xx.out' from
> >> CRLData
> >> > where cRLNumber=3167;
> >> >
> >> > and I got an error saying that the result was more
> >> than max_allowed_packet.
> >> >
> >> > So then I did the same thing, but for the mysql command
> I used:
> >> >
> >> > mysql -u root -p --max_allowed_packet=100M
> >> >
> >> > and then when I did that same select, I was able to
> output the entire
> >> > CRL as a PEM.
> >> >
> >> > So, I am theorizing that that is what is happening with
> EJBCA, i.e.,
> >> > when I run the "openssl ocsp" command, EJBCA is querying the
> > base64CRL
> >> > but the result is too large and so that is causing the
> query to fail.
> >> >
> >> > So the question now is: How can I cause EJBCA OCSP
> Responder to run
> >> > with a larger max_allowed_packet so that the query
> succeeds (and then
> >> > hopefully EJBCA OCSP Responder will be able to check the
> revocation
> >> > properly)?
> >> >
> >> > Does anyone know how that can be done?
> >> >
> >> > Thanks,
> >> > Jim
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Friday, June 28, 2019, 07:16:39 PM UTC, o haya
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> >> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>> wrote:
> >> >
> >> >
> >> > Hi,
> >> >
> >> > As discussed in an earlier thread, it looks like I was
> able to
> >> import a
> >> > large (> 71MB) CRL file into EJBCA (as an OCSP
> Responder), so now,
> >> I am
> >> > doing testing, using "openssl ocsp" to send requests
> into the
> >> EJBCA OCSP
> >> > Responder:
> >> >
> >> > openssl ocsp -issuer <CA_CERT> -VAfile <SIGNING_CERT>
> >> -req_text -serial
> >> > 0x<CERT_SERIAL> -host <HOST:PORT>
> >> >
> >> > But it looks like no matter what serial number I use,
> whether the
> >> serial
> >> > number is in the CRL or not, returns a response like:
> >> >
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash: 37...00
> >> > Issuer Key Hash: 39...E4
> >> > Serial Number: 2FFFF9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 041078B1311984A110B5E19C8DC4895F485B
> >> > Response verify OK
> >> > 0x2FFFF9: unknown
> >> > This Update: Jun 28 18:56:20 2019 GMT
> >> >
> >> > i.e., it looks like the EJBCA OCSP Responder is not able
> to actually
> >> > find any revoked certs in the CRL?
> >> >
> >> >
> >> > FYI, I *KNOW* that the EJBCA OCSP Responder is functioning
> >> properly, in
> >> > general, because I can do the same test against a
> smaller CRL in the
> >> > same EJBCA OCSP Responder, for CA simpleca, and I can
> get an actual
> >> > "Revoked" response:
> >> >
> >> > [root@ejbca <mailto:root@ejbca> <mailto:root@ejbca
> <mailto:root@ejbca>> <mailto:root@ejbca <mailto:root@ejbca>
> > <mailto:root@ejbca <mailto:root@ejbca>>> testocspresponder]#
> openssl ocsp
> >> -issuer simpleca.crt
> >> > -VAfile simplecabinding2.crt -serial
> 0x008486394c03e1f5d9 -req_text
> >> > -host 192.168.0.28:80
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash:
> 0C16107310427EA4ADB3C6436915CE44A15FFE55
> >> > Issuer Key Hash:
> E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
> >> > Serial Number: 8486394C03E1F5D9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
> >> > Response verify OK
> >> > 0x008486394c03e1f5d9: revoked
> >> > This Update: Jun 28 19:12:48 2019 GMT
> >> > Reason: unspecified
> >> > Revocation Time: May 26 12:30:44 2019 GMT
> >> >
> >> > So it seems like the EJBCA OCSP Responder is either not
> > processing the
> >> > large CRL properly, or maybe the import didn't import
> the large
> >> CRL into
> >> > the OCSP Responder correctly.
> >> >
> >> > Can anyone tell me how do I diagnose this problem?
> >> >
> >> > Thanks,
> >> > Jim
> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > Ejbca-develop mailing list
> >> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> >
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: Tomas G. <to...@pr...> - 2019-07-18 15:37:59
|
What command did you use to create the database?
On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
> Hi,
>This is really getting frustrating!!!
>[root@ejbca sql-scripts]# mysql -u root -p'xxxxxx' ejbca <
>create-index-ejbca.sql
>mysql: [Warning] Using a password on the command line interface can be
>insecure.
>ERROR 1071 (42000) at line 23: Specified key was too long; max key
>length is 1000 bytes
>
>Jim
>
>
>
>
>
>On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya...
><oh...@ya...> wrote:
>
> Hi,
>The "machine" is a virtual machine under VirtualBox on a machine I have
>that has 64GB physical RAM. Drives are all 7200 RPM drives.
>I will restore the snapshot, then increase the memory and # of
>processors and try to run the create index again.
>Jim
>
>On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
> Hi,
>
>Just run create index script.
>
>Create index should run pretty fast even with millions of records in
>the database. I suspect that your database is configured very slow,
>perhaps running everything on disl, without RAM, using slow disks?
>
>A standard installation of mysql/mariadb on a centos machine should run
>faster. Then you have pretty low amount of ram for running everything
>on one box, handing millions of records.
>
>Regards,
>Tomas
>
>On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
> Hi,
>I had taken a snapshot of the machine before I tried to do the index
>earlier, so I restored that snapshot.
>Then, I tried running the "optimize-ejbca-mysql.sql" and I got a
>different/new error:
>ERROR 1091 (42000) at line 27: Can't DROP 'certificatedata_idx5'; check
>that column/key exists
>
>Jim
>
>
>
>
>
>
>On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
><oh...@ya...> wrote:
>
> Hi,
>Oh oh, I just got this from the mysql execution:
>ERROR 1071 (42000) at line 23: Specified key was too long; max key
>length is 1000 bytes
>
>Jim
>
>
>
>
>
>
>On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
><oh...@ya...> wrote:
>
> Hi,
>FYI, I am running the create-index-ejbca.sql script now, but that seems
>to be taking FOREVER?
>I had stopped the earlier import, and then deleted all the rows for
>that CA DN in CertificateData table, and then bounced the machine, then
>I ran the create-index-ejbca.sql.
>Why is it taking so long? Should I have done something in addition to
>deleting the rows? I am guessing that the reason the sql script is
>taking so long is that it's somehow still trying to index the rows were
>deleted or something like that?
>Thanks,Jim
>
>
>
>On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
><oh...@ya...> wrote:
>
> Hi,
>
>I notice that there is:
>- create-index-ejbca.sql
>- optimize-ejbca-mysql.sql
>Should I run BOTH of them, or just the "create-index-ejbca.sql "?
>Thanks,Jim
>
>
>On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>Yes, i could see from your output from top that mysql takes all CPU.
>This indicates that you don't have any indexes in your database, which
>makes it horribly slow as numbers grow.
>
>Apply database indexes from doc/sql-scripts.
>
>Regards,
>Tomas
>
>
>On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
> Hi,
>So far, processed only 185706 entries, and it looks like the processing
>is going to a crawl (really slow now).
>Which kind of makes me wonder if maybe part of reason for the slowness
>is that it is having to search the certificate database for the serial
>number, and now, as there are more dummy certs in the database, that
>search is taking longer and longer?
>Jim
>
>On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>This is what top is showing now (the import is still running):
>top - 13:56:19 up 1 day, 1:09, 3 users, load average: 1.18, 1.11,
>1.07
>Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
>%Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0 hi, 0.3
>si, 0.0 st
>KiB Mem : 4045880 total, 123192 free, 3053072 used, 869616
>buff/cache
>KiB Swap: 3905532 total, 3753980 free, 151552 used. 688136 avail
>Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>COMMAND
> 1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2 1227:41
>mysqld
> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2 139:05.67
>java
> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3 176:26.68
>java
>22466 root 20 0 0 0 0 S 0.7 0.0 0:00.73
>kworker/u4:1
>
>That is running on 2-CPU machine with 4GB RAM.
>Jim
>
>
>
>
>On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
> Yes, like top.
>
>
>On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
> Hi Tomas,
>I'm sorry that I missed the message below, but I am not clear what you
>are asking me to check/monitor? Do you mean like running something
>like "top" command?
>Thanks,Jim
>
>On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
> To verify that the changes actually happen...
>
>Do you monitor CPU usage to see what processes use CPU and memory?
>
>
>On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
> Hi,
>The import has been running for about an hour now and has imported
>17836 entries.
>Per what I had said previously, this CRL file has 1002255entries, that
>means it would take about 56+ hours to import this CRL.
>So, this import processing is still way too slow.
>I don't know what is involved, or even how much it would improve
>things, but I think that the next thing to try is to eliminated both
>the check for the serial number in the DB AND the dummy cert creation.
>Jim
>
>On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>Ok, I rebuilt the JAR using your Java class, and it is running now, and
>creating rows in the CertificateData, but it still SEEMS kind of slow.
>
>
>I am running now and will try to get a measurement of revoked
>entries/time and will post back.
>Jim
>
>
>
>On Tuesday, July 16, 2019, 5:25:32 PM UTC, oh...@ya...
><oh...@ya...> wrote:
>
> Hi,
>So I just tried what I said (emptying the "if") and you are right, it
>doesn't create the rows in the CeritificateData table :(...
>Ok, I will try your version...
>Jim
>
>On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>
>Hi,
>
>I don't think so, no quite actually. In order to use the API to
>populate
>the database you need the certificate object it seems. Something like
>the attached file should work, it still creates a dummy certificate,
>but
>uses a static hard coded key, so no key generation.
>
>The file assumes you are on 6.15.x.
>
>Regards,
>Tomas
>
>On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
>> For testing (only...), I am thinking that if I modify this If clause
>in
>> CaImportCrlCommand.java:
>>
>> if (adaptive)
>> {
>> Date time = new Date();
>> KeyPair key_pair = KeyTools.genKeys("2048", "RSA");
>>
>> SubjectPublicKeyInfo pkinfo =
>> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
>> X500Name dnName = new X500Name("CN=Dummy Missing in
>> Imported CRL, serialNumber=" + serialHex);
>> Date notAfter = new Date(time.getTime() +
>315360000000L);
>> X509v3CertificateBuilder certbuilder = new
>> X509v3CertificateBuilder(X500Name.getInstance(cacert
>> .getSubjectX500Principal().getEncoded()), serialNr,
>> time, notAfter, dnName, pkinfo);
>> ContentSigner signer = new BufferingContentSigner(new
>>
>JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
>> .getPrivate()), 20480);
>> X509CertificateHolder certHolder =
>certbuilder.build(signer);
>> X509Certificate certificate =
>>
>(X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
>> X509Certificate.class);
>>
>> String fingerprint =
>> CertTools.getFingerprintAsString(certificate);
>>
>> EndEntityInformation missingUserEndEntityInformation =
>>
>((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
>> missing_user_name);
>> if (missingUserEndEntityInformation == null)
>> {
>> log.debug("Loading/updating user " +
>missing_user_name);
>>
>> EndEntityInformation userdataNew = new
>> EndEntityInformation(missing_user_name,
>> CertTools.getSubjectDN(certificate), cainfo.getCAId(), null, null,
>10,
>> new EndEntityType(EndEntityTypes.ENDUSER), 1, 1, null, null, 1, 0,
>null);
>>
>> userdataNew.setPassword("foo123");
>>
>>
>((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
>> userdataNew, false);
>>
>> log.info("User '" + missing_user_name + "' has been
>> added.");
>>
>>
>((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
>> missing_user_name, 50);
>>
>> log.info("User '" + missing_user_name + "' has been
>> updated.");
>> }
>>
>>
>((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>>
>> EJBTools.wrap(certificate), missing_user_name,
>> fingerprint, 20, 1, 1, 0, null, new Date()
>>
>> .getTime());
>> log.info("Dummy certificate '" + serialHex + "' has
>been
>> stored.");
>> }
>>
>> to (for example):
>>
>> if (adaptive)
>> {
>>
>> log.info("*** SKIPPING *** Dummy certificate '" +
>> serialHex + "' has been stored.");
>> }
>>
>>
>> Would THAT work??
>>
>> Thanks,
>> Jim
>>
>>
>>
>>
>> On Tuesday, July 16, 2019, 10:59:31 AM UTC, ohaya--- via
>Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> Hi Tomas (or anyone else),
>>
>> Sorry - I was typing my last msg on phone, and it wasn't as detailed
>or
>> clear as I wanted.
>>
>> Basically what I was saying is I was thinking that, as a quick sanity
>> test, if there was one or two places where the code in
>> the CaImportCRLCommand.java was maybe making a method call to create
>the
>> dummy cert and put it into the database, then maybe I could decompile
>> and modify the CaImportCRLCommand.java, and comment out those method
>> calls, then, re-do my large CRL import test and time it?
>>
>> If so, can you let me know specifically where those calls are, and if
>I
>> just comment them out, would that work?
>>
>> Ironically, that (not doing the dummy cert creation and writing) was
>> what I *thought* the "LENIENT" option was doing).
>>
>> Thanks,
>> Jim
>>
>> P. S. I have time today to try this if I figure out what needs to be
>done.
>>
>>
>>
>> On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya--- via Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>>
>> Yes, we are definitely interested/willing to test patches. I
>actually
>> was wondering if it would be a quick test to mod the code somewhere
>to
>> avoid a call that creates the dummy cert, to get an idea if that
>would
>> reduce the import time enough??
>>
>>
>> Sent from Yahoo Mail on Android
>>
><https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>>
>> On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
>> <to...@pr...> wrote:
>> Most large users, some very very large, typically uses on line
>> updates directly from the CA. Many CAs today don't accept only
>> importing CRLs, as they need to keep track of issued certificates
>as
>> well not only revoked (can't use "non existing is good").
>> Having said thst, it's interesting to be able to import large
>CRLs,
>> so improving it is of interest.
>>
>> Importing CRLs would actually not have to generate a dummy cert at
>> all, since a certificate object is not needed on the responder,
>> which should make it very fast.
>>
>> If you are willing to test patches, we can work with you on
>improving.
>>
>> Regards,
>> Tomas
>>
>>
>> On July 15, 2019 5:26:08 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...> wrote:
>>
>> Hi,
>>
>> I am curious whether you have other users that import large
>CRLs
>> like this?
>>
>> Are we the only ones who would encounter these really long
>> import times, because the other users who might have large
>CRLs
>> use the EJBCA CA also, so they don't run into having the dummy
>> certs being created for every entry in the CRL?
>>
>> Jim
>>
>> On Monday, July 15, 2019, 1:42:57 PM UTC, Tomas Gustavsson
>> <to...@pr...> wrote:
>>
>>
>>
>> I see...
>>
>> I can see that a new key pair is being generated for each
>dummy
>> cert.
>>
>> In CaImportCRLCommand.java:
>>
>> final KeyPair key_pair = KeyTools.genKeys("2048",
>> AlgorithmConstants.KEYALGORITHM_RSA);
>>
>> Chaning that to 1024 or 512 would be an easy improvement. A
>bit more
>> complex, but even larger speedup would be to use a "hard
>coded"
>> dummy
>> key, i.e. same key for all dummy certs.
>>
>> You can experiment by changing 2048 to 512 (not secure but it
>> doesn't
>> have to be for this dummy cert).
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-11 14:19, ohaya--- via Ejbca-develop wrote:
>> > FYI, the is running, but it is REALLY slow. The CRL file
>has
>> 1002255
>> > entries in it, and the import is only up to 8899 now, after
>> about 40
>> > minutes... which would take about 75 hours!!
>> >
>> > In our production systems, we are required to update some
>CRLs
>> every
>> > hour, so that would be a problem.
>> >
>> > What can I do to speed that up??
>> >
>> > Thanks,
>> > Jim
>> >
>> >
>> > On Thursday, July 11, 2019, 11:09:33 AM UTC, ohaya--- via
>> Ejbca-develop
>> > <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>> >
>> >
>> > AHHH... Ok, it is running now, and from checking the
>> CertificateData in
>> > the Mysql database, I can see that it is now populating the
>> > CertificateData!!
>> >
>> > Thanks!
>> >
>> > Jim
>> >
>> >
>> >
>> > On Thursday, July 11, 2019, 11:07:22 AM UTC, ohaya--- via
>> Ejbca-develop
>> > <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>> >
>> >
>> > I wasn't sure if we should use ADAPTIVE, since we are not
>> using the CA part.
>> >
>> >
>> > I will try with "-o ADAPTIVE" now.
>> >
>> >
>> >
>> >
>> > On Thursday, July 11, 2019, 7:42:14 AM UTC, Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>> wrote:
>> >
>> >
>> >
>> > It should be ADAPTIVE.
>> >
>> > The documentation says:
>> > bin/ejbca.sh ca importcrl --help
>> > ---
>> > ADAPTIVE means that missing certficates will be replaced by
>dummy
>> > certificates to cater for proper CRLs for missing
>certificates.
>> > ---
>> >
>> > Regards,
>> > Tomas
>> >
>> > On 2019-07-11 08:05, ohaya--- via Ejbca-develop wrote:
>> >> Hi,
>> >>
>> >> I've been looking at the code in CaImportCRLCommand, and I
>am
>> wondering
>> >> if, when the command is run in LENIENT mode and it doesn't
>> find the cert
>> >> in the database (and outputs the "missing in database"
>error
>> message),
>> >> what is supposed to happen?
>> >>
>> >> Mind you, the only error messages I am seeing are the
>"missing in
>> >> database" error messages. I don't see any of the other
>error
>> messages
>> >> that are in/from CaImportCRLCommand.java.
>> >>
>> >> I am wondering if, when the command is run with "-o
>LENIENT",
>> and when
>> >> it outputs the "missing in database" error message, then
>the
>> ejbca.sh is
>> >> basically not doing anything (and not storing anything into
>the
>> >> CertificateData table)?
>> >>
>> >> From the lack of any other error messages (vs. what I see
>in the
>> >> CaImportCRLCommand.java code), that is what it looks like
>is
>> happening.
>> >>
>> >> So given I am using EJBCA for OCSP responder only, is
>running the
>> >> ejbca.sh in the "-o LENIENT" mode correct?
>> >>
>> >> Jim
>> >>
>> >>
>> >>
>> >> On Thursday, July 11, 2019, 5:43:37 AM UTC, ohaya--- via
>> Ejbca-develop
>> >> <ejb...@li...
>> <mailto:ejb...@li...>
>> > <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I've been doing a bunch of testing with trying to import
>the
>> large CRL,
>> >> and wanted to summarize:
>> >>
>> >> - The ejbca.sh runs fine - it completes without outputting
>an
>> errors,
>> >> except for outputting a bunch of the "cert xxxx missing in
>> database"
>> >> messages.
>> >>
>> >> - When the import completes, and I check the Mysql
>database,
>> the crlData
>> >> (the full PEM of the CRL) is in the database, but there is
>no
>> rows in
>> >> the CertificateData table for that CA at all.
>> >>
>> >> - I even figured out how to get the ejbca.sh to log using
>> >> log4j.properties, and it output a bunch of logging, mainly
>> the missing
>> >> in database errors, but other than that I didn't see any
>> obvious errors.
>> >>
>> >>
>> >> So, bottom line is that even though the ejbca.sh seems to
>run
>> without
>> >> errors (other than the missing in database errors), the
>import is
>> >> causing the CRL PEM to be stored in the database, but the
>> >> CertificateData table is not being populated at all for
>that CA.
>> >>
>> >>
>> >> Jim
>> >>
>> >>
>> >>
>> >> On Wednesday, July 10, 2019, 7:40:16 PM UTC, ohaya--- via
>> Ejbca-develop
>> >> <ejb...@li...
>> <mailto:ejb...@li...>
>> > <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> Ok, I found Tomas' original message... it was in my Spam
>> folder....
>> >>
>> >> I also tried re-running the "./ejbca.sh ca importcrl"
>command
>> a bunch of
>> >> ways. I think it is running without outputting any errors,
>> but still,
>> >> no rows are appearing with that issuerDN at all.
>> >>
>> >>
>> >>
>> >>
>> >> On Tuesday, July 9, 2019, 10:45:01 AM UTC, Tomas Gustavsson
>> >> <to...@pr... <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> OCSP responses does not have anything to do with the CRL
>> itself. Once
>> >> the CRL has been imported each certificate status is in its
>> own record
>> >> in the CertificateData table. This is where status is read
>> when reponses
>> >> are created.
>> >> If there are issues, it is in that case related to the
>import
>> of the CRL.
>> >>
>> >> Verify that certificate records are created in
>> CertificateData when
>> >> importing the CRL.
>> >> Also, a debug log when making OCSP request will tell exact
>> details of
>> >> what the responder finds in the database or not.
>> >>
>> >> Regards,
>> >> Tomas
>> >>
>> >>
>> >> On July 8, 2019 4:59:06 PM GMT+02:00, ohaya--- via
>Ejbca-develop
>> >> <ejb...@li...
>> <mailto:ejb...@li...>
>> > <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>> >>
>> >> [Re-sending, as this (handling of large CRLs) is still a
>> problem and
>> >> is still not working.]
>> >>
>> >>
>> >>
>> >> On Monday, July 1, 2019, 12:47:23 PM UTC,
><oh...@ya...
>> <mailto:oh...@ya...>
>> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I think that I've found how to set/change the
>> max_allowed_packet on
>> >> the client (i.e., EJBCA) side, by editing the Mysql URL
>in the
>> >> standalone/configuration/standalone.xml to:
>> >>
>> >>
>> >
>>
><connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
>> >>
>> >> However, when I test the OCSP Responder with a serial
>> number (and CA
>> >> cert) from the CA with the large (>71MB) CRL file, I am
>still
>> >> getting "unknown" and EJBCA OCSP Responder is still NOT
>> FINDING the
>> >> cert serial number.
>> >>
>> >> NOTE, again, that I can test with certs (and CA cert)
>that
>> have
>> >> smaller CRL files, and those tests work.
>> >>
>> >> So it is only when testing with the large CRL file that
>is
>> failing
>> >> with the OCSP Responder.
>> >>
>> >> This seems like a potential BUG for EJBCA OCSP
>Responder?
>> >>
>> >> Please advise.
>> >>
>> >> Thanks,
>> >> Jim
>> >> On Saturday, June 29, 2019, 3:36:18 PM EDT,
>> <oh...@ya... <mailto:oh...@ya...>
>> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> BTW, just to check, I have actually gone through the
>> process of
>> >> getting the crlData from the MySQL using mysql client (I
>> had to add
>> >> the --max_allowed-packet=100M to the mysql command) and
>I
>> was able
>> >> to actually get the PEM from the Mysql database that
>way.
>> >>
>> >> So, I KNOW that the entire CRL is in the Mysql database,
>> and yet
>> >> when I run an "openssl ocsp" test against the EJBCA OCSP
>> Responder,
>> >> it says it cannot find the revocation.
>> >>
>> >> OCSP Request Data:
>> >> Version: 1 (0x0)
>> >> Requestor List:
>> >> Certificate ID:
>> >> Hash Algorithm: sha1
>> >> Issuer Name Hash:
>37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
>> >> Issuer Key Hash:
>3929A80864E28F83AA816147119133F85A37A6E4
>> >> Serial Number: 2FFFF4
>> >> Request Extensions:
>> >> OCSP Nonce:
>> >> 04103054A60A55E389B239A55228033D8F62
>> >> Response verify OK
>> >> 0x2FFFF4: unknown
>> >> This Update: Jun 29 19:31:02 2019 GMT
>> >>
>> >>
>> >> In the server log:
>> >>
>> >> 2019-06-29 15:31:02,150 DEBUG
>> >>
>>
>[org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
>> >> (default task-1) Unable to find revocation information
>for
>> >> certificate with serial '2ffff4' from issuer
>'CN=XXX....'
>> >>
>> >>
>> >> Honestly, I am no 100% sure that the problem is that
>> >> max_allowe_packet is too small on the mysql client side,
>> but that is
>> >> my best guess.
>> >>
>> >> What I AM sure is that the CRL is in mysql, but when I
>do
>> a test
>> >> with OCSP Responder, it cannot find the cert serial
>number
>> in the CRL.
>> >>
>> >> Jim
>> >>
>> >>
>> >> On Saturday, June 29, 2019, 4:52:20 AM EDT,
>> <oh...@ya... <mailto:oh...@ya...>
>> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
>> >>
>> >>
>> >> Hi Tomas,
>> >>
>> >> I already had the max_allowed_packet configured on the
>Mysql
>> >> server-side (in /etc/my.cnf).
>> >>
>> >> So as I said, when testing with mysql (client) I had to
>> include
>> >> "--max_allowed_packet=100M" on the mysql (client)
>command
>> line, e.g.:
>> >>
>> >> mysql -u root -p --max_allowed_packet=100M
>> >>
>> >> If I did NOT include the "--max_allowed_packet=100M" on
>> that command
>> >> line, then when I did a "select" on the field base64Crl,
>I
>> would get
>> >> an error.
>> >>
>> >> I suspect that EJBCA is acting as a Mysql client, so, in
>> order to
>> >> work with such large CRLs, there needs to be some way to
>> set the
>> >> equivalent of the "--max_allowed_packet=100M" on the
>EJBCA
>> side.
>> >>
>> >> How can I do that?
>> >>
>> >> If that can't be done, then I think that when EJBCA
>tries to
>> >> retrieve the CRL from the Mysql, it (EJBCA) will see the
>> same error.
>> >>
>> >> Thanks,
>> >> Jim
>> >>
>> >> On Saturday, June 29, 2019, 3:25:47 AM EDT, Tomas
>Gustavsson
>> >> <to...@pr... <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
>> >>
>> >>
>> >>
>> >> You configure max allowed packet size in you MySQL
>> configuration on the
>> >> MySQL server.
>> >>
>> >> /Tomas
>> >>
>> >> On 2019-06-29 02:19, o haya via Ejbca-develop wrote:
>> >> > I am not 100% sure, but I think I know what might be
>> causing this
>> >> problem.
>> >> >
>> >> > I did a test, where I just tried to log into mysql and
>> do the
>> >> following
>> >> > select:
>> >> >
>> >> > select base64Crl INTO OUTFILE
>> '/var/lib/mysql-files/xx.out' from
>> >> CRLData
>> >> > where cRLNumber=3167;
>> >> >
>> >> > and I got an error saying that the result was more
>> >> than max_allowed_packet.
>> >> >
>> >> > So then I did the same thing, but for the mysql
>command
>> I used:
>> >> >
>> >> > mysql -u root -p --max_allowed_packet=100M
>> >> >
>> >> > and then when I did that same select, I was able to
>> output the entire
>> >> > CRL as a PEM.
>> >> >
>> >> > So, I am theorizing that that is what is happening
>with
>> EJBCA, i.e.,
>> >> > when I run the "openssl ocsp" command, EJBCA is
>querying the
>> > base64CRL
>> >> > but the result is too large and so that is causing the
>> query to fail.
>> >> >
>> >> > So the question now is: How can I cause EJBCA OCSP
>> Responder to run
>> >> > with a larger max_allowed_packet so that the query
>> succeeds (and then
>> >> > hopefully EJBCA OCSP Responder will be able to check
>the
>> revocation
>> >> > properly)?
>> >> >
>> >> > Does anyone know how that can be done?
>> >> >
>> >> > Thanks,
>> >> > Jim
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Friday, June 28, 2019, 07:16:39 PM UTC, o haya
>> > <oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> >> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>> wrote:
>> >> >
>> >> >
>> >> > Hi,
>> >> >
>> >> > As discussed in an earlier thread, it looks like I was
>> able to
>> >> import a
>> >> > large (> 71MB) CRL file into EJBCA (as an OCSP
>> Responder), so now,
>> >> I am
>> >> > doing testing, using "openssl ocsp" to send requests
>> into the
>> >> EJBCA OCSP
>> >> > Responder:
>> >> >
>> >> > openssl ocsp -issuer <CA_CERT> -VAfile <SIGNING_CERT>
>> >> -req_text -serial
>> >> > 0x<CERT_SERIAL> -host <HOST:PORT>
>> >> >
>> >> > But it looks like no matter what serial number I use,
>> whether the
>> >> serial
>> >> > number is in the CRL or not, returns a response like:
>> >> >
>> >> > OCSP Request Data:
>> >> > Version: 1 (0x0)
>> >> > Requestor List:
>> >> > Certificate ID:
>> >> > Hash Algorithm: sha1
>> >> > Issuer Name Hash: 37...00
>> >> > Issuer Key Hash: 39...E4
>> >> > Serial Number: 2FFFF9
>> >> > Request Extensions:
>> >> > OCSP Nonce:
>> >> > 041078B1311984A110B5E19C8DC4895F485B
>> >> > Response verify OK
>> >> > 0x2FFFF9: unknown
>> >> > This Update: Jun 28 18:56:20 2019 GMT
>> >> >
>> >> > i.e., it looks like the EJBCA OCSP Responder is not
>able
>> to actually
>> >> > find any revoked certs in the CRL?
>> >> >
>> >> >
>> >> > FYI, I *KNOW* that the EJBCA OCSP Responder is
>functioning
>> >> properly, in
>> >> > general, because I can do the same test against a
>> smaller CRL in the
>> >> > same EJBCA OCSP Responder, for CA simpleca, and I can
>> get an actual
>> >> > "Revoked" response:
>> >> >
>> >> > [root@ejbca <mailto:root@ejbca> <mailto:root@ejbca
>> <mailto:root@ejbca>> <mailto:root@ejbca <mailto:root@ejbca>
>> > <mailto:root@ejbca <mailto:root@ejbca>>> testocspresponder]#
>> openssl ocsp
>> >> -issuer simpleca.crt
>> >> > -VAfile simplecabinding2.crt -serial
>> 0x008486394c03e1f5d9 -req_text
>> >> > -host 192.168.0.28:80
>> >> > OCSP Request Data:
>> >> > Version: 1 (0x0)
>> >> > Requestor List:
>> >> > Certificate ID:
>> >> > Hash Algorithm: sha1
>> >> > Issuer Name Hash:
>> 0C16107310427EA4ADB3C6436915CE44A15FFE55
>> >> > Issuer Key Hash:
>> E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
>> >> > Serial Number: 8486394C03E1F5D9
>> >> > Request Extensions:
>> >> > OCSP Nonce:
>> >> > 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
>> >> > Response verify OK
>> >> > 0x008486394c03e1f5d9: revoked
>> >> > This Update: Jun 28 19:12:48 2019 GMT
>> >> > Reason: unspecified
>> >> > Revocation Time: May 26 12:30:44 2019 GMT
>> >> >
>> >> > So it seems like the EJBCA OCSP Responder is either
>not
>> > processing the
>> >> > large CRL properly, or maybe the import didn't import
>> the large
>> >> CRL into
>> >> > the OCSP Responder correctly.
>> >> >
>> >> > Can anyone tell me how do I diagnose this problem?
>> >> >
>> >> > Thanks,
>> >> > Jim
>> >>
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Ejbca-develop mailing list
>> >> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>>
>> >> >
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >> >
>> >>
>> >>
>> >> _______________________________________________
>> >> Ejbca-develop mailing list
>> >> Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>>
>> >>
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >>
>> >> _______________________________________________
>> >> Ejbca-develop mailing list
>> >> Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>>
>> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >> _______________________________________________
>> >> Ejbca-develop mailing list
>> >> Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>>
>> >
>> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >>
>> >>
>> >> _______________________________________________
>> >> Ejbca-develop mailing list
>> >> Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >>
>> >
>> >
>> > _______________________________________________
>> > Ejbca-develop mailing list
>> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> > _______________________________________________
>> > Ejbca-develop mailing list
>> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> > _______________________________________________
>> > Ejbca-develop mailing list
>> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >
>> >
>> > _______________________________________________
>> > Ejbca-develop mailing list
>> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >
>>
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>>
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>>
>_______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
>
|
|
From: <oh...@ya...> - 2019-07-18 15:32:51
|
Hi,
This is really getting frustrating!!!
[root@ejbca sql-scripts]# mysql -u root -p'xxxxxx' ejbca < create-index-ejbca.sql
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1071 (42000) at line 23: Specified key was too long; max key length is 1000 bytes
Jim
On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
The "machine" is a virtual machine under VirtualBox on a machine I have that has 64GB physical RAM. Drives are all 7200 RPM drives.
I will restore the snapshot, then increase the memory and # of processors and try to run the create index again.
Jim
On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
Just run create index script.
Create index should run pretty fast even with millions of records in the database. I suspect that your database is configured very slow, perhaps running everything on disl, without RAM, using slow disks?
A standard installation of mysql/mariadb on a centos machine should run faster. Then you have pretty low amount of ram for running everything on one box, handing millions of records.
Regards,
Tomas
On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I had taken a snapshot of the machine before I tried to do the index earlier, so I restored that snapshot.
Then, I tried running the "optimize-ejbca-mysql.sql" and I got a different/new error:
ERROR 1091 (42000) at line 27: Can't DROP 'certificatedata_idx5'; check that column/key exists
Jim
On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
Oh oh, I just got this from the mysql execution:
ERROR 1071 (42000) at line 23: Specified key was too long; max key length is 1000 bytes
Jim
On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
FYI, I am running the create-index-ejbca.sql script now, but that seems to be taking FOREVER?
I had stopped the earlier import, and then deleted all the rows for that CA DN in CertificateData table, and then bounced the machine, then I ran the create-index-ejbca.sql.
Why is it taking so long? Should I have done something in addition to deleting the rows? I am guessing that the reason the sql script is taking so long is that it's somehow still trying to index the rows were deleted or something like that?
Thanks,Jim
On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
I notice that there is:
- create-index-ejbca.sql
- optimize-ejbca-mysql.sql
Should I run BOTH of them, or just the "create-index-ejbca.sql "?
Thanks,Jim
On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, i could see from your output from top that mysql takes all CPU. This indicates that you don't have any indexes in your database, which makes it horribly slow as numbers grow.
Apply database indexes from doc/sql-scripts.
Regards,
Tomas
On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
So far, processed only 185706 entries, and it looks like the processing is going to a crawl (really slow now).
Which kind of makes me wonder if maybe part of reason for the slowness is that it is having to search the certificate database for the serial number, and now, as there are more dummy certs in the database, that search is taking longer and longer?
Jim
On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
This is what top is showing now (the import is still running):
top - 13:56:19 up 1 day, 1:09, 3 users, load average: 1.18, 1.11, 1.07
Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
%Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 4045880 total, 123192 free, 3053072 used, 869616 buff/cache
KiB Swap: 3905532 total, 3753980 free, 151552 used. 688136 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2 1227:41 mysqld
3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2 139:05.67 java
3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3 176:26.68 java
22466 root 20 0 0 0 0 S 0.7 0.0 0:00.73 kworker/u4:1
That is running on 2-CPU machine with 4GB RAM.
Jim
On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, like top.
On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi Tomas,
I'm sorry that I missed the message below, but I am not clear what you are asking me to check/monitor? Do you mean like running something like "top" command?
Thanks,Jim
On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
To verify that the changes actually happen...
Do you monitor CPU usage to see what processes use CPU and memory?
On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
The import has been running for about an hour now and has imported 17836 entries.
Per what I had said previously, this CRL file has 1002255entries, that means it would take about 56+ hours to import this CRL.
So, this import processing is still way too slow.
I don't know what is involved, or even how much it would improve things, but I think that the next thing to try is to eliminated both the check for the serial number in the DB AND the dummy cert creation.
Jim
On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
Ok, I rebuilt the JAR using your Java class, and it is running now, and creating rows in the CertificateData, but it still SEEMS kind of slow.
I am running now and will try to get a measurement of revoked entries/time and will post back.
Jim
On Tuesday, July 16, 2019, 5:25:32 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
So I just tried what I said (emptying the "if") and you are right, it doesn't create the rows in the CeritificateData table :(...
Ok, I will try your version...
Jim
On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
I don't think so, no quite actually. In order to use the API to populate
the database you need the certificate object it seems. Something like
the attached file should work, it still creates a dummy certificate, but
uses a static hard coded key, so no key generation.
The file assumes you are on 6.15.x.
Regards,
Tomas
On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
> For testing (only...), I am thinking that if I modify this If clause in
> CaImportCrlCommand.java:
>
> if (adaptive)
> {
> Date time = new Date();
> KeyPair key_pair = KeyTools.genKeys("2048", "RSA");
>
> SubjectPublicKeyInfo pkinfo =
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
> X500Name dnName = new X500Name("CN=Dummy Missing in
> Imported CRL, serialNumber=" + serialHex);
> Date notAfter = new Date(time.getTime() + 315360000000L);
> X509v3CertificateBuilder certbuilder = new
> X509v3CertificateBuilder(X500Name.getInstance(cacert
> .getSubjectX500Principal().getEncoded()), serialNr,
> time, notAfter, dnName, pkinfo);
> ContentSigner signer = new BufferingContentSigner(new
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
> .getPrivate()), 20480);
> X509CertificateHolder certHolder = certbuilder.build(signer);
> X509Certificate certificate =
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
> X509Certificate.class);
>
> String fingerprint =
> CertTools.getFingerprintAsString(certificate);
>
> EndEntityInformation missingUserEndEntityInformation =
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
> missing_user_name);
> if (missingUserEndEntityInformation == null)
> {
> log.debug("Loading/updating user " + missing_user_name);
>
> EndEntityInformation userdataNew = new
> EndEntityInformation(missing_user_name,
> CertTools.getSubjectDN(certificate), cainfo.getCAId(), null, null, 10,
> new EndEntityType(EndEntityTypes.ENDUSER), 1, 1, null, null, 1, 0, null);
>
> userdataNew.setPassword("foo123");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
> userdataNew, false);
>
> log.info("User '" + missing_user_name + "' has been
> added.");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
> missing_user_name, 50);
>
> log.info("User '" + missing_user_name + "' has been
> updated.");
> }
>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>
> EJBTools.wrap(certificate), missing_user_name,
> fingerprint, 20, 1, 1, 0, null, new Date()
>
> .getTime());
> log.info("Dummy certificate '" + serialHex + "' has been
> stored.");
> }
>
> to (for example):
>
> if (adaptive)
> {
>
> log.info("*** SKIPPING *** Dummy certificate '" +
> serialHex + "' has been stored.");
> }
>
>
> Would THAT work??
>
> Thanks,
> Jim
>
>
>
>
> On Tuesday, July 16, 2019, 10:59:31 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi Tomas (or anyone else),
>
> Sorry - I was typing my last msg on phone, and it wasn't as detailed or
> clear as I wanted.
>
> Basically what I was saying is I was thinking that, as a quick sanity
> test, if there was one or two places where the code in
> the CaImportCRLCommand.java was maybe making a method call to create the
> dummy cert and put it into the database, then maybe I could decompile
> and modify the CaImportCRLCommand.java, and comment out those method
> calls, then, re-do my large CRL import test and time it?
>
> If so, can you let me know specifically where those calls are, and if I
> just comment them out, would that work?
>
> Ironically, that (not doing the dummy cert creation and writing) was
> what I *thought* the "LENIENT" option was doing).
>
> Thanks,
> Jim
>
> P. S. I have time today to try this if I figure out what needs to be done.
>
>
>
> On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
>
> Yes, we are definitely interested/willing to test patches. I actually
> was wondering if it would be a quick test to mod the code somewhere to
> avoid a call that creates the dummy cert, to get an idea if that would
> reduce the import time enough??
>
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>
> On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
> <to...@pr...> wrote:
> Most large users, some very very large, typically uses on line
> updates directly from the CA. Many CAs today don't accept only
> importing CRLs, as they need to keep track of issued certificates as
> well not only revoked (can't use "non existing is good").
> Having said thst, it's interesting to be able to import large CRLs,
> so improving it is of interest.
>
> Importing CRLs would actually not have to generate a dummy cert at
> all, since a certificate object is not needed on the responder,
> which should make it very fast.
>
> If you are willing to test patches, we can work with you on improving.
>
> Regards,
> Tomas
>
>
> On July 15, 2019 5:26:08 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> I am curious whether you have other users that import large CRLs
> like this?
>
> Are we the only ones who would encounter these really long
> import times, because the other users who might have large CRLs
> use the EJBCA CA also, so they don't run into having the dummy
> certs being created for every entry in the CRL?
>
> Jim
>
> On Monday, July 15, 2019, 1:42:57 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> I see...
>
> I can see that a new key pair is being generated for each dummy
> cert.
>
> In CaImportCRLCommand.java:
>
> final KeyPair key_pair = KeyTools.genKeys("2048",
> AlgorithmConstants.KEYALGORITHM_RSA);
>
> Chaning that to 1024 or 512 would be an easy improvement. A bit more
> complex, but even larger speedup would be to use a "hard coded"
> dummy
> key, i.e. same key for all dummy certs.
>
> You can experiment by changing 2048 to 512 (not secure but it
> doesn't
> have to be for this dummy cert).
>
> Regards,
> Tomas
>
> On 2019-07-11 14:19, ohaya--- via Ejbca-develop wrote:
> > FYI, the is running, but it is REALLY slow. The CRL file has
> 1002255
> > entries in it, and the import is only up to 8899 now, after
> about 40
> > minutes... which would take about 75 hours!!
> >
> > In our production systems, we are required to update some CRLs
> every
> > hour, so that would be a problem.
> >
> > What can I do to speed that up??
> >
> > Thanks,
> > Jim
> >
> >
> > On Thursday, July 11, 2019, 11:09:33 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > AHHH... Ok, it is running now, and from checking the
> CertificateData in
> > the Mysql database, I can see that it is now populating the
> > CertificateData!!
> >
> > Thanks!
> >
> > Jim
> >
> >
> >
> > On Thursday, July 11, 2019, 11:07:22 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > I wasn't sure if we should use ADAPTIVE, since we are not
> using the CA part.
> >
> >
> > I will try with "-o ADAPTIVE" now.
> >
> >
> >
> >
> > On Thursday, July 11, 2019, 7:42:14 AM UTC, Tomas Gustavsson
> > <to...@pr... <mailto:to...@pr...>> wrote:
> >
> >
> >
> > It should be ADAPTIVE.
> >
> > The documentation says:
> > bin/ejbca.sh ca importcrl --help
> > ---
> > ADAPTIVE means that missing certficates will be replaced by dummy
> > certificates to cater for proper CRLs for missing certificates.
> > ---
> >
> > Regards,
> > Tomas
> >
> > On 2019-07-11 08:05, ohaya--- via Ejbca-develop wrote:
> >> Hi,
> >>
> >> I've been looking at the code in CaImportCRLCommand, and I am
> wondering
> >> if, when the command is run in LENIENT mode and it doesn't
> find the cert
> >> in the database (and outputs the "missing in database" error
> message),
> >> what is supposed to happen?
> >>
> >> Mind you, the only error messages I am seeing are the "missing in
> >> database" error messages. I don't see any of the other error
> messages
> >> that are in/from CaImportCRLCommand.java.
> >>
> >> I am wondering if, when the command is run with "-o LENIENT",
> and when
> >> it outputs the "missing in database" error message, then the
> ejbca.sh is
> >> basically not doing anything (and not storing anything into the
> >> CertificateData table)?
> >>
> >> From the lack of any other error messages (vs. what I see in the
> >> CaImportCRLCommand.java code), that is what it looks like is
> happening.
> >>
> >> So given I am using EJBCA for OCSP responder only, is running the
> >> ejbca.sh in the "-o LENIENT" mode correct?
> >>
> >> Jim
> >>
> >>
> >>
> >> On Thursday, July 11, 2019, 5:43:37 AM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I've been doing a bunch of testing with trying to import the
> large CRL,
> >> and wanted to summarize:
> >>
> >> - The ejbca.sh runs fine - it completes without outputting an
> errors,
> >> except for outputting a bunch of the "cert xxxx missing in
> database"
> >> messages.
> >>
> >> - When the import completes, and I check the Mysql database,
> the crlData
> >> (the full PEM of the CRL) is in the database, but there is no
> rows in
> >> the CertificateData table for that CA at all.
> >>
> >> - I even figured out how to get the ejbca.sh to log using
> >> log4j.properties, and it output a bunch of logging, mainly
> the missing
> >> in database errors, but other than that I didn't see any
> obvious errors.
> >>
> >>
> >> So, bottom line is that even though the ejbca.sh seems to run
> without
> >> errors (other than the missing in database errors), the import is
> >> causing the CRL PEM to be stored in the database, but the
> >> CertificateData table is not being populated at all for that CA.
> >>
> >>
> >> Jim
> >>
> >>
> >>
> >> On Wednesday, July 10, 2019, 7:40:16 PM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> Ok, I found Tomas' original message... it was in my Spam
> folder....
> >>
> >> I also tried re-running the "./ejbca.sh ca importcrl" command
> a bunch of
> >> ways. I think it is running without outputting any errors,
> but still,
> >> no rows are appearing with that issuerDN at all.
> >>
> >>
> >>
> >>
> >> On Tuesday, July 9, 2019, 10:45:01 AM UTC, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> OCSP responses does not have anything to do with the CRL
> itself. Once
> >> the CRL has been imported each certificate status is in its
> own record
> >> in the CertificateData table. This is where status is read
> when reponses
> >> are created.
> >> If there are issues, it is in that case related to the import
> of the CRL.
> >>
> >> Verify that certificate records are created in
> CertificateData when
> >> importing the CRL.
> >> Also, a debug log when making OCSP request will tell exact
> details of
> >> what the responder finds in the database or not.
> >>
> >> Regards,
> >> Tomas
> >>
> >>
> >> On July 8, 2019 4:59:06 PM GMT+02:00, ohaya--- via Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >> [Re-sending, as this (handling of large CRLs) is still a
> problem and
> >> is still not working.]
> >>
> >>
> >>
> >> On Monday, July 1, 2019, 12:47:23 PM UTC, <oh...@ya...
> <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I think that I've found how to set/change the
> max_allowed_packet on
> >> the client (i.e., EJBCA) side, by editing the Mysql URL in the
> >> standalone/configuration/standalone.xml to:
> >>
> >>
> >
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
> >>
> >> However, when I test the OCSP Responder with a serial
> number (and CA
> >> cert) from the CA with the large (>71MB) CRL file, I am still
> >> getting "unknown" and EJBCA OCSP Responder is still NOT
> FINDING the
> >> cert serial number.
> >>
> >> NOTE, again, that I can test with certs (and CA cert) that
> have
> >> smaller CRL files, and those tests work.
> >>
> >> So it is only when testing with the large CRL file that is
> failing
> >> with the OCSP Responder.
> >>
> >> This seems like a potential BUG for EJBCA OCSP Responder?
> >>
> >> Please advise.
> >>
> >> Thanks,
> >> Jim
> >> On Saturday, June 29, 2019, 3:36:18 PM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> BTW, just to check, I have actually gone through the
> process of
> >> getting the crlData from the MySQL using mysql client (I
> had to add
> >> the --max_allowed-packet=100M to the mysql command) and I
> was able
> >> to actually get the PEM from the Mysql database that way.
> >>
> >> So, I KNOW that the entire CRL is in the Mysql database,
> and yet
> >> when I run an "openssl ocsp" test against the EJBCA OCSP
> Responder,
> >> it says it cannot find the revocation.
> >>
> >> OCSP Request Data:
> >> Version: 1 (0x0)
> >> Requestor List:
> >> Certificate ID:
> >> Hash Algorithm: sha1
> >> Issuer Name Hash: 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
> >> Issuer Key Hash: 3929A80864E28F83AA816147119133F85A37A6E4
> >> Serial Number: 2FFFF4
> >> Request Extensions:
> >> OCSP Nonce:
> >> 04103054A60A55E389B239A55228033D8F62
> >> Response verify OK
> >> 0x2FFFF4: unknown
> >> This Update: Jun 29 19:31:02 2019 GMT
> >>
> >>
> >> In the server log:
> >>
> >> 2019-06-29 15:31:02,150 DEBUG
> >>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
> >> (default task-1) Unable to find revocation information for
> >> certificate with serial '2ffff4' from issuer 'CN=XXX....'
> >>
> >>
> >> Honestly, I am no 100% sure that the problem is that
> >> max_allowe_packet is too small on the mysql client side,
> but that is
> >> my best guess.
> >>
> >> What I AM sure is that the CRL is in mysql, but when I do
> a test
> >> with OCSP Responder, it cannot find the cert serial number
> in the CRL.
> >>
> >> Jim
> >>
> >>
> >> On Saturday, June 29, 2019, 4:52:20 AM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi Tomas,
> >>
> >> I already had the max_allowed_packet configured on the Mysql
> >> server-side (in /etc/my.cnf).
> >>
> >> So as I said, when testing with mysql (client) I had to
> include
> >> "--max_allowed_packet=100M" on the mysql (client) command
> line, e.g.:
> >>
> >> mysql -u root -p --max_allowed_packet=100M
> >>
> >> If I did NOT include the "--max_allowed_packet=100M" on
> that command
> >> line, then when I did a "select" on the field base64Crl, I
> would get
> >> an error.
> >>
> >> I suspect that EJBCA is acting as a Mysql client, so, in
> order to
> >> work with such large CRLs, there needs to be some way to
> set the
> >> equivalent of the "--max_allowed_packet=100M" on the EJBCA
> side.
> >>
> >> How can I do that?
> >>
> >> If that can't be done, then I think that when EJBCA tries to
> >> retrieve the CRL from the Mysql, it (EJBCA) will see the
> same error.
> >>
> >> Thanks,
> >> Jim
> >>
> >> On Saturday, June 29, 2019, 3:25:47 AM EDT, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >>
> >> You configure max allowed packet size in you MySQL
> configuration on the
> >> MySQL server.
> >>
> >> /Tomas
> >>
> >> On 2019-06-29 02:19, o haya via Ejbca-develop wrote:
> >> > I am not 100% sure, but I think I know what might be
> causing this
> >> problem.
> >> >
> >> > I did a test, where I just tried to log into mysql and
> do the
> >> following
> >> > select:
> >> >
> >> > select base64Crl INTO OUTFILE
> '/var/lib/mysql-files/xx.out' from
> >> CRLData
> >> > where cRLNumber=3167;
> >> >
> >> > and I got an error saying that the result was more
> >> than max_allowed_packet.
> >> >
> >> > So then I did the same thing, but for the mysql command
> I used:
> >> >
> >> > mysql -u root -p --max_allowed_packet=100M
> >> >
> >> > and then when I did that same select, I was able to
> output the entire
> >> > CRL as a PEM.
> >> >
> >> > So, I am theorizing that that is what is happening with
> EJBCA, i.e.,
> >> > when I run the "openssl ocsp" command, EJBCA is querying the
> > base64CRL
> >> > but the result is too large and so that is causing the
> query to fail.
> >> >
> >> > So the question now is: How can I cause EJBCA OCSP
> Responder to run
> >> > with a larger max_allowed_packet so that the query
> succeeds (and then
> >> > hopefully EJBCA OCSP Responder will be able to check the
> revocation
> >> > properly)?
> >> >
> >> > Does anyone know how that can be done?
> >> >
> >> > Thanks,
> >> > Jim
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Friday, June 28, 2019, 07:16:39 PM UTC, o haya
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> >> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>> wrote:
> >> >
> >> >
> >> > Hi,
> >> >
> >> > As discussed in an earlier thread, it looks like I was
> able to
> >> import a
> >> > large (> 71MB) CRL file into EJBCA (as an OCSP
> Responder), so now,
> >> I am
> >> > doing testing, using "openssl ocsp" to send requests
> into the
> >> EJBCA OCSP
> >> > Responder:
> >> >
> >> > openssl ocsp -issuer <CA_CERT> -VAfile <SIGNING_CERT>
> >> -req_text -serial
> >> > 0x<CERT_SERIAL> -host <HOST:PORT>
> >> >
> >> > But it looks like no matter what serial number I use,
> whether the
> >> serial
> >> > number is in the CRL or not, returns a response like:
> >> >
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash: 37...00
> >> > Issuer Key Hash: 39...E4
> >> > Serial Number: 2FFFF9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 041078B1311984A110B5E19C8DC4895F485B
> >> > Response verify OK
> >> > 0x2FFFF9: unknown
> >> > This Update: Jun 28 18:56:20 2019 GMT
> >> >
> >> > i.e., it looks like the EJBCA OCSP Responder is not able
> to actually
> >> > find any revoked certs in the CRL?
> >> >
> >> >
> >> > FYI, I *KNOW* that the EJBCA OCSP Responder is functioning
> >> properly, in
> >> > general, because I can do the same test against a
> smaller CRL in the
> >> > same EJBCA OCSP Responder, for CA simpleca, and I can
> get an actual
> >> > "Revoked" response:
> >> >
> >> > [root@ejbca <mailto:root@ejbca> <mailto:root@ejbca
> <mailto:root@ejbca>> <mailto:root@ejbca <mailto:root@ejbca>
> > <mailto:root@ejbca <mailto:root@ejbca>>> testocspresponder]#
> openssl ocsp
> >> -issuer simpleca.crt
> >> > -VAfile simplecabinding2.crt -serial
> 0x008486394c03e1f5d9 -req_text
> >> > -host 192.168.0.28:80
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash:
> 0C16107310427EA4ADB3C6436915CE44A15FFE55
> >> > Issuer Key Hash:
> E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
> >> > Serial Number: 8486394C03E1F5D9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
> >> > Response verify OK
> >> > 0x008486394c03e1f5d9: revoked
> >> > This Update: Jun 28 19:12:48 2019 GMT
> >> > Reason: unspecified
> >> > Revocation Time: May 26 12:30:44 2019 GMT
> >> >
> >> > So it seems like the EJBCA OCSP Responder is either not
> > processing the
> >> > large CRL properly, or maybe the import didn't import
> the large
> >> CRL into
> >> > the OCSP Responder correctly.
> >> >
> >> > Can anyone tell me how do I diagnose this problem?
> >> >
> >> > Thanks,
> >> > Jim
> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > Ejbca-develop mailing list
> >> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> >
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-18 14:59:52
|
Hi,
The "machine" is a virtual machine under VirtualBox on a machine I have that has 64GB physical RAM. Drives are all 7200 RPM drives.
I will restore the snapshot, then increase the memory and # of processors and try to run the create index again.
Jim
On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
Just run create index script.
Create index should run pretty fast even with millions of records in the database. I suspect that your database is configured very slow, perhaps running everything on disl, without RAM, using slow disks?
A standard installation of mysql/mariadb on a centos machine should run faster. Then you have pretty low amount of ram for running everything on one box, handing millions of records.
Regards,
Tomas
On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
I had taken a snapshot of the machine before I tried to do the index earlier, so I restored that snapshot.
Then, I tried running the "optimize-ejbca-mysql.sql" and I got a different/new error:
ERROR 1091 (42000) at line 27: Can't DROP 'certificatedata_idx5'; check that column/key exists
Jim
On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
Oh oh, I just got this from the mysql execution:
ERROR 1071 (42000) at line 23: Specified key was too long; max key length is 1000 bytes
Jim
On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
FYI, I am running the create-index-ejbca.sql script now, but that seems to be taking FOREVER?
I had stopped the earlier import, and then deleted all the rows for that CA DN in CertificateData table, and then bounced the machine, then I ran the create-index-ejbca.sql.
Why is it taking so long? Should I have done something in addition to deleting the rows? I am guessing that the reason the sql script is taking so long is that it's somehow still trying to index the rows were deleted or something like that?
Thanks,Jim
On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
I notice that there is:
- create-index-ejbca.sql
- optimize-ejbca-mysql.sql
Should I run BOTH of them, or just the "create-index-ejbca.sql "?
Thanks,Jim
On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, i could see from your output from top that mysql takes all CPU. This indicates that you don't have any indexes in your database, which makes it horribly slow as numbers grow.
Apply database indexes from doc/sql-scripts.
Regards,
Tomas
On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
So far, processed only 185706 entries, and it looks like the processing is going to a crawl (really slow now).
Which kind of makes me wonder if maybe part of reason for the slowness is that it is having to search the certificate database for the serial number, and now, as there are more dummy certs in the database, that search is taking longer and longer?
Jim
On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
This is what top is showing now (the import is still running):
top - 13:56:19 up 1 day, 1:09, 3 users, load average: 1.18, 1.11, 1.07
Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
%Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 4045880 total, 123192 free, 3053072 used, 869616 buff/cache
KiB Swap: 3905532 total, 3753980 free, 151552 used. 688136 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2 1227:41 mysqld
3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2 139:05.67 java
3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3 176:26.68 java
22466 root 20 0 0 0 0 S 0.7 0.0 0:00.73 kworker/u4:1
That is running on 2-CPU machine with 4GB RAM.
Jim
On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, like top.
On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi Tomas,
I'm sorry that I missed the message below, but I am not clear what you are asking me to check/monitor? Do you mean like running something like "top" command?
Thanks,Jim
On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
To verify that the changes actually happen...
Do you monitor CPU usage to see what processes use CPU and memory?
On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
The import has been running for about an hour now and has imported 17836 entries.
Per what I had said previously, this CRL file has 1002255entries, that means it would take about 56+ hours to import this CRL.
So, this import processing is still way too slow.
I don't know what is involved, or even how much it would improve things, but I think that the next thing to try is to eliminated both the check for the serial number in the DB AND the dummy cert creation.
Jim
On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
Ok, I rebuilt the JAR using your Java class, and it is running now, and creating rows in the CertificateData, but it still SEEMS kind of slow.
I am running now and will try to get a measurement of revoked entries/time and will post back.
Jim
On Tuesday, July 16, 2019, 5:25:32 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
So I just tried what I said (emptying the "if") and you are right, it doesn't create the rows in the CeritificateData table :(...
Ok, I will try your version...
Jim
On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
I don't think so, no quite actually. In order to use the API to populate
the database you need the certificate object it seems. Something like
the attached file should work, it still creates a dummy certificate, but
uses a static hard coded key, so no key generation.
The file assumes you are on 6.15.x.
Regards,
Tomas
On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
> For testing (only...), I am thinking that if I modify this If clause in
> CaImportCrlCommand.java:
>
> if (adaptive)
> {
> Date time = new Date();
> KeyPair key_pair = KeyTools.genKeys("2048", "RSA");
>
> SubjectPublicKeyInfo pkinfo =
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
> X500Name dnName = new X500Name("CN=Dummy Missing in
> Imported CRL, serialNumber=" + serialHex);
> Date notAfter = new Date(time.getTime() + 315360000000L);
> X509v3CertificateBuilder certbuilder = new
> X509v3CertificateBuilder(X500Name.getInstance(cacert
> .getSubjectX500Principal().getEncoded()), serialNr,
> time, notAfter, dnName, pkinfo);
> ContentSigner signer = new BufferingContentSigner(new
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
> .getPrivate()), 20480);
> X509CertificateHolder certHolder = certbuilder.build(signer);
> X509Certificate certificate =
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
> X509Certificate.class);
>
> String fingerprint =
> CertTools.getFingerprintAsString(certificate);
>
> EndEntityInformation missingUserEndEntityInformation =
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
> missing_user_name);
> if (missingUserEndEntityInformation == null)
> {
> log.debug("Loading/updating user " + missing_user_name);
>
> EndEntityInformation userdataNew = new
> EndEntityInformation(missing_user_name,
> CertTools.getSubjectDN(certificate), cainfo.getCAId(), null, null, 10,
> new EndEntityType(EndEntityTypes.ENDUSER), 1, 1, null, null, 1, 0, null);
>
> userdataNew.setPassword("foo123");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
> userdataNew, false);
>
> log.info("User '" + missing_user_name + "' has been
> added.");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
> missing_user_name, 50);
>
> log.info("User '" + missing_user_name + "' has been
> updated.");
> }
>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>
> EJBTools.wrap(certificate), missing_user_name,
> fingerprint, 20, 1, 1, 0, null, new Date()
>
> .getTime());
> log.info("Dummy certificate '" + serialHex + "' has been
> stored.");
> }
>
> to (for example):
>
> if (adaptive)
> {
>
> log.info("*** SKIPPING *** Dummy certificate '" +
> serialHex + "' has been stored.");
> }
>
>
> Would THAT work??
>
> Thanks,
> Jim
>
>
>
>
> On Tuesday, July 16, 2019, 10:59:31 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi Tomas (or anyone else),
>
> Sorry - I was typing my last msg on phone, and it wasn't as detailed or
> clear as I wanted.
>
> Basically what I was saying is I was thinking that, as a quick sanity
> test, if there was one or two places where the code in
> the CaImportCRLCommand.java was maybe making a method call to create the
> dummy cert and put it into the database, then maybe I could decompile
> and modify the CaImportCRLCommand.java, and comment out those method
> calls, then, re-do my large CRL import test and time it?
>
> If so, can you let me know specifically where those calls are, and if I
> just comment them out, would that work?
>
> Ironically, that (not doing the dummy cert creation and writing) was
> what I *thought* the "LENIENT" option was doing).
>
> Thanks,
> Jim
>
> P. S. I have time today to try this if I figure out what needs to be done.
>
>
>
> On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
>
> Yes, we are definitely interested/willing to test patches. I actually
> was wondering if it would be a quick test to mod the code somewhere to
> avoid a call that creates the dummy cert, to get an idea if that would
> reduce the import time enough??
>
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>
> On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
> <to...@pr...> wrote:
> Most large users, some very very large, typically uses on line
> updates directly from the CA. Many CAs today don't accept only
> importing CRLs, as they need to keep track of issued certificates as
> well not only revoked (can't use "non existing is good").
> Having said thst, it's interesting to be able to import large CRLs,
> so improving it is of interest.
>
> Importing CRLs would actually not have to generate a dummy cert at
> all, since a certificate object is not needed on the responder,
> which should make it very fast.
>
> If you are willing to test patches, we can work with you on improving.
>
> Regards,
> Tomas
>
>
> On July 15, 2019 5:26:08 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> I am curious whether you have other users that import large CRLs
> like this?
>
> Are we the only ones who would encounter these really long
> import times, because the other users who might have large CRLs
> use the EJBCA CA also, so they don't run into having the dummy
> certs being created for every entry in the CRL?
>
> Jim
>
> On Monday, July 15, 2019, 1:42:57 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> I see...
>
> I can see that a new key pair is being generated for each dummy
> cert.
>
> In CaImportCRLCommand.java:
>
> final KeyPair key_pair = KeyTools.genKeys("2048",
> AlgorithmConstants.KEYALGORITHM_RSA);
>
> Chaning that to 1024 or 512 would be an easy improvement. A bit more
> complex, but even larger speedup would be to use a "hard coded"
> dummy
> key, i.e. same key for all dummy certs.
>
> You can experiment by changing 2048 to 512 (not secure but it
> doesn't
> have to be for this dummy cert).
>
> Regards,
> Tomas
>
> On 2019-07-11 14:19, ohaya--- via Ejbca-develop wrote:
> > FYI, the is running, but it is REALLY slow. The CRL file has
> 1002255
> > entries in it, and the import is only up to 8899 now, after
> about 40
> > minutes... which would take about 75 hours!!
> >
> > In our production systems, we are required to update some CRLs
> every
> > hour, so that would be a problem.
> >
> > What can I do to speed that up??
> >
> > Thanks,
> > Jim
> >
> >
> > On Thursday, July 11, 2019, 11:09:33 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > AHHH... Ok, it is running now, and from checking the
> CertificateData in
> > the Mysql database, I can see that it is now populating the
> > CertificateData!!
> >
> > Thanks!
> >
> > Jim
> >
> >
> >
> > On Thursday, July 11, 2019, 11:07:22 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > I wasn't sure if we should use ADAPTIVE, since we are not
> using the CA part.
> >
> >
> > I will try with "-o ADAPTIVE" now.
> >
> >
> >
> >
> > On Thursday, July 11, 2019, 7:42:14 AM UTC, Tomas Gustavsson
> > <to...@pr... <mailto:to...@pr...>> wrote:
> >
> >
> >
> > It should be ADAPTIVE.
> >
> > The documentation says:
> > bin/ejbca.sh ca importcrl --help
> > ---
> > ADAPTIVE means that missing certficates will be replaced by dummy
> > certificates to cater for proper CRLs for missing certificates.
> > ---
> >
> > Regards,
> > Tomas
> >
> > On 2019-07-11 08:05, ohaya--- via Ejbca-develop wrote:
> >> Hi,
> >>
> >> I've been looking at the code in CaImportCRLCommand, and I am
> wondering
> >> if, when the command is run in LENIENT mode and it doesn't
> find the cert
> >> in the database (and outputs the "missing in database" error
> message),
> >> what is supposed to happen?
> >>
> >> Mind you, the only error messages I am seeing are the "missing in
> >> database" error messages. I don't see any of the other error
> messages
> >> that are in/from CaImportCRLCommand.java.
> >>
> >> I am wondering if, when the command is run with "-o LENIENT",
> and when
> >> it outputs the "missing in database" error message, then the
> ejbca.sh is
> >> basically not doing anything (and not storing anything into the
> >> CertificateData table)?
> >>
> >> From the lack of any other error messages (vs. what I see in the
> >> CaImportCRLCommand.java code), that is what it looks like is
> happening.
> >>
> >> So given I am using EJBCA for OCSP responder only, is running the
> >> ejbca.sh in the "-o LENIENT" mode correct?
> >>
> >> Jim
> >>
> >>
> >>
> >> On Thursday, July 11, 2019, 5:43:37 AM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I've been doing a bunch of testing with trying to import the
> large CRL,
> >> and wanted to summarize:
> >>
> >> - The ejbca.sh runs fine - it completes without outputting an
> errors,
> >> except for outputting a bunch of the "cert xxxx missing in
> database"
> >> messages.
> >>
> >> - When the import completes, and I check the Mysql database,
> the crlData
> >> (the full PEM of the CRL) is in the database, but there is no
> rows in
> >> the CertificateData table for that CA at all.
> >>
> >> - I even figured out how to get the ejbca.sh to log using
> >> log4j.properties, and it output a bunch of logging, mainly
> the missing
> >> in database errors, but other than that I didn't see any
> obvious errors.
> >>
> >>
> >> So, bottom line is that even though the ejbca.sh seems to run
> without
> >> errors (other than the missing in database errors), the import is
> >> causing the CRL PEM to be stored in the database, but the
> >> CertificateData table is not being populated at all for that CA.
> >>
> >>
> >> Jim
> >>
> >>
> >>
> >> On Wednesday, July 10, 2019, 7:40:16 PM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> Ok, I found Tomas' original message... it was in my Spam
> folder....
> >>
> >> I also tried re-running the "./ejbca.sh ca importcrl" command
> a bunch of
> >> ways. I think it is running without outputting any errors,
> but still,
> >> no rows are appearing with that issuerDN at all.
> >>
> >>
> >>
> >>
> >> On Tuesday, July 9, 2019, 10:45:01 AM UTC, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> OCSP responses does not have anything to do with the CRL
> itself. Once
> >> the CRL has been imported each certificate status is in its
> own record
> >> in the CertificateData table. This is where status is read
> when reponses
> >> are created.
> >> If there are issues, it is in that case related to the import
> of the CRL.
> >>
> >> Verify that certificate records are created in
> CertificateData when
> >> importing the CRL.
> >> Also, a debug log when making OCSP request will tell exact
> details of
> >> what the responder finds in the database or not.
> >>
> >> Regards,
> >> Tomas
> >>
> >>
> >> On July 8, 2019 4:59:06 PM GMT+02:00, ohaya--- via Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >> [Re-sending, as this (handling of large CRLs) is still a
> problem and
> >> is still not working.]
> >>
> >>
> >>
> >> On Monday, July 1, 2019, 12:47:23 PM UTC, <oh...@ya...
> <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I think that I've found how to set/change the
> max_allowed_packet on
> >> the client (i.e., EJBCA) side, by editing the Mysql URL in the
> >> standalone/configuration/standalone.xml to:
> >>
> >>
> >
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
> >>
> >> However, when I test the OCSP Responder with a serial
> number (and CA
> >> cert) from the CA with the large (>71MB) CRL file, I am still
> >> getting "unknown" and EJBCA OCSP Responder is still NOT
> FINDING the
> >> cert serial number.
> >>
> >> NOTE, again, that I can test with certs (and CA cert) that
> have
> >> smaller CRL files, and those tests work.
> >>
> >> So it is only when testing with the large CRL file that is
> failing
> >> with the OCSP Responder.
> >>
> >> This seems like a potential BUG for EJBCA OCSP Responder?
> >>
> >> Please advise.
> >>
> >> Thanks,
> >> Jim
> >> On Saturday, June 29, 2019, 3:36:18 PM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> BTW, just to check, I have actually gone through the
> process of
> >> getting the crlData from the MySQL using mysql client (I
> had to add
> >> the --max_allowed-packet=100M to the mysql command) and I
> was able
> >> to actually get the PEM from the Mysql database that way.
> >>
> >> So, I KNOW that the entire CRL is in the Mysql database,
> and yet
> >> when I run an "openssl ocsp" test against the EJBCA OCSP
> Responder,
> >> it says it cannot find the revocation.
> >>
> >> OCSP Request Data:
> >> Version: 1 (0x0)
> >> Requestor List:
> >> Certificate ID:
> >> Hash Algorithm: sha1
> >> Issuer Name Hash: 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
> >> Issuer Key Hash: 3929A80864E28F83AA816147119133F85A37A6E4
> >> Serial Number: 2FFFF4
> >> Request Extensions:
> >> OCSP Nonce:
> >> 04103054A60A55E389B239A55228033D8F62
> >> Response verify OK
> >> 0x2FFFF4: unknown
> >> This Update: Jun 29 19:31:02 2019 GMT
> >>
> >>
> >> In the server log:
> >>
> >> 2019-06-29 15:31:02,150 DEBUG
> >>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
> >> (default task-1) Unable to find revocation information for
> >> certificate with serial '2ffff4' from issuer 'CN=XXX....'
> >>
> >>
> >> Honestly, I am no 100% sure that the problem is that
> >> max_allowe_packet is too small on the mysql client side,
> but that is
> >> my best guess.
> >>
> >> What I AM sure is that the CRL is in mysql, but when I do
> a test
> >> with OCSP Responder, it cannot find the cert serial number
> in the CRL.
> >>
> >> Jim
> >>
> >>
> >> On Saturday, June 29, 2019, 4:52:20 AM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi Tomas,
> >>
> >> I already had the max_allowed_packet configured on the Mysql
> >> server-side (in /etc/my.cnf).
> >>
> >> So as I said, when testing with mysql (client) I had to
> include
> >> "--max_allowed_packet=100M" on the mysql (client) command
> line, e.g.:
> >>
> >> mysql -u root -p --max_allowed_packet=100M
> >>
> >> If I did NOT include the "--max_allowed_packet=100M" on
> that command
> >> line, then when I did a "select" on the field base64Crl, I
> would get
> >> an error.
> >>
> >> I suspect that EJBCA is acting as a Mysql client, so, in
> order to
> >> work with such large CRLs, there needs to be some way to
> set the
> >> equivalent of the "--max_allowed_packet=100M" on the EJBCA
> side.
> >>
> >> How can I do that?
> >>
> >> If that can't be done, then I think that when EJBCA tries to
> >> retrieve the CRL from the Mysql, it (EJBCA) will see the
> same error.
> >>
> >> Thanks,
> >> Jim
> >>
> >> On Saturday, June 29, 2019, 3:25:47 AM EDT, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >>
> >> You configure max allowed packet size in you MySQL
> configuration on the
> >> MySQL server.
> >>
> >> /Tomas
> >>
> >> On 2019-06-29 02:19, o haya via Ejbca-develop wrote:
> >> > I am not 100% sure, but I think I know what might be
> causing this
> >> problem.
> >> >
> >> > I did a test, where I just tried to log into mysql and
> do the
> >> following
> >> > select:
> >> >
> >> > select base64Crl INTO OUTFILE
> '/var/lib/mysql-files/xx.out' from
> >> CRLData
> >> > where cRLNumber=3167;
> >> >
> >> > and I got an error saying that the result was more
> >> than max_allowed_packet.
> >> >
> >> > So then I did the same thing, but for the mysql command
> I used:
> >> >
> >> > mysql -u root -p --max_allowed_packet=100M
> >> >
> >> > and then when I did that same select, I was able to
> output the entire
> >> > CRL as a PEM.
> >> >
> >> > So, I am theorizing that that is what is happening with
> EJBCA, i.e.,
> >> > when I run the "openssl ocsp" command, EJBCA is querying the
> > base64CRL
> >> > but the result is too large and so that is causing the
> query to fail.
> >> >
> >> > So the question now is: How can I cause EJBCA OCSP
> Responder to run
> >> > with a larger max_allowed_packet so that the query
> succeeds (and then
> >> > hopefully EJBCA OCSP Responder will be able to check the
> revocation
> >> > properly)?
> >> >
> >> > Does anyone know how that can be done?
> >> >
> >> > Thanks,
> >> > Jim
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Friday, June 28, 2019, 07:16:39 PM UTC, o haya
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> >> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>> wrote:
> >> >
> >> >
> >> > Hi,
> >> >
> >> > As discussed in an earlier thread, it looks like I was
> able to
> >> import a
> >> > large (> 71MB) CRL file into EJBCA (as an OCSP
> Responder), so now,
> >> I am
> >> > doing testing, using "openssl ocsp" to send requests
> into the
> >> EJBCA OCSP
> >> > Responder:
> >> >
> >> > openssl ocsp -issuer <CA_CERT> -VAfile <SIGNING_CERT>
> >> -req_text -serial
> >> > 0x<CERT_SERIAL> -host <HOST:PORT>
> >> >
> >> > But it looks like no matter what serial number I use,
> whether the
> >> serial
> >> > number is in the CRL or not, returns a response like:
> >> >
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash: 37...00
> >> > Issuer Key Hash: 39...E4
> >> > Serial Number: 2FFFF9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 041078B1311984A110B5E19C8DC4895F485B
> >> > Response verify OK
> >> > 0x2FFFF9: unknown
> >> > This Update: Jun 28 18:56:20 2019 GMT
> >> >
> >> > i.e., it looks like the EJBCA OCSP Responder is not able
> to actually
> >> > find any revoked certs in the CRL?
> >> >
> >> >
> >> > FYI, I *KNOW* that the EJBCA OCSP Responder is functioning
> >> properly, in
> >> > general, because I can do the same test against a
> smaller CRL in the
> >> > same EJBCA OCSP Responder, for CA simpleca, and I can
> get an actual
> >> > "Revoked" response:
> >> >
> >> > [root@ejbca <mailto:root@ejbca> <mailto:root@ejbca
> <mailto:root@ejbca>> <mailto:root@ejbca <mailto:root@ejbca>
> > <mailto:root@ejbca <mailto:root@ejbca>>> testocspresponder]#
> openssl ocsp
> >> -issuer simpleca.crt
> >> > -VAfile simplecabinding2.crt -serial
> 0x008486394c03e1f5d9 -req_text
> >> > -host 192.168.0.28:80
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash:
> 0C16107310427EA4ADB3C6436915CE44A15FFE55
> >> > Issuer Key Hash:
> E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
> >> > Serial Number: 8486394C03E1F5D9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
> >> > Response verify OK
> >> > 0x008486394c03e1f5d9: revoked
> >> > This Update: Jun 28 19:12:48 2019 GMT
> >> > Reason: unspecified
> >> > Revocation Time: May 26 12:30:44 2019 GMT
> >> >
> >> > So it seems like the EJBCA OCSP Responder is either not
> > processing the
> >> > large CRL properly, or maybe the import didn't import
> the large
> >> CRL into
> >> > the OCSP Responder correctly.
> >> >
> >> > Can anyone tell me how do I diagnose this problem?
> >> >
> >> > Thanks,
> >> > Jim
> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > Ejbca-develop mailing list
> >> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> >
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: Tomas G. <to...@pr...> - 2019-07-18 14:32:22
|
Hi,
Just run create index script.
Create index should run pretty fast even with millions of records in the database. I suspect that your database is configured very slow, perhaps running everything on disl, without RAM, using slow disks?
A standard installation of mysql/mariadb on a centos machine should run faster. Then you have pretty low amount of ram for running everything on one box, handing millions of records.
Regards,
Tomas
On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
> Hi,
>I had taken a snapshot of the machine before I tried to do the index
>earlier, so I restored that snapshot.
>Then, I tried running the "optimize-ejbca-mysql.sql" and I got a
>different/new error:
>ERROR 1091 (42000) at line 27: Can't DROP 'certificatedata_idx5'; check
>that column/key exists
>
>Jim
>
>
>
>
>
>
>On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
><oh...@ya...> wrote:
>
> Hi,
>Oh oh, I just got this from the mysql execution:
>ERROR 1071 (42000) at line 23: Specified key was too long; max key
>length is 1000 bytes
>
>Jim
>
>
>
>
>
>
>On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
><oh...@ya...> wrote:
>
> Hi,
>FYI, I am running the create-index-ejbca.sql script now, but that seems
>to be taking FOREVER?
>I had stopped the earlier import, and then deleted all the rows for
>that CA DN in CertificateData table, and then bounced the machine, then
>I ran the create-index-ejbca.sql.
>Why is it taking so long? Should I have done something in addition to
>deleting the rows? I am guessing that the reason the sql script is
>taking so long is that it's somehow still trying to index the rows were
>deleted or something like that?
>Thanks,Jim
>
>
>
>On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
><oh...@ya...> wrote:
>
> Hi,
>
>I notice that there is:
>- create-index-ejbca.sql
>- optimize-ejbca-mysql.sql
>Should I run BOTH of them, or just the "create-index-ejbca.sql "?
>Thanks,Jim
>
>
>On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>Yes, i could see from your output from top that mysql takes all CPU.
>This indicates that you don't have any indexes in your database, which
>makes it horribly slow as numbers grow.
>
>Apply database indexes from doc/sql-scripts.
>
>Regards,
>Tomas
>
>
>On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
> Hi,
>So far, processed only 185706 entries, and it looks like the processing
>is going to a crawl (really slow now).
>Which kind of makes me wonder if maybe part of reason for the slowness
>is that it is having to search the certificate database for the serial
>number, and now, as there are more dummy certs in the database, that
>search is taking longer and longer?
>Jim
>
>On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>This is what top is showing now (the import is still running):
>top - 13:56:19 up 1 day, 1:09, 3 users, load average: 1.18, 1.11,
>1.07
>Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
>%Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0 hi, 0.3
>si, 0.0 st
>KiB Mem : 4045880 total, 123192 free, 3053072 used, 869616
>buff/cache
>KiB Swap: 3905532 total, 3753980 free, 151552 used. 688136 avail
>Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>COMMAND
> 1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2 1227:41
>mysqld
> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2 139:05.67
>java
> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3 176:26.68
>java
>22466 root 20 0 0 0 0 S 0.7 0.0 0:00.73
>kworker/u4:1
>
>That is running on 2-CPU machine with 4GB RAM.
>Jim
>
>
>
>
>On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
> Yes, like top.
>
>
>On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
> Hi Tomas,
>I'm sorry that I missed the message below, but I am not clear what you
>are asking me to check/monitor? Do you mean like running something
>like "top" command?
>Thanks,Jim
>
>On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
> To verify that the changes actually happen...
>
>Do you monitor CPU usage to see what processes use CPU and memory?
>
>
>On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
> Hi,
>The import has been running for about an hour now and has imported
>17836 entries.
>Per what I had said previously, this CRL file has 1002255entries, that
>means it would take about 56+ hours to import this CRL.
>So, this import processing is still way too slow.
>I don't know what is involved, or even how much it would improve
>things, but I think that the next thing to try is to eliminated both
>the check for the serial number in the DB AND the dummy cert creation.
>Jim
>
>On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya--- via Ejbca-develop
><ejb...@li...> wrote:
>
> Hi,
>Ok, I rebuilt the JAR using your Java class, and it is running now, and
>creating rows in the CertificateData, but it still SEEMS kind of slow.
>
>
>I am running now and will try to get a measurement of revoked
>entries/time and will post back.
>Jim
>
>
>
>On Tuesday, July 16, 2019, 5:25:32 PM UTC, oh...@ya...
><oh...@ya...> wrote:
>
> Hi,
>So I just tried what I said (emptying the "if") and you are right, it
>doesn't create the rows in the CeritificateData table :(...
>Ok, I will try your version...
>Jim
>
>On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>
>Hi,
>
>I don't think so, no quite actually. In order to use the API to
>populate
>the database you need the certificate object it seems. Something like
>the attached file should work, it still creates a dummy certificate,
>but
>uses a static hard coded key, so no key generation.
>
>The file assumes you are on 6.15.x.
>
>Regards,
>Tomas
>
>On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
>> For testing (only...), I am thinking that if I modify this If clause
>in
>> CaImportCrlCommand.java:
>>
>> if (adaptive)
>> {
>> Date time = new Date();
>> KeyPair key_pair = KeyTools.genKeys("2048", "RSA");
>>
>> SubjectPublicKeyInfo pkinfo =
>> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
>> X500Name dnName = new X500Name("CN=Dummy Missing in
>> Imported CRL, serialNumber=" + serialHex);
>> Date notAfter = new Date(time.getTime() +
>315360000000L);
>> X509v3CertificateBuilder certbuilder = new
>> X509v3CertificateBuilder(X500Name.getInstance(cacert
>> .getSubjectX500Principal().getEncoded()), serialNr,
>> time, notAfter, dnName, pkinfo);
>> ContentSigner signer = new BufferingContentSigner(new
>>
>JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
>> .getPrivate()), 20480);
>> X509CertificateHolder certHolder =
>certbuilder.build(signer);
>> X509Certificate certificate =
>>
>(X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
>> X509Certificate.class);
>>
>> String fingerprint =
>> CertTools.getFingerprintAsString(certificate);
>>
>> EndEntityInformation missingUserEndEntityInformation =
>>
>((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
>> missing_user_name);
>> if (missingUserEndEntityInformation == null)
>> {
>> log.debug("Loading/updating user " +
>missing_user_name);
>>
>> EndEntityInformation userdataNew = new
>> EndEntityInformation(missing_user_name,
>> CertTools.getSubjectDN(certificate), cainfo.getCAId(), null, null,
>10,
>> new EndEntityType(EndEntityTypes.ENDUSER), 1, 1, null, null, 1, 0,
>null);
>>
>> userdataNew.setPassword("foo123");
>>
>>
>((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
>> userdataNew, false);
>>
>> log.info("User '" + missing_user_name + "' has been
>> added.");
>>
>>
>((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
>> missing_user_name, 50);
>>
>> log.info("User '" + missing_user_name + "' has been
>> updated.");
>> }
>>
>>
>((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>>
>> EJBTools.wrap(certificate), missing_user_name,
>> fingerprint, 20, 1, 1, 0, null, new Date()
>>
>> .getTime());
>> log.info("Dummy certificate '" + serialHex + "' has
>been
>> stored.");
>> }
>>
>> to (for example):
>>
>> if (adaptive)
>> {
>>
>> log.info("*** SKIPPING *** Dummy certificate '" +
>> serialHex + "' has been stored.");
>> }
>>
>>
>> Would THAT work??
>>
>> Thanks,
>> Jim
>>
>>
>>
>>
>> On Tuesday, July 16, 2019, 10:59:31 AM UTC, ohaya--- via
>Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>> Hi Tomas (or anyone else),
>>
>> Sorry - I was typing my last msg on phone, and it wasn't as detailed
>or
>> clear as I wanted.
>>
>> Basically what I was saying is I was thinking that, as a quick sanity
>> test, if there was one or two places where the code in
>> the CaImportCRLCommand.java was maybe making a method call to create
>the
>> dummy cert and put it into the database, then maybe I could decompile
>> and modify the CaImportCRLCommand.java, and comment out those method
>> calls, then, re-do my large CRL import test and time it?
>>
>> If so, can you let me know specifically where those calls are, and if
>I
>> just comment them out, would that work?
>>
>> Ironically, that (not doing the dummy cert creation and writing) was
>> what I *thought* the "LENIENT" option was doing).
>>
>> Thanks,
>> Jim
>>
>> P. S. I have time today to try this if I figure out what needs to be
>done.
>>
>>
>>
>> On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya--- via Ejbca-develop
>> <ejb...@li...> wrote:
>>
>>
>>
>> Yes, we are definitely interested/willing to test patches. I
>actually
>> was wondering if it would be a quick test to mod the code somewhere
>to
>> avoid a call that creates the dummy cert, to get an idea if that
>would
>> reduce the import time enough??
>>
>>
>> Sent from Yahoo Mail on Android
>>
><https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>>
>> On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
>> <to...@pr...> wrote:
>> Most large users, some very very large, typically uses on line
>> updates directly from the CA. Many CAs today don't accept only
>> importing CRLs, as they need to keep track of issued certificates
>as
>> well not only revoked (can't use "non existing is good").
>> Having said thst, it's interesting to be able to import large
>CRLs,
>> so improving it is of interest.
>>
>> Importing CRLs would actually not have to generate a dummy cert at
>> all, since a certificate object is not needed on the responder,
>> which should make it very fast.
>>
>> If you are willing to test patches, we can work with you on
>improving.
>>
>> Regards,
>> Tomas
>>
>>
>> On July 15, 2019 5:26:08 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...> wrote:
>>
>> Hi,
>>
>> I am curious whether you have other users that import large
>CRLs
>> like this?
>>
>> Are we the only ones who would encounter these really long
>> import times, because the other users who might have large
>CRLs
>> use the EJBCA CA also, so they don't run into having the dummy
>> certs being created for every entry in the CRL?
>>
>> Jim
>>
>> On Monday, July 15, 2019, 1:42:57 PM UTC, Tomas Gustavsson
>> <to...@pr...> wrote:
>>
>>
>>
>> I see...
>>
>> I can see that a new key pair is being generated for each
>dummy
>> cert.
>>
>> In CaImportCRLCommand.java:
>>
>> final KeyPair key_pair = KeyTools.genKeys("2048",
>> AlgorithmConstants.KEYALGORITHM_RSA);
>>
>> Chaning that to 1024 or 512 would be an easy improvement. A
>bit more
>> complex, but even larger speedup would be to use a "hard
>coded"
>> dummy
>> key, i.e. same key for all dummy certs.
>>
>> You can experiment by changing 2048 to 512 (not secure but it
>> doesn't
>> have to be for this dummy cert).
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-11 14:19, ohaya--- via Ejbca-develop wrote:
>> > FYI, the is running, but it is REALLY slow. The CRL file
>has
>> 1002255
>> > entries in it, and the import is only up to 8899 now, after
>> about 40
>> > minutes... which would take about 75 hours!!
>> >
>> > In our production systems, we are required to update some
>CRLs
>> every
>> > hour, so that would be a problem.
>> >
>> > What can I do to speed that up??
>> >
>> > Thanks,
>> > Jim
>> >
>> >
>> > On Thursday, July 11, 2019, 11:09:33 AM UTC, ohaya--- via
>> Ejbca-develop
>> > <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>> >
>> >
>> > AHHH... Ok, it is running now, and from checking the
>> CertificateData in
>> > the Mysql database, I can see that it is now populating the
>> > CertificateData!!
>> >
>> > Thanks!
>> >
>> > Jim
>> >
>> >
>> >
>> > On Thursday, July 11, 2019, 11:07:22 AM UTC, ohaya--- via
>> Ejbca-develop
>> > <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>> >
>> >
>> > I wasn't sure if we should use ADAPTIVE, since we are not
>> using the CA part.
>> >
>> >
>> > I will try with "-o ADAPTIVE" now.
>> >
>> >
>> >
>> >
>> > On Thursday, July 11, 2019, 7:42:14 AM UTC, Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>> wrote:
>> >
>> >
>> >
>> > It should be ADAPTIVE.
>> >
>> > The documentation says:
>> > bin/ejbca.sh ca importcrl --help
>> > ---
>> > ADAPTIVE means that missing certficates will be replaced by
>dummy
>> > certificates to cater for proper CRLs for missing
>certificates.
>> > ---
>> >
>> > Regards,
>> > Tomas
>> >
>> > On 2019-07-11 08:05, ohaya--- via Ejbca-develop wrote:
>> >> Hi,
>> >>
>> >> I've been looking at the code in CaImportCRLCommand, and I
>am
>> wondering
>> >> if, when the command is run in LENIENT mode and it doesn't
>> find the cert
>> >> in the database (and outputs the "missing in database"
>error
>> message),
>> >> what is supposed to happen?
>> >>
>> >> Mind you, the only error messages I am seeing are the
>"missing in
>> >> database" error messages. I don't see any of the other
>error
>> messages
>> >> that are in/from CaImportCRLCommand.java.
>> >>
>> >> I am wondering if, when the command is run with "-o
>LENIENT",
>> and when
>> >> it outputs the "missing in database" error message, then
>the
>> ejbca.sh is
>> >> basically not doing anything (and not storing anything into
>the
>> >> CertificateData table)?
>> >>
>> >> From the lack of any other error messages (vs. what I see
>in the
>> >> CaImportCRLCommand.java code), that is what it looks like
>is
>> happening.
>> >>
>> >> So given I am using EJBCA for OCSP responder only, is
>running the
>> >> ejbca.sh in the "-o LENIENT" mode correct?
>> >>
>> >> Jim
>> >>
>> >>
>> >>
>> >> On Thursday, July 11, 2019, 5:43:37 AM UTC, ohaya--- via
>> Ejbca-develop
>> >> <ejb...@li...
>> <mailto:ejb...@li...>
>> > <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I've been doing a bunch of testing with trying to import
>the
>> large CRL,
>> >> and wanted to summarize:
>> >>
>> >> - The ejbca.sh runs fine - it completes without outputting
>an
>> errors,
>> >> except for outputting a bunch of the "cert xxxx missing in
>> database"
>> >> messages.
>> >>
>> >> - When the import completes, and I check the Mysql
>database,
>> the crlData
>> >> (the full PEM of the CRL) is in the database, but there is
>no
>> rows in
>> >> the CertificateData table for that CA at all.
>> >>
>> >> - I even figured out how to get the ejbca.sh to log using
>> >> log4j.properties, and it output a bunch of logging, mainly
>> the missing
>> >> in database errors, but other than that I didn't see any
>> obvious errors.
>> >>
>> >>
>> >> So, bottom line is that even though the ejbca.sh seems to
>run
>> without
>> >> errors (other than the missing in database errors), the
>import is
>> >> causing the CRL PEM to be stored in the database, but the
>> >> CertificateData table is not being populated at all for
>that CA.
>> >>
>> >>
>> >> Jim
>> >>
>> >>
>> >>
>> >> On Wednesday, July 10, 2019, 7:40:16 PM UTC, ohaya--- via
>> Ejbca-develop
>> >> <ejb...@li...
>> <mailto:ejb...@li...>
>> > <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> Ok, I found Tomas' original message... it was in my Spam
>> folder....
>> >>
>> >> I also tried re-running the "./ejbca.sh ca importcrl"
>command
>> a bunch of
>> >> ways. I think it is running without outputting any errors,
>> but still,
>> >> no rows are appearing with that issuerDN at all.
>> >>
>> >>
>> >>
>> >>
>> >> On Tuesday, July 9, 2019, 10:45:01 AM UTC, Tomas Gustavsson
>> >> <to...@pr... <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> OCSP responses does not have anything to do with the CRL
>> itself. Once
>> >> the CRL has been imported each certificate status is in its
>> own record
>> >> in the CertificateData table. This is where status is read
>> when reponses
>> >> are created.
>> >> If there are issues, it is in that case related to the
>import
>> of the CRL.
>> >>
>> >> Verify that certificate records are created in
>> CertificateData when
>> >> importing the CRL.
>> >> Also, a debug log when making OCSP request will tell exact
>> details of
>> >> what the responder finds in the database or not.
>> >>
>> >> Regards,
>> >> Tomas
>> >>
>> >>
>> >> On July 8, 2019 4:59:06 PM GMT+02:00, ohaya--- via
>Ejbca-develop
>> >> <ejb...@li...
>> <mailto:ejb...@li...>
>> > <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>> >>
>> >> [Re-sending, as this (handling of large CRLs) is still a
>> problem and
>> >> is still not working.]
>> >>
>> >>
>> >>
>> >> On Monday, July 1, 2019, 12:47:23 PM UTC,
><oh...@ya...
>> <mailto:oh...@ya...>
>> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I think that I've found how to set/change the
>> max_allowed_packet on
>> >> the client (i.e., EJBCA) side, by editing the Mysql URL
>in the
>> >> standalone/configuration/standalone.xml to:
>> >>
>> >>
>> >
>>
><connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
>> >>
>> >> However, when I test the OCSP Responder with a serial
>> number (and CA
>> >> cert) from the CA with the large (>71MB) CRL file, I am
>still
>> >> getting "unknown" and EJBCA OCSP Responder is still NOT
>> FINDING the
>> >> cert serial number.
>> >>
>> >> NOTE, again, that I can test with certs (and CA cert)
>that
>> have
>> >> smaller CRL files, and those tests work.
>> >>
>> >> So it is only when testing with the large CRL file that
>is
>> failing
>> >> with the OCSP Responder.
>> >>
>> >> This seems like a potential BUG for EJBCA OCSP
>Responder?
>> >>
>> >> Please advise.
>> >>
>> >> Thanks,
>> >> Jim
>> >> On Saturday, June 29, 2019, 3:36:18 PM EDT,
>> <oh...@ya... <mailto:oh...@ya...>
>> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> BTW, just to check, I have actually gone through the
>> process of
>> >> getting the crlData from the MySQL using mysql client (I
>> had to add
>> >> the --max_allowed-packet=100M to the mysql command) and
>I
>> was able
>> >> to actually get the PEM from the Mysql database that
>way.
>> >>
>> >> So, I KNOW that the entire CRL is in the Mysql database,
>> and yet
>> >> when I run an "openssl ocsp" test against the EJBCA OCSP
>> Responder,
>> >> it says it cannot find the revocation.
>> >>
>> >> OCSP Request Data:
>> >> Version: 1 (0x0)
>> >> Requestor List:
>> >> Certificate ID:
>> >> Hash Algorithm: sha1
>> >> Issuer Name Hash:
>37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
>> >> Issuer Key Hash:
>3929A80864E28F83AA816147119133F85A37A6E4
>> >> Serial Number: 2FFFF4
>> >> Request Extensions:
>> >> OCSP Nonce:
>> >> 04103054A60A55E389B239A55228033D8F62
>> >> Response verify OK
>> >> 0x2FFFF4: unknown
>> >> This Update: Jun 29 19:31:02 2019 GMT
>> >>
>> >>
>> >> In the server log:
>> >>
>> >> 2019-06-29 15:31:02,150 DEBUG
>> >>
>>
>[org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
>> >> (default task-1) Unable to find revocation information
>for
>> >> certificate with serial '2ffff4' from issuer
>'CN=XXX....'
>> >>
>> >>
>> >> Honestly, I am no 100% sure that the problem is that
>> >> max_allowe_packet is too small on the mysql client side,
>> but that is
>> >> my best guess.
>> >>
>> >> What I AM sure is that the CRL is in mysql, but when I
>do
>> a test
>> >> with OCSP Responder, it cannot find the cert serial
>number
>> in the CRL.
>> >>
>> >> Jim
>> >>
>> >>
>> >> On Saturday, June 29, 2019, 4:52:20 AM EDT,
>> <oh...@ya... <mailto:oh...@ya...>
>> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
>> >>
>> >>
>> >> Hi Tomas,
>> >>
>> >> I already had the max_allowed_packet configured on the
>Mysql
>> >> server-side (in /etc/my.cnf).
>> >>
>> >> So as I said, when testing with mysql (client) I had to
>> include
>> >> "--max_allowed_packet=100M" on the mysql (client)
>command
>> line, e.g.:
>> >>
>> >> mysql -u root -p --max_allowed_packet=100M
>> >>
>> >> If I did NOT include the "--max_allowed_packet=100M" on
>> that command
>> >> line, then when I did a "select" on the field base64Crl,
>I
>> would get
>> >> an error.
>> >>
>> >> I suspect that EJBCA is acting as a Mysql client, so, in
>> order to
>> >> work with such large CRLs, there needs to be some way to
>> set the
>> >> equivalent of the "--max_allowed_packet=100M" on the
>EJBCA
>> side.
>> >>
>> >> How can I do that?
>> >>
>> >> If that can't be done, then I think that when EJBCA
>tries to
>> >> retrieve the CRL from the Mysql, it (EJBCA) will see the
>> same error.
>> >>
>> >> Thanks,
>> >> Jim
>> >>
>> >> On Saturday, June 29, 2019, 3:25:47 AM EDT, Tomas
>Gustavsson
>> >> <to...@pr... <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
>> >>
>> >>
>> >>
>> >> You configure max allowed packet size in you MySQL
>> configuration on the
>> >> MySQL server.
>> >>
>> >> /Tomas
>> >>
>> >> On 2019-06-29 02:19, o haya via Ejbca-develop wrote:
>> >> > I am not 100% sure, but I think I know what might be
>> causing this
>> >> problem.
>> >> >
>> >> > I did a test, where I just tried to log into mysql and
>> do the
>> >> following
>> >> > select:
>> >> >
>> >> > select base64Crl INTO OUTFILE
>> '/var/lib/mysql-files/xx.out' from
>> >> CRLData
>> >> > where cRLNumber=3167;
>> >> >
>> >> > and I got an error saying that the result was more
>> >> than max_allowed_packet.
>> >> >
>> >> > So then I did the same thing, but for the mysql
>command
>> I used:
>> >> >
>> >> > mysql -u root -p --max_allowed_packet=100M
>> >> >
>> >> > and then when I did that same select, I was able to
>> output the entire
>> >> > CRL as a PEM.
>> >> >
>> >> > So, I am theorizing that that is what is happening
>with
>> EJBCA, i.e.,
>> >> > when I run the "openssl ocsp" command, EJBCA is
>querying the
>> > base64CRL
>> >> > but the result is too large and so that is causing the
>> query to fail.
>> >> >
>> >> > So the question now is: How can I cause EJBCA OCSP
>> Responder to run
>> >> > with a larger max_allowed_packet so that the query
>> succeeds (and then
>> >> > hopefully EJBCA OCSP Responder will be able to check
>the
>> revocation
>> >> > properly)?
>> >> >
>> >> > Does anyone know how that can be done?
>> >> >
>> >> > Thanks,
>> >> > Jim
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Friday, June 28, 2019, 07:16:39 PM UTC, o haya
>> > <oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> >> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>> wrote:
>> >> >
>> >> >
>> >> > Hi,
>> >> >
>> >> > As discussed in an earlier thread, it looks like I was
>> able to
>> >> import a
>> >> > large (> 71MB) CRL file into EJBCA (as an OCSP
>> Responder), so now,
>> >> I am
>> >> > doing testing, using "openssl ocsp" to send requests
>> into the
>> >> EJBCA OCSP
>> >> > Responder:
>> >> >
>> >> > openssl ocsp -issuer <CA_CERT> -VAfile <SIGNING_CERT>
>> >> -req_text -serial
>> >> > 0x<CERT_SERIAL> -host <HOST:PORT>
>> >> >
>> >> > But it looks like no matter what serial number I use,
>> whether the
>> >> serial
>> >> > number is in the CRL or not, returns a response like:
>> >> >
>> >> > OCSP Request Data:
>> >> > Version: 1 (0x0)
>> >> > Requestor List:
>> >> > Certificate ID:
>> >> > Hash Algorithm: sha1
>> >> > Issuer Name Hash: 37...00
>> >> > Issuer Key Hash: 39...E4
>> >> > Serial Number: 2FFFF9
>> >> > Request Extensions:
>> >> > OCSP Nonce:
>> >> > 041078B1311984A110B5E19C8DC4895F485B
>> >> > Response verify OK
>> >> > 0x2FFFF9: unknown
>> >> > This Update: Jun 28 18:56:20 2019 GMT
>> >> >
>> >> > i.e., it looks like the EJBCA OCSP Responder is not
>able
>> to actually
>> >> > find any revoked certs in the CRL?
>> >> >
>> >> >
>> >> > FYI, I *KNOW* that the EJBCA OCSP Responder is
>functioning
>> >> properly, in
>> >> > general, because I can do the same test against a
>> smaller CRL in the
>> >> > same EJBCA OCSP Responder, for CA simpleca, and I can
>> get an actual
>> >> > "Revoked" response:
>> >> >
>> >> > [root@ejbca <mailto:root@ejbca> <mailto:root@ejbca
>> <mailto:root@ejbca>> <mailto:root@ejbca <mailto:root@ejbca>
>> > <mailto:root@ejbca <mailto:root@ejbca>>> testocspresponder]#
>> openssl ocsp
>> >> -issuer simpleca.crt
>> >> > -VAfile simplecabinding2.crt -serial
>> 0x008486394c03e1f5d9 -req_text
>> >> > -host 192.168.0.28:80
>> >> > OCSP Request Data:
>> >> > Version: 1 (0x0)
>> >> > Requestor List:
>> >> > Certificate ID:
>> >> > Hash Algorithm: sha1
>> >> > Issuer Name Hash:
>> 0C16107310427EA4ADB3C6436915CE44A15FFE55
>> >> > Issuer Key Hash:
>> E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
>> >> > Serial Number: 8486394C03E1F5D9
>> >> > Request Extensions:
>> >> > OCSP Nonce:
>> >> > 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
>> >> > Response verify OK
>> >> > 0x008486394c03e1f5d9: revoked
>> >> > This Update: Jun 28 19:12:48 2019 GMT
>> >> > Reason: unspecified
>> >> > Revocation Time: May 26 12:30:44 2019 GMT
>> >> >
>> >> > So it seems like the EJBCA OCSP Responder is either
>not
>> > processing the
>> >> > large CRL properly, or maybe the import didn't import
>> the large
>> >> CRL into
>> >> > the OCSP Responder correctly.
>> >> >
>> >> > Can anyone tell me how do I diagnose this problem?
>> >> >
>> >> > Thanks,
>> >> > Jim
>> >>
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Ejbca-develop mailing list
>> >> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>>
>> >> >
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >> >
>> >>
>> >>
>> >> _______________________________________________
>> >> Ejbca-develop mailing list
>> >> Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>>
>> >>
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >>
>> >> _______________________________________________
>> >> Ejbca-develop mailing list
>> >> Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>>
>> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >> _______________________________________________
>> >> Ejbca-develop mailing list
>> >> Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>>
>> >
>> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >>
>> >>
>> >> _______________________________________________
>> >> Ejbca-develop mailing list
>> >> Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >>
>> >
>> >
>> > _______________________________________________
>> > Ejbca-develop mailing list
>> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> > _______________________________________________
>> > Ejbca-develop mailing list
>> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> > _______________________________________________
>> > Ejbca-develop mailing list
>> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > <mailto:Ejb...@li...
>> <mailto:Ejb...@li...>>
>> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >
>> >
>> > _______________________________________________
>> > Ejbca-develop mailing list
>> > Ejb...@li...
>> <mailto:Ejb...@li...>
>> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> >
>>
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> <mailto:Ejb...@li...>
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>>
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>>
>_______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
> _______________________________________________
>Ejbca-develop mailing list
>Ejb...@li...
>https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
|
|
From: <oh...@ya...> - 2019-07-18 14:23:14
|
Hi,
I had taken a snapshot of the machine before I tried to do the index earlier, so I restored that snapshot.
Then, I tried running the "optimize-ejbca-mysql.sql" and I got a different/new error:
ERROR 1091 (42000) at line 27: Can't DROP 'certificatedata_idx5'; check that column/key exists
Jim
On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
Oh oh, I just got this from the mysql execution:
ERROR 1071 (42000) at line 23: Specified key was too long; max key length is 1000 bytes
Jim
On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
FYI, I am running the create-index-ejbca.sql script now, but that seems to be taking FOREVER?
I had stopped the earlier import, and then deleted all the rows for that CA DN in CertificateData table, and then bounced the machine, then I ran the create-index-ejbca.sql.
Why is it taking so long? Should I have done something in addition to deleting the rows? I am guessing that the reason the sql script is taking so long is that it's somehow still trying to index the rows were deleted or something like that?
Thanks,Jim
On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
I notice that there is:
- create-index-ejbca.sql
- optimize-ejbca-mysql.sql
Should I run BOTH of them, or just the "create-index-ejbca.sql "?
Thanks,Jim
On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, i could see from your output from top that mysql takes all CPU. This indicates that you don't have any indexes in your database, which makes it horribly slow as numbers grow.
Apply database indexes from doc/sql-scripts.
Regards,
Tomas
On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
So far, processed only 185706 entries, and it looks like the processing is going to a crawl (really slow now).
Which kind of makes me wonder if maybe part of reason for the slowness is that it is having to search the certificate database for the serial number, and now, as there are more dummy certs in the database, that search is taking longer and longer?
Jim
On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
This is what top is showing now (the import is still running):
top - 13:56:19 up 1 day, 1:09, 3 users, load average: 1.18, 1.11, 1.07
Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
%Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 4045880 total, 123192 free, 3053072 used, 869616 buff/cache
KiB Swap: 3905532 total, 3753980 free, 151552 used. 688136 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2 1227:41 mysqld
3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2 139:05.67 java
3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3 176:26.68 java
22466 root 20 0 0 0 0 S 0.7 0.0 0:00.73 kworker/u4:1
That is running on 2-CPU machine with 4GB RAM.
Jim
On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, like top.
On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi Tomas,
I'm sorry that I missed the message below, but I am not clear what you are asking me to check/monitor? Do you mean like running something like "top" command?
Thanks,Jim
On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
To verify that the changes actually happen...
Do you monitor CPU usage to see what processes use CPU and memory?
On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
The import has been running for about an hour now and has imported 17836 entries.
Per what I had said previously, this CRL file has 1002255entries, that means it would take about 56+ hours to import this CRL.
So, this import processing is still way too slow.
I don't know what is involved, or even how much it would improve things, but I think that the next thing to try is to eliminated both the check for the serial number in the DB AND the dummy cert creation.
Jim
On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
Ok, I rebuilt the JAR using your Java class, and it is running now, and creating rows in the CertificateData, but it still SEEMS kind of slow.
I am running now and will try to get a measurement of revoked entries/time and will post back.
Jim
On Tuesday, July 16, 2019, 5:25:32 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
So I just tried what I said (emptying the "if") and you are right, it doesn't create the rows in the CeritificateData table :(...
Ok, I will try your version...
Jim
On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
I don't think so, no quite actually. In order to use the API to populate
the database you need the certificate object it seems. Something like
the attached file should work, it still creates a dummy certificate, but
uses a static hard coded key, so no key generation.
The file assumes you are on 6.15.x.
Regards,
Tomas
On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
> For testing (only...), I am thinking that if I modify this If clause in
> CaImportCrlCommand.java:
>
> if (adaptive)
> {
> Date time = new Date();
> KeyPair key_pair = KeyTools.genKeys("2048", "RSA");
>
> SubjectPublicKeyInfo pkinfo =
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
> X500Name dnName = new X500Name("CN=Dummy Missing in
> Imported CRL, serialNumber=" + serialHex);
> Date notAfter = new Date(time.getTime() + 315360000000L);
> X509v3CertificateBuilder certbuilder = new
> X509v3CertificateBuilder(X500Name.getInstance(cacert
> .getSubjectX500Principal().getEncoded()), serialNr,
> time, notAfter, dnName, pkinfo);
> ContentSigner signer = new BufferingContentSigner(new
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
> .getPrivate()), 20480);
> X509CertificateHolder certHolder = certbuilder.build(signer);
> X509Certificate certificate =
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
> X509Certificate.class);
>
> String fingerprint =
> CertTools.getFingerprintAsString(certificate);
>
> EndEntityInformation missingUserEndEntityInformation =
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
> missing_user_name);
> if (missingUserEndEntityInformation == null)
> {
> log.debug("Loading/updating user " + missing_user_name);
>
> EndEntityInformation userdataNew = new
> EndEntityInformation(missing_user_name,
> CertTools.getSubjectDN(certificate), cainfo.getCAId(), null, null, 10,
> new EndEntityType(EndEntityTypes.ENDUSER), 1, 1, null, null, 1, 0, null);
>
> userdataNew.setPassword("foo123");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
> userdataNew, false);
>
> log.info("User '" + missing_user_name + "' has been
> added.");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
> missing_user_name, 50);
>
> log.info("User '" + missing_user_name + "' has been
> updated.");
> }
>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>
> EJBTools.wrap(certificate), missing_user_name,
> fingerprint, 20, 1, 1, 0, null, new Date()
>
> .getTime());
> log.info("Dummy certificate '" + serialHex + "' has been
> stored.");
> }
>
> to (for example):
>
> if (adaptive)
> {
>
> log.info("*** SKIPPING *** Dummy certificate '" +
> serialHex + "' has been stored.");
> }
>
>
> Would THAT work??
>
> Thanks,
> Jim
>
>
>
>
> On Tuesday, July 16, 2019, 10:59:31 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi Tomas (or anyone else),
>
> Sorry - I was typing my last msg on phone, and it wasn't as detailed or
> clear as I wanted.
>
> Basically what I was saying is I was thinking that, as a quick sanity
> test, if there was one or two places where the code in
> the CaImportCRLCommand.java was maybe making a method call to create the
> dummy cert and put it into the database, then maybe I could decompile
> and modify the CaImportCRLCommand.java, and comment out those method
> calls, then, re-do my large CRL import test and time it?
>
> If so, can you let me know specifically where those calls are, and if I
> just comment them out, would that work?
>
> Ironically, that (not doing the dummy cert creation and writing) was
> what I *thought* the "LENIENT" option was doing).
>
> Thanks,
> Jim
>
> P. S. I have time today to try this if I figure out what needs to be done.
>
>
>
> On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
>
> Yes, we are definitely interested/willing to test patches. I actually
> was wondering if it would be a quick test to mod the code somewhere to
> avoid a call that creates the dummy cert, to get an idea if that would
> reduce the import time enough??
>
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>
> On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
> <to...@pr...> wrote:
> Most large users, some very very large, typically uses on line
> updates directly from the CA. Many CAs today don't accept only
> importing CRLs, as they need to keep track of issued certificates as
> well not only revoked (can't use "non existing is good").
> Having said thst, it's interesting to be able to import large CRLs,
> so improving it is of interest.
>
> Importing CRLs would actually not have to generate a dummy cert at
> all, since a certificate object is not needed on the responder,
> which should make it very fast.
>
> If you are willing to test patches, we can work with you on improving.
>
> Regards,
> Tomas
>
>
> On July 15, 2019 5:26:08 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> I am curious whether you have other users that import large CRLs
> like this?
>
> Are we the only ones who would encounter these really long
> import times, because the other users who might have large CRLs
> use the EJBCA CA also, so they don't run into having the dummy
> certs being created for every entry in the CRL?
>
> Jim
>
> On Monday, July 15, 2019, 1:42:57 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> I see...
>
> I can see that a new key pair is being generated for each dummy
> cert.
>
> In CaImportCRLCommand.java:
>
> final KeyPair key_pair = KeyTools.genKeys("2048",
> AlgorithmConstants.KEYALGORITHM_RSA);
>
> Chaning that to 1024 or 512 would be an easy improvement. A bit more
> complex, but even larger speedup would be to use a "hard coded"
> dummy
> key, i.e. same key for all dummy certs.
>
> You can experiment by changing 2048 to 512 (not secure but it
> doesn't
> have to be for this dummy cert).
>
> Regards,
> Tomas
>
> On 2019-07-11 14:19, ohaya--- via Ejbca-develop wrote:
> > FYI, the is running, but it is REALLY slow. The CRL file has
> 1002255
> > entries in it, and the import is only up to 8899 now, after
> about 40
> > minutes... which would take about 75 hours!!
> >
> > In our production systems, we are required to update some CRLs
> every
> > hour, so that would be a problem.
> >
> > What can I do to speed that up??
> >
> > Thanks,
> > Jim
> >
> >
> > On Thursday, July 11, 2019, 11:09:33 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > AHHH... Ok, it is running now, and from checking the
> CertificateData in
> > the Mysql database, I can see that it is now populating the
> > CertificateData!!
> >
> > Thanks!
> >
> > Jim
> >
> >
> >
> > On Thursday, July 11, 2019, 11:07:22 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > I wasn't sure if we should use ADAPTIVE, since we are not
> using the CA part.
> >
> >
> > I will try with "-o ADAPTIVE" now.
> >
> >
> >
> >
> > On Thursday, July 11, 2019, 7:42:14 AM UTC, Tomas Gustavsson
> > <to...@pr... <mailto:to...@pr...>> wrote:
> >
> >
> >
> > It should be ADAPTIVE.
> >
> > The documentation says:
> > bin/ejbca.sh ca importcrl --help
> > ---
> > ADAPTIVE means that missing certficates will be replaced by dummy
> > certificates to cater for proper CRLs for missing certificates.
> > ---
> >
> > Regards,
> > Tomas
> >
> > On 2019-07-11 08:05, ohaya--- via Ejbca-develop wrote:
> >> Hi,
> >>
> >> I've been looking at the code in CaImportCRLCommand, and I am
> wondering
> >> if, when the command is run in LENIENT mode and it doesn't
> find the cert
> >> in the database (and outputs the "missing in database" error
> message),
> >> what is supposed to happen?
> >>
> >> Mind you, the only error messages I am seeing are the "missing in
> >> database" error messages. I don't see any of the other error
> messages
> >> that are in/from CaImportCRLCommand.java.
> >>
> >> I am wondering if, when the command is run with "-o LENIENT",
> and when
> >> it outputs the "missing in database" error message, then the
> ejbca.sh is
> >> basically not doing anything (and not storing anything into the
> >> CertificateData table)?
> >>
> >> From the lack of any other error messages (vs. what I see in the
> >> CaImportCRLCommand.java code), that is what it looks like is
> happening.
> >>
> >> So given I am using EJBCA for OCSP responder only, is running the
> >> ejbca.sh in the "-o LENIENT" mode correct?
> >>
> >> Jim
> >>
> >>
> >>
> >> On Thursday, July 11, 2019, 5:43:37 AM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I've been doing a bunch of testing with trying to import the
> large CRL,
> >> and wanted to summarize:
> >>
> >> - The ejbca.sh runs fine - it completes without outputting an
> errors,
> >> except for outputting a bunch of the "cert xxxx missing in
> database"
> >> messages.
> >>
> >> - When the import completes, and I check the Mysql database,
> the crlData
> >> (the full PEM of the CRL) is in the database, but there is no
> rows in
> >> the CertificateData table for that CA at all.
> >>
> >> - I even figured out how to get the ejbca.sh to log using
> >> log4j.properties, and it output a bunch of logging, mainly
> the missing
> >> in database errors, but other than that I didn't see any
> obvious errors.
> >>
> >>
> >> So, bottom line is that even though the ejbca.sh seems to run
> without
> >> errors (other than the missing in database errors), the import is
> >> causing the CRL PEM to be stored in the database, but the
> >> CertificateData table is not being populated at all for that CA.
> >>
> >>
> >> Jim
> >>
> >>
> >>
> >> On Wednesday, July 10, 2019, 7:40:16 PM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> Ok, I found Tomas' original message... it was in my Spam
> folder....
> >>
> >> I also tried re-running the "./ejbca.sh ca importcrl" command
> a bunch of
> >> ways. I think it is running without outputting any errors,
> but still,
> >> no rows are appearing with that issuerDN at all.
> >>
> >>
> >>
> >>
> >> On Tuesday, July 9, 2019, 10:45:01 AM UTC, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> OCSP responses does not have anything to do with the CRL
> itself. Once
> >> the CRL has been imported each certificate status is in its
> own record
> >> in the CertificateData table. This is where status is read
> when reponses
> >> are created.
> >> If there are issues, it is in that case related to the import
> of the CRL.
> >>
> >> Verify that certificate records are created in
> CertificateData when
> >> importing the CRL.
> >> Also, a debug log when making OCSP request will tell exact
> details of
> >> what the responder finds in the database or not.
> >>
> >> Regards,
> >> Tomas
> >>
> >>
> >> On July 8, 2019 4:59:06 PM GMT+02:00, ohaya--- via Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >> [Re-sending, as this (handling of large CRLs) is still a
> problem and
> >> is still not working.]
> >>
> >>
> >>
> >> On Monday, July 1, 2019, 12:47:23 PM UTC, <oh...@ya...
> <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I think that I've found how to set/change the
> max_allowed_packet on
> >> the client (i.e., EJBCA) side, by editing the Mysql URL in the
> >> standalone/configuration/standalone.xml to:
> >>
> >>
> >
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
> >>
> >> However, when I test the OCSP Responder with a serial
> number (and CA
> >> cert) from the CA with the large (>71MB) CRL file, I am still
> >> getting "unknown" and EJBCA OCSP Responder is still NOT
> FINDING the
> >> cert serial number.
> >>
> >> NOTE, again, that I can test with certs (and CA cert) that
> have
> >> smaller CRL files, and those tests work.
> >>
> >> So it is only when testing with the large CRL file that is
> failing
> >> with the OCSP Responder.
> >>
> >> This seems like a potential BUG for EJBCA OCSP Responder?
> >>
> >> Please advise.
> >>
> >> Thanks,
> >> Jim
> >> On Saturday, June 29, 2019, 3:36:18 PM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> BTW, just to check, I have actually gone through the
> process of
> >> getting the crlData from the MySQL using mysql client (I
> had to add
> >> the --max_allowed-packet=100M to the mysql command) and I
> was able
> >> to actually get the PEM from the Mysql database that way.
> >>
> >> So, I KNOW that the entire CRL is in the Mysql database,
> and yet
> >> when I run an "openssl ocsp" test against the EJBCA OCSP
> Responder,
> >> it says it cannot find the revocation.
> >>
> >> OCSP Request Data:
> >> Version: 1 (0x0)
> >> Requestor List:
> >> Certificate ID:
> >> Hash Algorithm: sha1
> >> Issuer Name Hash: 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
> >> Issuer Key Hash: 3929A80864E28F83AA816147119133F85A37A6E4
> >> Serial Number: 2FFFF4
> >> Request Extensions:
> >> OCSP Nonce:
> >> 04103054A60A55E389B239A55228033D8F62
> >> Response verify OK
> >> 0x2FFFF4: unknown
> >> This Update: Jun 29 19:31:02 2019 GMT
> >>
> >>
> >> In the server log:
> >>
> >> 2019-06-29 15:31:02,150 DEBUG
> >>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
> >> (default task-1) Unable to find revocation information for
> >> certificate with serial '2ffff4' from issuer 'CN=XXX....'
> >>
> >>
> >> Honestly, I am no 100% sure that the problem is that
> >> max_allowe_packet is too small on the mysql client side,
> but that is
> >> my best guess.
> >>
> >> What I AM sure is that the CRL is in mysql, but when I do
> a test
> >> with OCSP Responder, it cannot find the cert serial number
> in the CRL.
> >>
> >> Jim
> >>
> >>
> >> On Saturday, June 29, 2019, 4:52:20 AM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi Tomas,
> >>
> >> I already had the max_allowed_packet configured on the Mysql
> >> server-side (in /etc/my.cnf).
> >>
> >> So as I said, when testing with mysql (client) I had to
> include
> >> "--max_allowed_packet=100M" on the mysql (client) command
> line, e.g.:
> >>
> >> mysql -u root -p --max_allowed_packet=100M
> >>
> >> If I did NOT include the "--max_allowed_packet=100M" on
> that command
> >> line, then when I did a "select" on the field base64Crl, I
> would get
> >> an error.
> >>
> >> I suspect that EJBCA is acting as a Mysql client, so, in
> order to
> >> work with such large CRLs, there needs to be some way to
> set the
> >> equivalent of the "--max_allowed_packet=100M" on the EJBCA
> side.
> >>
> >> How can I do that?
> >>
> >> If that can't be done, then I think that when EJBCA tries to
> >> retrieve the CRL from the Mysql, it (EJBCA) will see the
> same error.
> >>
> >> Thanks,
> >> Jim
> >>
> >> On Saturday, June 29, 2019, 3:25:47 AM EDT, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >>
> >> You configure max allowed packet size in you MySQL
> configuration on the
> >> MySQL server.
> >>
> >> /Tomas
> >>
> >> On 2019-06-29 02:19, o haya via Ejbca-develop wrote:
> >> > I am not 100% sure, but I think I know what might be
> causing this
> >> problem.
> >> >
> >> > I did a test, where I just tried to log into mysql and
> do the
> >> following
> >> > select:
> >> >
> >> > select base64Crl INTO OUTFILE
> '/var/lib/mysql-files/xx.out' from
> >> CRLData
> >> > where cRLNumber=3167;
> >> >
> >> > and I got an error saying that the result was more
> >> than max_allowed_packet.
> >> >
> >> > So then I did the same thing, but for the mysql command
> I used:
> >> >
> >> > mysql -u root -p --max_allowed_packet=100M
> >> >
> >> > and then when I did that same select, I was able to
> output the entire
> >> > CRL as a PEM.
> >> >
> >> > So, I am theorizing that that is what is happening with
> EJBCA, i.e.,
> >> > when I run the "openssl ocsp" command, EJBCA is querying the
> > base64CRL
> >> > but the result is too large and so that is causing the
> query to fail.
> >> >
> >> > So the question now is: How can I cause EJBCA OCSP
> Responder to run
> >> > with a larger max_allowed_packet so that the query
> succeeds (and then
> >> > hopefully EJBCA OCSP Responder will be able to check the
> revocation
> >> > properly)?
> >> >
> >> > Does anyone know how that can be done?
> >> >
> >> > Thanks,
> >> > Jim
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Friday, June 28, 2019, 07:16:39 PM UTC, o haya
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> >> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>> wrote:
> >> >
> >> >
> >> > Hi,
> >> >
> >> > As discussed in an earlier thread, it looks like I was
> able to
> >> import a
> >> > large (> 71MB) CRL file into EJBCA (as an OCSP
> Responder), so now,
> >> I am
> >> > doing testing, using "openssl ocsp" to send requests
> into the
> >> EJBCA OCSP
> >> > Responder:
> >> >
> >> > openssl ocsp -issuer <CA_CERT> -VAfile <SIGNING_CERT>
> >> -req_text -serial
> >> > 0x<CERT_SERIAL> -host <HOST:PORT>
> >> >
> >> > But it looks like no matter what serial number I use,
> whether the
> >> serial
> >> > number is in the CRL or not, returns a response like:
> >> >
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash: 37...00
> >> > Issuer Key Hash: 39...E4
> >> > Serial Number: 2FFFF9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 041078B1311984A110B5E19C8DC4895F485B
> >> > Response verify OK
> >> > 0x2FFFF9: unknown
> >> > This Update: Jun 28 18:56:20 2019 GMT
> >> >
> >> > i.e., it looks like the EJBCA OCSP Responder is not able
> to actually
> >> > find any revoked certs in the CRL?
> >> >
> >> >
> >> > FYI, I *KNOW* that the EJBCA OCSP Responder is functioning
> >> properly, in
> >> > general, because I can do the same test against a
> smaller CRL in the
> >> > same EJBCA OCSP Responder, for CA simpleca, and I can
> get an actual
> >> > "Revoked" response:
> >> >
> >> > [root@ejbca <mailto:root@ejbca> <mailto:root@ejbca
> <mailto:root@ejbca>> <mailto:root@ejbca <mailto:root@ejbca>
> > <mailto:root@ejbca <mailto:root@ejbca>>> testocspresponder]#
> openssl ocsp
> >> -issuer simpleca.crt
> >> > -VAfile simplecabinding2.crt -serial
> 0x008486394c03e1f5d9 -req_text
> >> > -host 192.168.0.28:80
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash:
> 0C16107310427EA4ADB3C6436915CE44A15FFE55
> >> > Issuer Key Hash:
> E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
> >> > Serial Number: 8486394C03E1F5D9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
> >> > Response verify OK
> >> > 0x008486394c03e1f5d9: revoked
> >> > This Update: Jun 28 19:12:48 2019 GMT
> >> > Reason: unspecified
> >> > Revocation Time: May 26 12:30:44 2019 GMT
> >> >
> >> > So it seems like the EJBCA OCSP Responder is either not
> > processing the
> >> > large CRL properly, or maybe the import didn't import
> the large
> >> CRL into
> >> > the OCSP Responder correctly.
> >> >
> >> > Can anyone tell me how do I diagnose this problem?
> >> >
> >> > Thanks,
> >> > Jim
> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > Ejbca-develop mailing list
> >> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> >
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-18 14:15:23
|
Hi,
Oh oh, I just got this from the mysql execution:
ERROR 1071 (42000) at line 23: Specified key was too long; max key length is 1000 bytes
Jim
On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
FYI, I am running the create-index-ejbca.sql script now, but that seems to be taking FOREVER?
I had stopped the earlier import, and then deleted all the rows for that CA DN in CertificateData table, and then bounced the machine, then I ran the create-index-ejbca.sql.
Why is it taking so long? Should I have done something in addition to deleting the rows? I am guessing that the reason the sql script is taking so long is that it's somehow still trying to index the rows were deleted or something like that?
Thanks,Jim
On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
I notice that there is:
- create-index-ejbca.sql
- optimize-ejbca-mysql.sql
Should I run BOTH of them, or just the "create-index-ejbca.sql "?
Thanks,Jim
On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, i could see from your output from top that mysql takes all CPU. This indicates that you don't have any indexes in your database, which makes it horribly slow as numbers grow.
Apply database indexes from doc/sql-scripts.
Regards,
Tomas
On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
So far, processed only 185706 entries, and it looks like the processing is going to a crawl (really slow now).
Which kind of makes me wonder if maybe part of reason for the slowness is that it is having to search the certificate database for the serial number, and now, as there are more dummy certs in the database, that search is taking longer and longer?
Jim
On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
This is what top is showing now (the import is still running):
top - 13:56:19 up 1 day, 1:09, 3 users, load average: 1.18, 1.11, 1.07
Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
%Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 4045880 total, 123192 free, 3053072 used, 869616 buff/cache
KiB Swap: 3905532 total, 3753980 free, 151552 used. 688136 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2 1227:41 mysqld
3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2 139:05.67 java
3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3 176:26.68 java
22466 root 20 0 0 0 0 S 0.7 0.0 0:00.73 kworker/u4:1
That is running on 2-CPU machine with 4GB RAM.
Jim
On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, like top.
On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi Tomas,
I'm sorry that I missed the message below, but I am not clear what you are asking me to check/monitor? Do you mean like running something like "top" command?
Thanks,Jim
On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
To verify that the changes actually happen...
Do you monitor CPU usage to see what processes use CPU and memory?
On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
The import has been running for about an hour now and has imported 17836 entries.
Per what I had said previously, this CRL file has 1002255entries, that means it would take about 56+ hours to import this CRL.
So, this import processing is still way too slow.
I don't know what is involved, or even how much it would improve things, but I think that the next thing to try is to eliminated both the check for the serial number in the DB AND the dummy cert creation.
Jim
On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
Ok, I rebuilt the JAR using your Java class, and it is running now, and creating rows in the CertificateData, but it still SEEMS kind of slow.
I am running now and will try to get a measurement of revoked entries/time and will post back.
Jim
On Tuesday, July 16, 2019, 5:25:32 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
So I just tried what I said (emptying the "if") and you are right, it doesn't create the rows in the CeritificateData table :(...
Ok, I will try your version...
Jim
On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
I don't think so, no quite actually. In order to use the API to populate
the database you need the certificate object it seems. Something like
the attached file should work, it still creates a dummy certificate, but
uses a static hard coded key, so no key generation.
The file assumes you are on 6.15.x.
Regards,
Tomas
On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
> For testing (only...), I am thinking that if I modify this If clause in
> CaImportCrlCommand.java:
>
> if (adaptive)
> {
> Date time = new Date();
> KeyPair key_pair = KeyTools.genKeys("2048", "RSA");
>
> SubjectPublicKeyInfo pkinfo =
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
> X500Name dnName = new X500Name("CN=Dummy Missing in
> Imported CRL, serialNumber=" + serialHex);
> Date notAfter = new Date(time.getTime() + 315360000000L);
> X509v3CertificateBuilder certbuilder = new
> X509v3CertificateBuilder(X500Name.getInstance(cacert
> .getSubjectX500Principal().getEncoded()), serialNr,
> time, notAfter, dnName, pkinfo);
> ContentSigner signer = new BufferingContentSigner(new
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
> .getPrivate()), 20480);
> X509CertificateHolder certHolder = certbuilder.build(signer);
> X509Certificate certificate =
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
> X509Certificate.class);
>
> String fingerprint =
> CertTools.getFingerprintAsString(certificate);
>
> EndEntityInformation missingUserEndEntityInformation =
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
> missing_user_name);
> if (missingUserEndEntityInformation == null)
> {
> log.debug("Loading/updating user " + missing_user_name);
>
> EndEntityInformation userdataNew = new
> EndEntityInformation(missing_user_name,
> CertTools.getSubjectDN(certificate), cainfo.getCAId(), null, null, 10,
> new EndEntityType(EndEntityTypes.ENDUSER), 1, 1, null, null, 1, 0, null);
>
> userdataNew.setPassword("foo123");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
> userdataNew, false);
>
> log.info("User '" + missing_user_name + "' has been
> added.");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
> missing_user_name, 50);
>
> log.info("User '" + missing_user_name + "' has been
> updated.");
> }
>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>
> EJBTools.wrap(certificate), missing_user_name,
> fingerprint, 20, 1, 1, 0, null, new Date()
>
> .getTime());
> log.info("Dummy certificate '" + serialHex + "' has been
> stored.");
> }
>
> to (for example):
>
> if (adaptive)
> {
>
> log.info("*** SKIPPING *** Dummy certificate '" +
> serialHex + "' has been stored.");
> }
>
>
> Would THAT work??
>
> Thanks,
> Jim
>
>
>
>
> On Tuesday, July 16, 2019, 10:59:31 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi Tomas (or anyone else),
>
> Sorry - I was typing my last msg on phone, and it wasn't as detailed or
> clear as I wanted.
>
> Basically what I was saying is I was thinking that, as a quick sanity
> test, if there was one or two places where the code in
> the CaImportCRLCommand.java was maybe making a method call to create the
> dummy cert and put it into the database, then maybe I could decompile
> and modify the CaImportCRLCommand.java, and comment out those method
> calls, then, re-do my large CRL import test and time it?
>
> If so, can you let me know specifically where those calls are, and if I
> just comment them out, would that work?
>
> Ironically, that (not doing the dummy cert creation and writing) was
> what I *thought* the "LENIENT" option was doing).
>
> Thanks,
> Jim
>
> P. S. I have time today to try this if I figure out what needs to be done.
>
>
>
> On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
>
> Yes, we are definitely interested/willing to test patches. I actually
> was wondering if it would be a quick test to mod the code somewhere to
> avoid a call that creates the dummy cert, to get an idea if that would
> reduce the import time enough??
>
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>
> On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
> <to...@pr...> wrote:
> Most large users, some very very large, typically uses on line
> updates directly from the CA. Many CAs today don't accept only
> importing CRLs, as they need to keep track of issued certificates as
> well not only revoked (can't use "non existing is good").
> Having said thst, it's interesting to be able to import large CRLs,
> so improving it is of interest.
>
> Importing CRLs would actually not have to generate a dummy cert at
> all, since a certificate object is not needed on the responder,
> which should make it very fast.
>
> If you are willing to test patches, we can work with you on improving.
>
> Regards,
> Tomas
>
>
> On July 15, 2019 5:26:08 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> I am curious whether you have other users that import large CRLs
> like this?
>
> Are we the only ones who would encounter these really long
> import times, because the other users who might have large CRLs
> use the EJBCA CA also, so they don't run into having the dummy
> certs being created for every entry in the CRL?
>
> Jim
>
> On Monday, July 15, 2019, 1:42:57 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> I see...
>
> I can see that a new key pair is being generated for each dummy
> cert.
>
> In CaImportCRLCommand.java:
>
> final KeyPair key_pair = KeyTools.genKeys("2048",
> AlgorithmConstants.KEYALGORITHM_RSA);
>
> Chaning that to 1024 or 512 would be an easy improvement. A bit more
> complex, but even larger speedup would be to use a "hard coded"
> dummy
> key, i.e. same key for all dummy certs.
>
> You can experiment by changing 2048 to 512 (not secure but it
> doesn't
> have to be for this dummy cert).
>
> Regards,
> Tomas
>
> On 2019-07-11 14:19, ohaya--- via Ejbca-develop wrote:
> > FYI, the is running, but it is REALLY slow. The CRL file has
> 1002255
> > entries in it, and the import is only up to 8899 now, after
> about 40
> > minutes... which would take about 75 hours!!
> >
> > In our production systems, we are required to update some CRLs
> every
> > hour, so that would be a problem.
> >
> > What can I do to speed that up??
> >
> > Thanks,
> > Jim
> >
> >
> > On Thursday, July 11, 2019, 11:09:33 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > AHHH... Ok, it is running now, and from checking the
> CertificateData in
> > the Mysql database, I can see that it is now populating the
> > CertificateData!!
> >
> > Thanks!
> >
> > Jim
> >
> >
> >
> > On Thursday, July 11, 2019, 11:07:22 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > I wasn't sure if we should use ADAPTIVE, since we are not
> using the CA part.
> >
> >
> > I will try with "-o ADAPTIVE" now.
> >
> >
> >
> >
> > On Thursday, July 11, 2019, 7:42:14 AM UTC, Tomas Gustavsson
> > <to...@pr... <mailto:to...@pr...>> wrote:
> >
> >
> >
> > It should be ADAPTIVE.
> >
> > The documentation says:
> > bin/ejbca.sh ca importcrl --help
> > ---
> > ADAPTIVE means that missing certficates will be replaced by dummy
> > certificates to cater for proper CRLs for missing certificates.
> > ---
> >
> > Regards,
> > Tomas
> >
> > On 2019-07-11 08:05, ohaya--- via Ejbca-develop wrote:
> >> Hi,
> >>
> >> I've been looking at the code in CaImportCRLCommand, and I am
> wondering
> >> if, when the command is run in LENIENT mode and it doesn't
> find the cert
> >> in the database (and outputs the "missing in database" error
> message),
> >> what is supposed to happen?
> >>
> >> Mind you, the only error messages I am seeing are the "missing in
> >> database" error messages. I don't see any of the other error
> messages
> >> that are in/from CaImportCRLCommand.java.
> >>
> >> I am wondering if, when the command is run with "-o LENIENT",
> and when
> >> it outputs the "missing in database" error message, then the
> ejbca.sh is
> >> basically not doing anything (and not storing anything into the
> >> CertificateData table)?
> >>
> >> From the lack of any other error messages (vs. what I see in the
> >> CaImportCRLCommand.java code), that is what it looks like is
> happening.
> >>
> >> So given I am using EJBCA for OCSP responder only, is running the
> >> ejbca.sh in the "-o LENIENT" mode correct?
> >>
> >> Jim
> >>
> >>
> >>
> >> On Thursday, July 11, 2019, 5:43:37 AM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I've been doing a bunch of testing with trying to import the
> large CRL,
> >> and wanted to summarize:
> >>
> >> - The ejbca.sh runs fine - it completes without outputting an
> errors,
> >> except for outputting a bunch of the "cert xxxx missing in
> database"
> >> messages.
> >>
> >> - When the import completes, and I check the Mysql database,
> the crlData
> >> (the full PEM of the CRL) is in the database, but there is no
> rows in
> >> the CertificateData table for that CA at all.
> >>
> >> - I even figured out how to get the ejbca.sh to log using
> >> log4j.properties, and it output a bunch of logging, mainly
> the missing
> >> in database errors, but other than that I didn't see any
> obvious errors.
> >>
> >>
> >> So, bottom line is that even though the ejbca.sh seems to run
> without
> >> errors (other than the missing in database errors), the import is
> >> causing the CRL PEM to be stored in the database, but the
> >> CertificateData table is not being populated at all for that CA.
> >>
> >>
> >> Jim
> >>
> >>
> >>
> >> On Wednesday, July 10, 2019, 7:40:16 PM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> Ok, I found Tomas' original message... it was in my Spam
> folder....
> >>
> >> I also tried re-running the "./ejbca.sh ca importcrl" command
> a bunch of
> >> ways. I think it is running without outputting any errors,
> but still,
> >> no rows are appearing with that issuerDN at all.
> >>
> >>
> >>
> >>
> >> On Tuesday, July 9, 2019, 10:45:01 AM UTC, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> OCSP responses does not have anything to do with the CRL
> itself. Once
> >> the CRL has been imported each certificate status is in its
> own record
> >> in the CertificateData table. This is where status is read
> when reponses
> >> are created.
> >> If there are issues, it is in that case related to the import
> of the CRL.
> >>
> >> Verify that certificate records are created in
> CertificateData when
> >> importing the CRL.
> >> Also, a debug log when making OCSP request will tell exact
> details of
> >> what the responder finds in the database or not.
> >>
> >> Regards,
> >> Tomas
> >>
> >>
> >> On July 8, 2019 4:59:06 PM GMT+02:00, ohaya--- via Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >> [Re-sending, as this (handling of large CRLs) is still a
> problem and
> >> is still not working.]
> >>
> >>
> >>
> >> On Monday, July 1, 2019, 12:47:23 PM UTC, <oh...@ya...
> <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I think that I've found how to set/change the
> max_allowed_packet on
> >> the client (i.e., EJBCA) side, by editing the Mysql URL in the
> >> standalone/configuration/standalone.xml to:
> >>
> >>
> >
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
> >>
> >> However, when I test the OCSP Responder with a serial
> number (and CA
> >> cert) from the CA with the large (>71MB) CRL file, I am still
> >> getting "unknown" and EJBCA OCSP Responder is still NOT
> FINDING the
> >> cert serial number.
> >>
> >> NOTE, again, that I can test with certs (and CA cert) that
> have
> >> smaller CRL files, and those tests work.
> >>
> >> So it is only when testing with the large CRL file that is
> failing
> >> with the OCSP Responder.
> >>
> >> This seems like a potential BUG for EJBCA OCSP Responder?
> >>
> >> Please advise.
> >>
> >> Thanks,
> >> Jim
> >> On Saturday, June 29, 2019, 3:36:18 PM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> BTW, just to check, I have actually gone through the
> process of
> >> getting the crlData from the MySQL using mysql client (I
> had to add
> >> the --max_allowed-packet=100M to the mysql command) and I
> was able
> >> to actually get the PEM from the Mysql database that way.
> >>
> >> So, I KNOW that the entire CRL is in the Mysql database,
> and yet
> >> when I run an "openssl ocsp" test against the EJBCA OCSP
> Responder,
> >> it says it cannot find the revocation.
> >>
> >> OCSP Request Data:
> >> Version: 1 (0x0)
> >> Requestor List:
> >> Certificate ID:
> >> Hash Algorithm: sha1
> >> Issuer Name Hash: 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
> >> Issuer Key Hash: 3929A80864E28F83AA816147119133F85A37A6E4
> >> Serial Number: 2FFFF4
> >> Request Extensions:
> >> OCSP Nonce:
> >> 04103054A60A55E389B239A55228033D8F62
> >> Response verify OK
> >> 0x2FFFF4: unknown
> >> This Update: Jun 29 19:31:02 2019 GMT
> >>
> >>
> >> In the server log:
> >>
> >> 2019-06-29 15:31:02,150 DEBUG
> >>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
> >> (default task-1) Unable to find revocation information for
> >> certificate with serial '2ffff4' from issuer 'CN=XXX....'
> >>
> >>
> >> Honestly, I am no 100% sure that the problem is that
> >> max_allowe_packet is too small on the mysql client side,
> but that is
> >> my best guess.
> >>
> >> What I AM sure is that the CRL is in mysql, but when I do
> a test
> >> with OCSP Responder, it cannot find the cert serial number
> in the CRL.
> >>
> >> Jim
> >>
> >>
> >> On Saturday, June 29, 2019, 4:52:20 AM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi Tomas,
> >>
> >> I already had the max_allowed_packet configured on the Mysql
> >> server-side (in /etc/my.cnf).
> >>
> >> So as I said, when testing with mysql (client) I had to
> include
> >> "--max_allowed_packet=100M" on the mysql (client) command
> line, e.g.:
> >>
> >> mysql -u root -p --max_allowed_packet=100M
> >>
> >> If I did NOT include the "--max_allowed_packet=100M" on
> that command
> >> line, then when I did a "select" on the field base64Crl, I
> would get
> >> an error.
> >>
> >> I suspect that EJBCA is acting as a Mysql client, so, in
> order to
> >> work with such large CRLs, there needs to be some way to
> set the
> >> equivalent of the "--max_allowed_packet=100M" on the EJBCA
> side.
> >>
> >> How can I do that?
> >>
> >> If that can't be done, then I think that when EJBCA tries to
> >> retrieve the CRL from the Mysql, it (EJBCA) will see the
> same error.
> >>
> >> Thanks,
> >> Jim
> >>
> >> On Saturday, June 29, 2019, 3:25:47 AM EDT, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >>
> >> You configure max allowed packet size in you MySQL
> configuration on the
> >> MySQL server.
> >>
> >> /Tomas
> >>
> >> On 2019-06-29 02:19, o haya via Ejbca-develop wrote:
> >> > I am not 100% sure, but I think I know what might be
> causing this
> >> problem.
> >> >
> >> > I did a test, where I just tried to log into mysql and
> do the
> >> following
> >> > select:
> >> >
> >> > select base64Crl INTO OUTFILE
> '/var/lib/mysql-files/xx.out' from
> >> CRLData
> >> > where cRLNumber=3167;
> >> >
> >> > and I got an error saying that the result was more
> >> than max_allowed_packet.
> >> >
> >> > So then I did the same thing, but for the mysql command
> I used:
> >> >
> >> > mysql -u root -p --max_allowed_packet=100M
> >> >
> >> > and then when I did that same select, I was able to
> output the entire
> >> > CRL as a PEM.
> >> >
> >> > So, I am theorizing that that is what is happening with
> EJBCA, i.e.,
> >> > when I run the "openssl ocsp" command, EJBCA is querying the
> > base64CRL
> >> > but the result is too large and so that is causing the
> query to fail.
> >> >
> >> > So the question now is: How can I cause EJBCA OCSP
> Responder to run
> >> > with a larger max_allowed_packet so that the query
> succeeds (and then
> >> > hopefully EJBCA OCSP Responder will be able to check the
> revocation
> >> > properly)?
> >> >
> >> > Does anyone know how that can be done?
> >> >
> >> > Thanks,
> >> > Jim
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Friday, June 28, 2019, 07:16:39 PM UTC, o haya
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> >> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>> wrote:
> >> >
> >> >
> >> > Hi,
> >> >
> >> > As discussed in an earlier thread, it looks like I was
> able to
> >> import a
> >> > large (> 71MB) CRL file into EJBCA (as an OCSP
> Responder), so now,
> >> I am
> >> > doing testing, using "openssl ocsp" to send requests
> into the
> >> EJBCA OCSP
> >> > Responder:
> >> >
> >> > openssl ocsp -issuer <CA_CERT> -VAfile <SIGNING_CERT>
> >> -req_text -serial
> >> > 0x<CERT_SERIAL> -host <HOST:PORT>
> >> >
> >> > But it looks like no matter what serial number I use,
> whether the
> >> serial
> >> > number is in the CRL or not, returns a response like:
> >> >
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash: 37...00
> >> > Issuer Key Hash: 39...E4
> >> > Serial Number: 2FFFF9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 041078B1311984A110B5E19C8DC4895F485B
> >> > Response verify OK
> >> > 0x2FFFF9: unknown
> >> > This Update: Jun 28 18:56:20 2019 GMT
> >> >
> >> > i.e., it looks like the EJBCA OCSP Responder is not able
> to actually
> >> > find any revoked certs in the CRL?
> >> >
> >> >
> >> > FYI, I *KNOW* that the EJBCA OCSP Responder is functioning
> >> properly, in
> >> > general, because I can do the same test against a
> smaller CRL in the
> >> > same EJBCA OCSP Responder, for CA simpleca, and I can
> get an actual
> >> > "Revoked" response:
> >> >
> >> > [root@ejbca <mailto:root@ejbca> <mailto:root@ejbca
> <mailto:root@ejbca>> <mailto:root@ejbca <mailto:root@ejbca>
> > <mailto:root@ejbca <mailto:root@ejbca>>> testocspresponder]#
> openssl ocsp
> >> -issuer simpleca.crt
> >> > -VAfile simplecabinding2.crt -serial
> 0x008486394c03e1f5d9 -req_text
> >> > -host 192.168.0.28:80
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash:
> 0C16107310427EA4ADB3C6436915CE44A15FFE55
> >> > Issuer Key Hash:
> E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
> >> > Serial Number: 8486394C03E1F5D9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
> >> > Response verify OK
> >> > 0x008486394c03e1f5d9: revoked
> >> > This Update: Jun 28 19:12:48 2019 GMT
> >> > Reason: unspecified
> >> > Revocation Time: May 26 12:30:44 2019 GMT
> >> >
> >> > So it seems like the EJBCA OCSP Responder is either not
> > processing the
> >> > large CRL properly, or maybe the import didn't import
> the large
> >> CRL into
> >> > the OCSP Responder correctly.
> >> >
> >> > Can anyone tell me how do I diagnose this problem?
> >> >
> >> > Thanks,
> >> > Jim
> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > Ejbca-develop mailing list
> >> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> >
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
|
From: <oh...@ya...> - 2019-07-18 14:09:43
|
Hi,
FYI, I am running the create-index-ejbca.sql script now, but that seems to be taking FOREVER?
I had stopped the earlier import, and then deleted all the rows for that CA DN in CertificateData table, and then bounced the machine, then I ran the create-index-ejbca.sql.
Why is it taking so long? Should I have done something in addition to deleting the rows? I am guessing that the reason the sql script is taking so long is that it's somehow still trying to index the rows were deleted or something like that?
Thanks,Jim
On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
I notice that there is:
- create-index-ejbca.sql
- optimize-ejbca-mysql.sql
Should I run BOTH of them, or just the "create-index-ejbca.sql "?
Thanks,Jim
On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, i could see from your output from top that mysql takes all CPU. This indicates that you don't have any indexes in your database, which makes it horribly slow as numbers grow.
Apply database indexes from doc/sql-scripts.
Regards,
Tomas
On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
So far, processed only 185706 entries, and it looks like the processing is going to a crawl (really slow now).
Which kind of makes me wonder if maybe part of reason for the slowness is that it is having to search the certificate database for the serial number, and now, as there are more dummy certs in the database, that search is taking longer and longer?
Jim
On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
This is what top is showing now (the import is still running):
top - 13:56:19 up 1 day, 1:09, 3 users, load average: 1.18, 1.11, 1.07
Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
%Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 4045880 total, 123192 free, 3053072 used, 869616 buff/cache
KiB Swap: 3905532 total, 3753980 free, 151552 used. 688136 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2 1227:41 mysqld
3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2 139:05.67 java
3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3 176:26.68 java
22466 root 20 0 0 0 0 S 0.7 0.0 0:00.73 kworker/u4:1
That is running on 2-CPU machine with 4GB RAM.
Jim
On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, like top.
On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi Tomas,
I'm sorry that I missed the message below, but I am not clear what you are asking me to check/monitor? Do you mean like running something like "top" command?
Thanks,Jim
On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
To verify that the changes actually happen...
Do you monitor CPU usage to see what processes use CPU and memory?
On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
The import has been running for about an hour now and has imported 17836 entries.
Per what I had said previously, this CRL file has 1002255entries, that means it would take about 56+ hours to import this CRL.
So, this import processing is still way too slow.
I don't know what is involved, or even how much it would improve things, but I think that the next thing to try is to eliminated both the check for the serial number in the DB AND the dummy cert creation.
Jim
On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
Hi,
Ok, I rebuilt the JAR using your Java class, and it is running now, and creating rows in the CertificateData, but it still SEEMS kind of slow.
I am running now and will try to get a measurement of revoked entries/time and will post back.
Jim
On Tuesday, July 16, 2019, 5:25:32 PM UTC, oh...@ya... <oh...@ya...> wrote:
Hi,
So I just tried what I said (emptying the "if") and you are right, it doesn't create the rows in the CeritificateData table :(...
Ok, I will try your version...
Jim
On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas Gustavsson <to...@pr...> wrote:
Hi,
I don't think so, no quite actually. In order to use the API to populate
the database you need the certificate object it seems. Something like
the attached file should work, it still creates a dummy certificate, but
uses a static hard coded key, so no key generation.
The file assumes you are on 6.15.x.
Regards,
Tomas
On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
> For testing (only...), I am thinking that if I modify this If clause in
> CaImportCrlCommand.java:
>
> if (adaptive)
> {
> Date time = new Date();
> KeyPair key_pair = KeyTools.genKeys("2048", "RSA");
>
> SubjectPublicKeyInfo pkinfo =
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
> X500Name dnName = new X500Name("CN=Dummy Missing in
> Imported CRL, serialNumber=" + serialHex);
> Date notAfter = new Date(time.getTime() + 315360000000L);
> X509v3CertificateBuilder certbuilder = new
> X509v3CertificateBuilder(X500Name.getInstance(cacert
> .getSubjectX500Principal().getEncoded()), serialNr,
> time, notAfter, dnName, pkinfo);
> ContentSigner signer = new BufferingContentSigner(new
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
> .getPrivate()), 20480);
> X509CertificateHolder certHolder = certbuilder.build(signer);
> X509Certificate certificate =
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
> X509Certificate.class);
>
> String fingerprint =
> CertTools.getFingerprintAsString(certificate);
>
> EndEntityInformation missingUserEndEntityInformation =
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
> missing_user_name);
> if (missingUserEndEntityInformation == null)
> {
> log.debug("Loading/updating user " + missing_user_name);
>
> EndEntityInformation userdataNew = new
> EndEntityInformation(missing_user_name,
> CertTools.getSubjectDN(certificate), cainfo.getCAId(), null, null, 10,
> new EndEntityType(EndEntityTypes.ENDUSER), 1, 1, null, null, 1, 0, null);
>
> userdataNew.setPassword("foo123");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
> userdataNew, false);
>
> log.info("User '" + missing_user_name + "' has been
> added.");
>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
> missing_user_name, 50);
>
> log.info("User '" + missing_user_name + "' has been
> updated.");
> }
>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>
> EJBTools.wrap(certificate), missing_user_name,
> fingerprint, 20, 1, 1, 0, null, new Date()
>
> .getTime());
> log.info("Dummy certificate '" + serialHex + "' has been
> stored.");
> }
>
> to (for example):
>
> if (adaptive)
> {
>
> log.info("*** SKIPPING *** Dummy certificate '" +
> serialHex + "' has been stored.");
> }
>
>
> Would THAT work??
>
> Thanks,
> Jim
>
>
>
>
> On Tuesday, July 16, 2019, 10:59:31 AM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
> Hi Tomas (or anyone else),
>
> Sorry - I was typing my last msg on phone, and it wasn't as detailed or
> clear as I wanted.
>
> Basically what I was saying is I was thinking that, as a quick sanity
> test, if there was one or two places where the code in
> the CaImportCRLCommand.java was maybe making a method call to create the
> dummy cert and put it into the database, then maybe I could decompile
> and modify the CaImportCRLCommand.java, and comment out those method
> calls, then, re-do my large CRL import test and time it?
>
> If so, can you let me know specifically where those calls are, and if I
> just comment them out, would that work?
>
> Ironically, that (not doing the dummy cert creation and writing) was
> what I *thought* the "LENIENT" option was doing).
>
> Thanks,
> Jim
>
> P. S. I have time today to try this if I figure out what needs to be done.
>
>
>
> On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
>
>
> Yes, we are definitely interested/willing to test patches. I actually
> was wondering if it would be a quick test to mod the code somewhere to
> avoid a call that creates the dummy cert, to get an idea if that would
> reduce the import time enough??
>
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>
> On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
> <to...@pr...> wrote:
> Most large users, some very very large, typically uses on line
> updates directly from the CA. Many CAs today don't accept only
> importing CRLs, as they need to keep track of issued certificates as
> well not only revoked (can't use "non existing is good").
> Having said thst, it's interesting to be able to import large CRLs,
> so improving it is of interest.
>
> Importing CRLs would actually not have to generate a dummy cert at
> all, since a certificate object is not needed on the responder,
> which should make it very fast.
>
> If you are willing to test patches, we can work with you on improving.
>
> Regards,
> Tomas
>
>
> On July 15, 2019 5:26:08 PM GMT+02:00, ohaya--- via Ejbca-develop
> <ejb...@li...> wrote:
>
> Hi,
>
> I am curious whether you have other users that import large CRLs
> like this?
>
> Are we the only ones who would encounter these really long
> import times, because the other users who might have large CRLs
> use the EJBCA CA also, so they don't run into having the dummy
> certs being created for every entry in the CRL?
>
> Jim
>
> On Monday, July 15, 2019, 1:42:57 PM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> I see...
>
> I can see that a new key pair is being generated for each dummy
> cert.
>
> In CaImportCRLCommand.java:
>
> final KeyPair key_pair = KeyTools.genKeys("2048",
> AlgorithmConstants.KEYALGORITHM_RSA);
>
> Chaning that to 1024 or 512 would be an easy improvement. A bit more
> complex, but even larger speedup would be to use a "hard coded"
> dummy
> key, i.e. same key for all dummy certs.
>
> You can experiment by changing 2048 to 512 (not secure but it
> doesn't
> have to be for this dummy cert).
>
> Regards,
> Tomas
>
> On 2019-07-11 14:19, ohaya--- via Ejbca-develop wrote:
> > FYI, the is running, but it is REALLY slow. The CRL file has
> 1002255
> > entries in it, and the import is only up to 8899 now, after
> about 40
> > minutes... which would take about 75 hours!!
> >
> > In our production systems, we are required to update some CRLs
> every
> > hour, so that would be a problem.
> >
> > What can I do to speed that up??
> >
> > Thanks,
> > Jim
> >
> >
> > On Thursday, July 11, 2019, 11:09:33 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > AHHH... Ok, it is running now, and from checking the
> CertificateData in
> > the Mysql database, I can see that it is now populating the
> > CertificateData!!
> >
> > Thanks!
> >
> > Jim
> >
> >
> >
> > On Thursday, July 11, 2019, 11:07:22 AM UTC, ohaya--- via
> Ejbca-develop
> > <ejb...@li...
> <mailto:ejb...@li...>> wrote:
> >
> >
> > I wasn't sure if we should use ADAPTIVE, since we are not
> using the CA part.
> >
> >
> > I will try with "-o ADAPTIVE" now.
> >
> >
> >
> >
> > On Thursday, July 11, 2019, 7:42:14 AM UTC, Tomas Gustavsson
> > <to...@pr... <mailto:to...@pr...>> wrote:
> >
> >
> >
> > It should be ADAPTIVE.
> >
> > The documentation says:
> > bin/ejbca.sh ca importcrl --help
> > ---
> > ADAPTIVE means that missing certficates will be replaced by dummy
> > certificates to cater for proper CRLs for missing certificates.
> > ---
> >
> > Regards,
> > Tomas
> >
> > On 2019-07-11 08:05, ohaya--- via Ejbca-develop wrote:
> >> Hi,
> >>
> >> I've been looking at the code in CaImportCRLCommand, and I am
> wondering
> >> if, when the command is run in LENIENT mode and it doesn't
> find the cert
> >> in the database (and outputs the "missing in database" error
> message),
> >> what is supposed to happen?
> >>
> >> Mind you, the only error messages I am seeing are the "missing in
> >> database" error messages. I don't see any of the other error
> messages
> >> that are in/from CaImportCRLCommand.java.
> >>
> >> I am wondering if, when the command is run with "-o LENIENT",
> and when
> >> it outputs the "missing in database" error message, then the
> ejbca.sh is
> >> basically not doing anything (and not storing anything into the
> >> CertificateData table)?
> >>
> >> From the lack of any other error messages (vs. what I see in the
> >> CaImportCRLCommand.java code), that is what it looks like is
> happening.
> >>
> >> So given I am using EJBCA for OCSP responder only, is running the
> >> ejbca.sh in the "-o LENIENT" mode correct?
> >>
> >> Jim
> >>
> >>
> >>
> >> On Thursday, July 11, 2019, 5:43:37 AM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I've been doing a bunch of testing with trying to import the
> large CRL,
> >> and wanted to summarize:
> >>
> >> - The ejbca.sh runs fine - it completes without outputting an
> errors,
> >> except for outputting a bunch of the "cert xxxx missing in
> database"
> >> messages.
> >>
> >> - When the import completes, and I check the Mysql database,
> the crlData
> >> (the full PEM of the CRL) is in the database, but there is no
> rows in
> >> the CertificateData table for that CA at all.
> >>
> >> - I even figured out how to get the ejbca.sh to log using
> >> log4j.properties, and it output a bunch of logging, mainly
> the missing
> >> in database errors, but other than that I didn't see any
> obvious errors.
> >>
> >>
> >> So, bottom line is that even though the ejbca.sh seems to run
> without
> >> errors (other than the missing in database errors), the import is
> >> causing the CRL PEM to be stored in the database, but the
> >> CertificateData table is not being populated at all for that CA.
> >>
> >>
> >> Jim
> >>
> >>
> >>
> >> On Wednesday, July 10, 2019, 7:40:16 PM UTC, ohaya--- via
> Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> Ok, I found Tomas' original message... it was in my Spam
> folder....
> >>
> >> I also tried re-running the "./ejbca.sh ca importcrl" command
> a bunch of
> >> ways. I think it is running without outputting any errors,
> but still,
> >> no rows are appearing with that issuerDN at all.
> >>
> >>
> >>
> >>
> >> On Tuesday, July 9, 2019, 10:45:01 AM UTC, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> OCSP responses does not have anything to do with the CRL
> itself. Once
> >> the CRL has been imported each certificate status is in its
> own record
> >> in the CertificateData table. This is where status is read
> when reponses
> >> are created.
> >> If there are issues, it is in that case related to the import
> of the CRL.
> >>
> >> Verify that certificate records are created in
> CertificateData when
> >> importing the CRL.
> >> Also, a debug log when making OCSP request will tell exact
> details of
> >> what the responder finds in the database or not.
> >>
> >> Regards,
> >> Tomas
> >>
> >>
> >> On July 8, 2019 4:59:06 PM GMT+02:00, ohaya--- via Ejbca-develop
> >> <ejb...@li...
> <mailto:ejb...@li...>
> > <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
> >>
> >> [Re-sending, as this (handling of large CRLs) is still a
> problem and
> >> is still not working.]
> >>
> >>
> >>
> >> On Monday, July 1, 2019, 12:47:23 PM UTC, <oh...@ya...
> <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> I think that I've found how to set/change the
> max_allowed_packet on
> >> the client (i.e., EJBCA) side, by editing the Mysql URL in the
> >> standalone/configuration/standalone.xml to:
> >>
> >>
> >
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
> >>
> >> However, when I test the OCSP Responder with a serial
> number (and CA
> >> cert) from the CA with the large (>71MB) CRL file, I am still
> >> getting "unknown" and EJBCA OCSP Responder is still NOT
> FINDING the
> >> cert serial number.
> >>
> >> NOTE, again, that I can test with certs (and CA cert) that
> have
> >> smaller CRL files, and those tests work.
> >>
> >> So it is only when testing with the large CRL file that is
> failing
> >> with the OCSP Responder.
> >>
> >> This seems like a potential BUG for EJBCA OCSP Responder?
> >>
> >> Please advise.
> >>
> >> Thanks,
> >> Jim
> >> On Saturday, June 29, 2019, 3:36:18 PM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi,
> >>
> >> BTW, just to check, I have actually gone through the
> process of
> >> getting the crlData from the MySQL using mysql client (I
> had to add
> >> the --max_allowed-packet=100M to the mysql command) and I
> was able
> >> to actually get the PEM from the Mysql database that way.
> >>
> >> So, I KNOW that the entire CRL is in the Mysql database,
> and yet
> >> when I run an "openssl ocsp" test against the EJBCA OCSP
> Responder,
> >> it says it cannot find the revocation.
> >>
> >> OCSP Request Data:
> >> Version: 1 (0x0)
> >> Requestor List:
> >> Certificate ID:
> >> Hash Algorithm: sha1
> >> Issuer Name Hash: 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
> >> Issuer Key Hash: 3929A80864E28F83AA816147119133F85A37A6E4
> >> Serial Number: 2FFFF4
> >> Request Extensions:
> >> OCSP Nonce:
> >> 04103054A60A55E389B239A55228033D8F62
> >> Response verify OK
> >> 0x2FFFF4: unknown
> >> This Update: Jun 29 19:31:02 2019 GMT
> >>
> >>
> >> In the server log:
> >>
> >> 2019-06-29 15:31:02,150 DEBUG
> >>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
> >> (default task-1) Unable to find revocation information for
> >> certificate with serial '2ffff4' from issuer 'CN=XXX....'
> >>
> >>
> >> Honestly, I am no 100% sure that the problem is that
> >> max_allowe_packet is too small on the mysql client side,
> but that is
> >> my best guess.
> >>
> >> What I AM sure is that the CRL is in mysql, but when I do
> a test
> >> with OCSP Responder, it cannot find the cert serial number
> in the CRL.
> >>
> >> Jim
> >>
> >>
> >> On Saturday, June 29, 2019, 4:52:20 AM EDT,
> <oh...@ya... <mailto:oh...@ya...>
> > <mailto:oh...@ya... <mailto:oh...@ya...>>> wrote:
> >>
> >>
> >> Hi Tomas,
> >>
> >> I already had the max_allowed_packet configured on the Mysql
> >> server-side (in /etc/my.cnf).
> >>
> >> So as I said, when testing with mysql (client) I had to
> include
> >> "--max_allowed_packet=100M" on the mysql (client) command
> line, e.g.:
> >>
> >> mysql -u root -p --max_allowed_packet=100M
> >>
> >> If I did NOT include the "--max_allowed_packet=100M" on
> that command
> >> line, then when I did a "select" on the field base64Crl, I
> would get
> >> an error.
> >>
> >> I suspect that EJBCA is acting as a Mysql client, so, in
> order to
> >> work with such large CRLs, there needs to be some way to
> set the
> >> equivalent of the "--max_allowed_packet=100M" on the EJBCA
> side.
> >>
> >> How can I do that?
> >>
> >> If that can't be done, then I think that when EJBCA tries to
> >> retrieve the CRL from the Mysql, it (EJBCA) will see the
> same error.
> >>
> >> Thanks,
> >> Jim
> >>
> >> On Saturday, June 29, 2019, 3:25:47 AM EDT, Tomas Gustavsson
> >> <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>> wrote:
> >>
> >>
> >>
> >> You configure max allowed packet size in you MySQL
> configuration on the
> >> MySQL server.
> >>
> >> /Tomas
> >>
> >> On 2019-06-29 02:19, o haya via Ejbca-develop wrote:
> >> > I am not 100% sure, but I think I know what might be
> causing this
> >> problem.
> >> >
> >> > I did a test, where I just tried to log into mysql and
> do the
> >> following
> >> > select:
> >> >
> >> > select base64Crl INTO OUTFILE
> '/var/lib/mysql-files/xx.out' from
> >> CRLData
> >> > where cRLNumber=3167;
> >> >
> >> > and I got an error saying that the result was more
> >> than max_allowed_packet.
> >> >
> >> > So then I did the same thing, but for the mysql command
> I used:
> >> >
> >> > mysql -u root -p --max_allowed_packet=100M
> >> >
> >> > and then when I did that same select, I was able to
> output the entire
> >> > CRL as a PEM.
> >> >
> >> > So, I am theorizing that that is what is happening with
> EJBCA, i.e.,
> >> > when I run the "openssl ocsp" command, EJBCA is querying the
> > base64CRL
> >> > but the result is too large and so that is causing the
> query to fail.
> >> >
> >> > So the question now is: How can I cause EJBCA OCSP
> Responder to run
> >> > with a larger max_allowed_packet so that the query
> succeeds (and then
> >> > hopefully EJBCA OCSP Responder will be able to check the
> revocation
> >> > properly)?
> >> >
> >> > Does anyone know how that can be done?
> >> >
> >> > Thanks,
> >> > Jim
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Friday, June 28, 2019, 07:16:39 PM UTC, o haya
> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
> >> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>> wrote:
> >> >
> >> >
> >> > Hi,
> >> >
> >> > As discussed in an earlier thread, it looks like I was
> able to
> >> import a
> >> > large (> 71MB) CRL file into EJBCA (as an OCSP
> Responder), so now,
> >> I am
> >> > doing testing, using "openssl ocsp" to send requests
> into the
> >> EJBCA OCSP
> >> > Responder:
> >> >
> >> > openssl ocsp -issuer <CA_CERT> -VAfile <SIGNING_CERT>
> >> -req_text -serial
> >> > 0x<CERT_SERIAL> -host <HOST:PORT>
> >> >
> >> > But it looks like no matter what serial number I use,
> whether the
> >> serial
> >> > number is in the CRL or not, returns a response like:
> >> >
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash: 37...00
> >> > Issuer Key Hash: 39...E4
> >> > Serial Number: 2FFFF9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 041078B1311984A110B5E19C8DC4895F485B
> >> > Response verify OK
> >> > 0x2FFFF9: unknown
> >> > This Update: Jun 28 18:56:20 2019 GMT
> >> >
> >> > i.e., it looks like the EJBCA OCSP Responder is not able
> to actually
> >> > find any revoked certs in the CRL?
> >> >
> >> >
> >> > FYI, I *KNOW* that the EJBCA OCSP Responder is functioning
> >> properly, in
> >> > general, because I can do the same test against a
> smaller CRL in the
> >> > same EJBCA OCSP Responder, for CA simpleca, and I can
> get an actual
> >> > "Revoked" response:
> >> >
> >> > [root@ejbca <mailto:root@ejbca> <mailto:root@ejbca
> <mailto:root@ejbca>> <mailto:root@ejbca <mailto:root@ejbca>
> > <mailto:root@ejbca <mailto:root@ejbca>>> testocspresponder]#
> openssl ocsp
> >> -issuer simpleca.crt
> >> > -VAfile simplecabinding2.crt -serial
> 0x008486394c03e1f5d9 -req_text
> >> > -host 192.168.0.28:80
> >> > OCSP Request Data:
> >> > Version: 1 (0x0)
> >> > Requestor List:
> >> > Certificate ID:
> >> > Hash Algorithm: sha1
> >> > Issuer Name Hash:
> 0C16107310427EA4ADB3C6436915CE44A15FFE55
> >> > Issuer Key Hash:
> E2533BF85F8C7CA60A411BF5458B2DC3B5232B6E
> >> > Serial Number: 8486394C03E1F5D9
> >> > Request Extensions:
> >> > OCSP Nonce:
> >> > 04109A09CC1C56D26AA73CB4C3FB1EC7CF1F
> >> > Response verify OK
> >> > 0x008486394c03e1f5d9: revoked
> >> > This Update: Jun 28 19:12:48 2019 GMT
> >> > Reason: unspecified
> >> > Revocation Time: May 26 12:30:44 2019 GMT
> >> >
> >> > So it seems like the EJBCA OCSP Responder is either not
> > processing the
> >> > large CRL properly, or maybe the import didn't import
> the large
> >> CRL into
> >> > the OCSP Responder correctly.
> >> >
> >> > Can anyone tell me how do I diagnose this problem?
> >> >
> >> > Thanks,
> >> > Jim
> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > Ejbca-develop mailing list
> >> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> >
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> <mailto:Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>>
> >
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >>
> >> _______________________________________________
> >> Ejbca-develop mailing list
> >> Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> >> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >>
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > <mailto:Ejb...@li...
> <mailto:Ejb...@li...>>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
> >
> > _______________________________________________
> > Ejbca-develop mailing list
> > Ejb...@li...
> <mailto:Ejb...@li...>
> > https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> >
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> <mailto:Ejb...@li...>
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
_______________________________________________
Ejbca-develop mailing list
Ejb...@li...
https://lists.sourceforge.net/lists/listinfo/ejbca-develop
|
106 messages has been excluded from this view by a project administrator.
