Menu
▾
▴
Re: [Ejbca-develop] EJBCA OCSP Responder does not appear to
|
From: <oh...@ya...> - 2019-07-19 08:07:23
|
Hi Tomas,
No, I am not using Wildfly, but rather JBOSS... specifically "jboss-eap-7.2".
So I restored my snapshot from just after I had deleted all the rows in CertificateData that were created before.
And it looks like all the tables in the "ejbca" database are MyISAM.
So I found a one-liner that will convert them all to InnoDB:
mysql -u root -p ejbca -e "show table status where Engine='MyISAM';" | awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql -u root -p ejbca
Should I run that and convert all the tables in the "ejbca" to InnoDB???
And then try to run the create indexes SQL script?
Thanks for all your help!
Jim
On Friday, July 19, 2019, 8:01:15 AM UTC, Tomas Gustavsson <to...@pr...> wrote:
Yes, MyISAM is horrible and can not be used.
I'm not sure about the exact commands to change it. But I think there
are good internet guides.
The reason you have MyISAM is probably because you are using WildFly 14
(are you?). There was a change in Hibernate that caused this issue with
automatic database creation.
One way to modify is to mysqldump the database, change enging in the sql
file and restore it.
Regards,
Tomas
On 2019-07-19 09:19, ohaya--- via Ejbca-develop wrote:
> Hi,
>
> Mysql version: Server version: 5.7.26 MySQL Community Server (GPL)
>
>
>
>
> mysql> show create table CertificateData;
>
>
>
> | CertificateData | CREATE TABLE `CertificateData` (
> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `certificateProfileId` int(11) NOT NULL,
> `endEntityProfileId` int(11) DEFAULT NULL,
> `expireDate` bigint(20) NOT NULL,
> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `notBefore` bigint(20) DEFAULT NULL,
> `revocationDate` bigint(20) NOT NULL,
> `revocationReason` int(11) NOT NULL,
> `rowVersion` int(11) NOT NULL,
> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `status` int(11) NOT NULL,
> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `type` int(11) NOT NULL,
> `updateTime` bigint(20) NOT NULL,
> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `base64Cert` longtext,
> `rowProtection` longtext,
> PRIMARY KEY (`fingerprint`),
> KEY `certificatedata_idx2` (`username`)
> ) ENGINE=MyISAM DEFAULT CHARSET=utf8 |
>
>
>
> AH! Yes mine is showing as MyISAM and not InnoDB!
>
>
>
> Do I need to change the engine to InnoDB? And, if so, how do I do that?
>
> Do I need to do that for every table in the ejbca database?
>
>
> Thanks,
> Jim
>
>
>
>
> On Friday, July 19, 2019, 6:20:23 AM UTC, Tomas Gustavsson
> <to...@pr...> wrote:
>
>
>
> That looks about right. What version of database are you using? I just
> tried the index command (on line 23 in the SQL file) myself and it works.
>
> CREATE INDEX certificatedata_idx4 ON CertificateData (subjectDN);
>
>
> What does your database creation say?
>
>>show create table CertiifcateData;
>
> It should look something like this:
>
> | CertificateData | CREATE TABLE `CertificateData` (
> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `certificateProfileId` int(11) NOT NULL,
> `crlPartitionIndex` int(11) DEFAULT NULL,
> `endEntityProfileId` int(11) DEFAULT NULL,
> `expireDate` bigint(20) NOT NULL,
> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `notBefore` bigint(20) DEFAULT NULL,
> `revocationDate` bigint(20) NOT NULL,
> `revocationReason` int(11) NOT NULL,
> `rowVersion` int(11) NOT NULL,
> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `status` int(11) NOT NULL,
> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
> DEFAULT NULL,
> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `type` int(11) NOT NULL,
> `updateTime` bigint(20) NOT NULL,
> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
> `base64Cert` longtext,
> `certificateRequest` longtext,
> `rowProtection` longtext,
> PRIMARY KEY (`fingerprint`)
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
>
> Is the database engine wrong? I.e. not InnoDB?
>
> I am importing the CRL right now and processing about 27 records per
> second on my laptop.
>
> Regards,
> Tomas
>
>
> On 2019-07-18 18:09, ohaya--- via Ejbca-develop wrote:
>> According to my notes I used these commands:
>>
>> |*/CREATE DATABASE ejbca CHARACTER SET utf8 COLLATE utf8_general_ci;/*|
>>
>> |*/ /*|
>>
>> |*/GRANT ALL PRIVILEGES ON ejbca.* TO 'ejbca'@'localhost' IDENTIFIED BY
>> 'ejbca';/*|
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 3:37:51 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> What command did you use to create the database?
>>
>> On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> This is really getting frustrating!!!
>>
>> [root@ejbca <mailto:root@ejbca> sql-scripts]# mysql -u root
> -p'xxxxxx' ejbca <
>> create-index-ejbca.sql
>> mysql: [Warning] Using a password on the command line interface can
>> be insecure.
>> ERROR 1071 (42000) at line 23: Specified key was too long; max key
>> length is 1000 bytes
>>
>> Jim
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> The "machine" is a virtual machine under VirtualBox on a machine I
>> have that has 64GB physical RAM. Drives are all 7200 RPM drives.
>>
>> I will restore the snapshot, then increase the memory and # of
>> processors and try to run the create index again.
>>
>> Jim
>>
>> On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Hi,
>>
>> Just run create index script.
>>
>> Create index should run pretty fast even with millions of records in
>> the database. I suspect that your database is configured very slow,
>> perhaps running everything on disl, without RAM, using slow disks?
>>
>> A standard installation of mysql/mariadb on a centos machine should
>> run faster. Then you have pretty low amount of ram for running
>> everything on one box, handing millions of records.
>>
>> Regards,
>> Tomas
>>
>> On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> I had taken a snapshot of the machine before I tried to do the
>> index earlier, so I restored that snapshot.
>>
>> Then, I tried running the "optimize-ejbca-mysql.sql" and I got a
>> different/new error:
>>
>> ERROR 1091 (42000) at line 27: Can't DROP
>> 'certificatedata_idx5'; check that column/key exists
>>
>> Jim
>>
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> Oh oh, I just got this from the mysql execution:
>>
>> ERROR 1071 (42000) at line 23: Specified key was too long; max
>> key length is 1000 bytes
>>
>> Jim
>>
>>
>>
>>
>>
>>
>> On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> FYI, I am running the create-index-ejbca.sql script now, but
>> that seems to be taking FOREVER?
>>
>> I had stopped the earlier import, and then deleted all the rows
>> for that CA DN in CertificateData table, and then bounced the
>> machine, then I ran the create-index-ejbca.sql.
>>
>> Why is it taking so long? Should I have done something in
>> addition to deleting the rows? I am guessing that the reason
>> the sql script is taking so long is that it's somehow still
>> trying to index the rows were deleted or something like that?
>>
>> Thanks,
>> Jim
>>
>>
>>
>> On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
> <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> I notice that there is:
>>
>> - create-index-ejbca.sql
>> - optimize-ejbca-mysql.sql
>>
>> Should I run BOTH of them, or just the "create-index-ejbca.sql "?
>>
>> Thanks,
>> Jim
>>
>>
>> On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
>> <to...@pr... <mailto:to...@pr...>> wrote:
>>
>>
>> Yes, i could see from your output from top that mysql takes all
>> CPU. This indicates that you don't have any indexes in your
>> database, which makes it horribly slow as numbers grow.
>>
>> Apply database indexes from doc/sql-scripts.
>>
>> Regards,
>> Tomas
>>
>>
>> On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> So far, processed only 185706 entries, and it looks like the
>> processing is going to a crawl (really slow now).
>>
>> Which kind of makes me wonder if maybe part of reason for
>> the slowness is that it is having to search the certificate
>> database for the serial number, and now, as there are more
>> dummy certs in the database, that search is taking longer
>> and longer?
>>
>> Jim
>>
>>
>> On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> This is what top is showing now (the import is still running):
>>
>> top - 13:56:19 up 1 day, 1:09, 3 users, load average:
>> 1.18, 1.11, 1.07
>> Tasks: 167 total, 2 running, 165 sleeping, 0 stopped,
>> 0 zombie
>> %Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa, 0.0
>> hi, 0.3 si, 0.0 st
>> KiB Mem : 4045880 total, 123192 free, 3053072 used,
>> 869616 buff/cache
>> KiB Swap: 3905532 total, 3753980 free, 151552 used.
>> 688136 avail Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU
>> %MEM TIME+ COMMAND
>> 1489 mysql 20 0 1580320 129624 4396 S 92.7 3.2
>> 1227:41 mysqld
>> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2
>> 139:05.67 java
>> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3
>> 176:26.68 java
>> 22466 root 20 0 0 0 0 S 0.7 0.0
>> 0:00.73 kworker/u4:1
>>
>> That is running on 2-CPU machine with 4GB RAM.
>>
>> Jim
>>
>>
>>
>>
>> On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas
>> Gustavsson <to...@pr... <mailto:to...@pr...>>
> wrote:
>>
>>
>> Yes, like top.
>>
>>
>> On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi Tomas,
>>
>> I'm sorry that I missed the message below, but I am not
>> clear what you are asking me to check/monitor? Do you
>> mean like running something like "top" command?
>>
>> Thanks,
>> Jim
>>
>> On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas
>> Gustavsson <to...@pr...
> <mailto:to...@pr...>> wrote:
>>
>>
>> To verify that the changes actually happen...
>>
>> Do you monitor CPU usage to see what processes use CPU
>> and memory?
>>
>>
>> On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via
>> Ejbca-develop <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>> Hi,
>>
>> The import has been running for about an hour now
>> and has imported 17836 entries.
>>
>> Per what I had said previously, this CRL file has
>> 1002255 entries, that means it would take about 56+
>> hours to import this CRL.
>>
>> So, this import processing is still way too slow.
>>
>> I don't know what is involved, or even how much it
>> would improve things, but I think that the next
>> thing to try is to eliminated both the check for the
>> serial number in the DB AND the dummy cert creation.
>>
>> Jim
>>
>>
>> On Tuesday, July 16, 2019, 5:54:54 PM UTC, ohaya---
>> via Ejbca-develop
>> <ejb...@li...
> <mailto:ejb...@li...>> wrote:
>>
>>
>> Hi,
>>
>> Ok, I rebuilt the JAR using your Java class, and it
>> is running now, and creating rows in the
>> CertificateData, but it still SEEMS kind of slow.
>>
>> I am running now and will try to get a measurement
>> of revoked entries/time and will post back.
>>
>> Jim
>>
>>
>>
>>
>> On Tuesday, July 16, 2019, 5:25:32 PM UTC,
>> oh...@ya... <mailto:oh...@ya...>
> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>
>>
>> Hi,
>>
>> So I just tried what I said (emptying the "if") and
>> you are right, it doesn't create the rows in the
>> CeritificateData table :(...
>>
>> Ok, I will try your version...
>>
>> Jim
>>
>> On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas
>> Gustavsson <to...@pr...
> <mailto:to...@pr...>> wrote:
>>
>>
>>
>> Hi,
>>
>> I don't think so, no quite actually. In order to use
>> the API to populate
>> the database you need the certificate object it
>> seems. Something like
>> the attached file should work, it still creates a
>> dummy certificate, but
>> uses a static hard coded key, so no key generation.
>>
>> The file assumes you are on 6.15.x.
>>
>> Regards,
>> Tomas
>>
>> On 2019-07-16 15:02, ohaya--- via Ejbca-develop wrote:
>> > For testing (only...), I am thinking that if I
>> modify this If clause in
>> > CaImportCrlCommand.java:
>> >
>> > if (adaptive)
>> > {
>> > Date time = new Date();
>> > KeyPair key_pair =
>> KeyTools.genKeys("2048", "RSA");
>> >
>> > SubjectPublicKeyInfo pkinfo =
>> >
>>
> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
>> > X500Name dnName = new
>> X500Name("CN=Dummy Missing in
>> > Imported CRL, serialNumber=" + serialHex);
>> > Date notAfter = new
>> Date(time.getTime() + 315360000000L);
>> > X509v3CertificateBuilder certbuilder
>> = new
>> > X509v3CertificateBuilder(X500Name.getInstance(cacert
>> >
>> .getSubjectX500Principal().getEncoded()), serialNr,
>> > time, notAfter, dnName, pkinfo);
>> > ContentSigner signer = new
>> BufferingContentSigner(new
>> >
>>
> JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
>> > .getPrivate()), 20480);
>> > X509CertificateHolder certHolder =
>> certbuilder.build(signer);
>> > X509Certificate certificate =
>> >
>>
> (X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
>> > X509Certificate.class);
>> >
>> > String fingerprint =
>> > CertTools.getFingerprintAsString(certificate);
>> >
>> > EndEntityInformation
>> missingUserEndEntityInformation =
>> >
>>
> ((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
>> > missing_user_name);
>> > if (missingUserEndEntityInformation
>> == null)
>> > {
>> > log.debug("Loading/updating user "
>> + missing_user_name);
>> >
>> > EndEntityInformation userdataNew
> = new
>> > EndEntityInformation(missing_user_name,
>> > CertTools.getSubjectDN(certificate),
>> cainfo.getCAId(), null, null, 10,
>> > new EndEntityType(EndEntityTypes.ENDUSER), 1, 1,
>> null, null, 1, 0, null);
>> >
>> > userdataNew.setPassword("foo123");
>> >
>> >
>>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
>> > userdataNew, false);
>> >
>> > log.info("User '" +
>> missing_user_name + "' has been
>> > added.");
>> >
>> >
>>
> ((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
>> > missing_user_name, 50);
>> >
>> > log.info("User '" +
>> missing_user_name + "' has been
>> > updated.");
>> > }
>> >
>> >
>>
> ((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>> >
>> > EJBTools.wrap(certificate),
>> missing_user_name,
>> > fingerprint, 20, 1, 1, 0, null, new Date()
>> >
>> > .getTime());
>> > log.info("Dummy certificate '" +
>> serialHex + "' has been
>> > stored.");
>> > }
>> >
>> > to (for example):
>> >
>> > if (adaptive)
>> > {
>> >
>> > log.info("*** SKIPPING *** Dummy
>> certificate '" +
>> > serialHex + "' has been stored.");
>> > }
>> >
>> >
>> > Would THAT work??
>> >
>> > Thanks,
>> > Jim
>> >
>> >
>> >
>> >
>> > On Tuesday, July 16, 2019, 10:59:31 AM UTC,
>> ohaya--- via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> >
>> > Hi Tomas (or anyone else),
>> >
>> > Sorry - I was typing my last msg on phone, and it
>> wasn't as detailed or
>> > clear as I wanted.
>> >
>> > Basically what I was saying is I was thinking
>> that, as a quick sanity
>> > test, if there was one or two places where the
> code in
>> > the CaImportCRLCommand.java was maybe making a
>> method call to create the
>> > dummy cert and put it into the database, then
>> maybe I could decompile
>> > and modify the CaImportCRLCommand.java, and
>> comment out those method
>> > calls, then, re-do my large CRL import test and
>> time it?
>> >
>> > If so, can you let me know specifically where
>> those calls are, and if I
>> > just comment them out, would that work?
>> >
>> > Ironically, that (not doing the dummy cert
>> creation and writing) was
>> > what I *thought* the "LENIENT" option was doing).
>> >
>> > Thanks,
>> > Jim
>> >
>> > P. S. I have time today to try this if I figure
>> out what needs to be done.
>> >
>> >
>> >
>> > On Monday, July 15, 2019, 6:24:53 PM UTC, ohaya---
>> via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> >
>> >
>> > Yes, we are definitely interested/willing to test
>> patches. I actually
>> > was wondering if it would be a quick test to mod
>> the code somewhere to
>> > avoid a call that creates the dummy cert, to get
>> an idea if that would
>> > reduce the import time enough??
>> >
>> >
>> > Sent from Yahoo Mail on Android
>> >
>>
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>> >
>> > On Mon, Jul 15, 2019 at 12:10 PM, Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>
> <mailto:to...@pr... <mailto:to...@pr...>>>
>> wrote:
>> > Most large users, some very very large,
>> typically uses on line
>> > updates directly from the CA. Many CAs today
>> don't accept only
>> > importing CRLs, as they need to keep track of
>> issued certificates as
>> > well not only revoked (can't use "non existing
>> is good").
>> > Having said thst, it's interesting to be able
>> to import large CRLs,
>> > so improving it is of interest.
>> >
>> > Importing CRLs would actually not have to
>> generate a dummy cert at
>> > all, since a certificate object is not needed
>> on the responder,
>> > which should make it very fast.
>> >
>> > If you are willing to test patches, we can work
>> with you on improving.
>> >
>> > Regards,
>> > Tomas
>> >
>> >
>> > On July 15, 2019 5:26:08 PM GMT+02:00, ohaya---
>> via Ejbca-develop
>> > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>> wrote:
>> >
>> > Hi,
>> >
>> > I am curious whether you have other users
>> that import large CRLs
>> > like this?
>> >
>> > Are we the only ones who would encounter
>> these really long
>> > import times, because the other users who
>> might have large CRLs
>> > use the EJBCA CA also, so they don't run
>> into having the dummy
>> > certs being created for every entry in the
> CRL?
>> >
>> > Jim
>> >
>> > On Monday, July 15, 2019, 1:42:57 PM UTC,
>> Tomas Gustavsson
>> > <to...@pr... <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>> wrote:
>> >
>> >
>> >
>> > I see...
>> >
>> > I can see that a new key pair is being
>> generated for each dummy
>> > cert.
>> >
>> > In CaImportCRLCommand.java:
>> >
>> > �� final KeyPair key_pair =
>> KeyTools.genKeys("2048",
>> > AlgorithmConstants.KEYALGORITHM_RSA);
>> >
>> > Chaning that to 1024 or 512 would be an
>> easy improvement. A bit more
>> > complex, but even larger speedup would be
>> to use a "hard coded"
>> > dummy
>> > key, i.e. same key for all dummy certs.
>> >
>> > You can experiment by changing 2048 to 512
>> (not secure but it
>> > doesn't
>> > have to be for this dummy cert).
>> >
>> > Regards,
>> > Tomas
>> >
>> > On 2019-07-11 14:19, ohaya--- via
>> Ejbca-develop wrote:
>> > > FYI, the is running, but it is REALLY
>> slow. The CRL file has
>> > 1002255
>> > > entries in it, and the import is only up
>> to 8899 now, after
>> > about 40
>> > > minutes... which would take about 75
> hours!!
>> > >
>> > > In our production systems, we are
>> required to update some CRLs
>> > every
>> > > hour, so that would be a problem.
>> > >
>> > > What can I do to speed that up??
>> > >
>> > > Thanks,
>> > > Jim
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 11:09:33 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
>> > >
>> > >
>> > > AHHH... Ok, it is running now, and from
>> checking the
>> > CertificateData in
>> > > the Mysql database, I can see that it is
>> now populating the
>> > > CertificateData!!
>> > >
>> > > Thanks!
>> > >
>> > > Jim
>> > >
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 11:07:22 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > > <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>> wrote:
>> > >
>> > >
>> > > I wasn't sure if we should use ADAPTIVE,
>> since we are not
>> > using the CA part.
>> > >
>> > >
>> > > I will try with "-o ADAPTIVE" now.
>> > >
>> > >
>> > >
>> > >
>> > > On Thursday, July 11, 2019, 7:42:14 AM
>> UTC, Tomas Gustavsson
>> > > <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>> wrote:
>> > >
>> > >
>> > >
>> > > It should be ADAPTIVE.
>> > >
>> > > The documentation says:
>> > > bin/ejbca.sh ca importcrl --help
>> > > ---
>> > > ADAPTIVE means that missing certficates
>> will be replaced by dummy
>> > > certificates to cater for proper CRLs for
>> missing certificates.
>> > > ---
>> > >
>> > > Regards,
>> > > Tomas
>> > >
>> > > On 2019-07-11 08:05, ohaya--- via
>> Ejbca-develop wrote:
>> > >> Hi,
>> > >>
>> > >> I've been looking at the code in
>> CaImportCRLCommand, and I am
>> > wondering
>> > >> if, when the command is run in LENIENT
>> mode and it doesn't
>> > find the cert
>> > >> in the database (and outputs the
>> "missing in database" error
>> > message),
>> > >> what is supposed to happen?
>> > >>
>> > >> Mind you, the only error messages I am
>> seeing are the "missing in
>> > >> database" error messages. I don't see
>> any of the other error
>> > messages
>> > >> that are in/from CaImportCRLCommand.java.
>> > >>
>> > >> I am wondering if, when the command is
>> run with "-o LENIENT",
>> > and when
>> > >> it outputs the "missing in database"
>> error message, then the
>> > ejbca.sh is
>> > >> basically not doing anything (and not
>> storing anything into the
>> > >> CertificateData table)?
>> > >>
>> > >> From the lack of any other error
>> messages (vs. what I see in the
>> > >> CaImportCRLCommand.java code), that is
>> what it looks like is
>> > happening.
>> > >>
>> > >> So given I am using EJBCA for OCSP
>> responder only, is running the
>> > >> ejbca.sh in the "-o LENIENT" mode correct?
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >>
>> > >> On Thursday, July 11, 2019, 5:43:37 AM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> I've been doing a bunch of testing with
>> trying to import the
>> > large CRL,
>> > >> and wanted to summarize:
>> > >>
>> > >> - The ejbca.sh runs fine - it completes
>> without outputting an
>> > errors,
>> > >> except for outputting a bunch of the
>> "cert xxxx missing in
>> > database"
>> > >> messages.
>> > >>
>> > >> - When the import completes, and I check
>> the Mysql database,
>> > the crlData
>> > >> (the full PEM of the CRL) is in the
>> database, but there is no
>> > rows in
>> > >> the CertificateData table for that CA at
>> all.
>> > >>
>> > >> - I even figured out how to get the
>> ejbca.sh to log using
>> > >> log4j.properties, and it output a bunch
>> of logging, mainly
>> > the missing
>> > >> in database errors, but other than that
>> I didn't see any
>> > obvious errors.
>> > >>
>> > >>
>> > >> So, bottom line is that even though the
>> ejbca.sh seems to run
>> > without
>> > >> errors (other than the missing in
>> database errors), the import is
>> > >> causing the CRL PEM to be stored in the
>> database, but the
>> > >> CertificateData table is not being
>> populated at all for that CA.
>> > >>
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >>
>> > >> On Wednesday, July 10, 2019, 7:40:16 PM
>> UTC, ohaya--- via
>> > Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> Ok, I found Tomas' original message...
>> it was in my Spam
>> > folder....
>> > >>
>> > >> I also tried re-running the "./ejbca.sh
>> ca importcrl" command
>> > a bunch of
>> > >> ways. I think it is running without
>> outputting any errors,
>> > but still,
>> > >> no rows are appearing with that issuerDN
>> at all.
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Tuesday, July 9, 2019, 10:45:01 AM
>> UTC, Tomas Gustavsson
>> > >> <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>> > <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> OCSP responses does not have anything to
>> do with the CRL
>> > itself. Once
>> > >> the CRL has been imported each
>> certificate status is in its
>> > own record
>> > >> in the CertificateData table. This is
>> where status is read
>> > when reponses
>> > >> are created.
>> > >> If there are issues, it is in that case
>> related to the import
>> > of the CRL.
>> > >>
>> > >> Verify that certificate records are
>> created in
>> > CertificateData when
>> > >> importing the CRL.
>> > >> Also, a debug log when making OCSP
>> request will tell exact
>> > details of
>> > >> what the responder finds in the database
>> or not.
>> > >>
>> > >> Regards,
>> > >> Tomas
>> > >>
>> > >>
>> > >> On July 8, 2019 4:59:06 PM GMT+02:00,
>> ohaya--- via Ejbca-develop
>> > >> <ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>
>> > >
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>
>> >
> <mailto:ejb...@li...
> <mailto:ejb...@li...>
>> <mailto:ejb...@li...
> <mailto:ejb...@li...>>>>> wrote:
>> > >>
>> > >> [Re-sending, as this (handling of
>> large CRLs) is still a
>> > problem and
>> > >> is still not working.]
>> > >>
>> > >>
>> > >>
>> > >> On Monday, July 1, 2019, 12:47:23 PM
>> UTC, <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> I think that I've found how to
>> set/change the
>> > max_allowed_packet on
>> > >> the client (i.e., EJBCA) side, by
>> editing the Mysql URL in the
>> > >>
>> standalone/configuration/standalone.xml to:
>> > >>
>> > >>
>> > >
>> >
>>
> <connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
>> > >>
>> > >> However, when I test the OCSP
>> Responder with a serial
>> > number (and CA
>> > >> cert) from the CA with the large
>> (>71MB) CRL file, I am still
>> > >> getting "unknown" and EJBCA OCSP
>> Responder is still NOT
>> > FINDING the
>> > >> cert serial number.
>> > >>
>> > >> NOTE, again, that I can test with
>> certs (and CA cert) that
>> > have
>> > >> smaller CRL files, and those tests work.
>> > >>
>> > >> So it is only when testing with the
>> large CRL file that is
>> > failing
>> > >> with the OCSP Responder.
>> > >>
>> > >> This seems like a potential BUG for
>> EJBCA OCSP Responder?
>> > >>
>> > >> Please advise.
>> > >>
>> > >> Thanks,
>> > >> Jim
>> > >> On Saturday, June 29, 2019, 3:36:18
>> PM EDT,
>> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi,
>> > >>
>> > >> BTW, just to check, I have actually
>> gone through the
>> > process of
>> > >> getting the crlData from the MySQL
>> using mysql client (I
>> > had to add
>> > >> the --max_allowed-packet=100M to the
>> mysql command) and I
>> > was able
>> > >> to actually get the PEM from the
>> Mysql database that way.
>> > >>
>> > >> So, I KNOW that the entire CRL is in
>> the Mysql database,
>> > and yet
>> > >> when I run an "openssl ocsp" test
>> against the EJBCA OCSP
>> > Responder,
>> > >> it says it cannot find the revocation.
>> > >>
>> > >> OCSP Request Data:
>> > >> Version: 1 (0x0)
>> > >> Requestor List:
>> > >> Certificate ID:
>> > >> Hash Algorithm: sha1
>> > >> Issuer Name Hash:
>> 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
>> > >> Issuer Key Hash:
>> 3929A80864E28F83AA816147119133F85A37A6E4
>> > >> Serial Number: 2FFFF4
>> > >> Request Extensions:
>> > >> OCSP Nonce:
>> > >> 04103054A60A55E389B239A55228033D8F62
>> > >> Response verify OK
>> > >> 0x2FFFF4: unknown
>> > >> This Update: Jun 29 19:31:02 2019 GMT
>> > >>
>> > >>
>> > >> In the server log:
>> > >>
>> > >> 2019-06-29 15:31:02,150 DEBUG
>> > >>
>> >
>>
> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
>> > >> (default task-1) Unable to find
>> revocation information for
>> > >> certificate with serial '2ffff4' from
>> issuer 'CN=XXX....'
>> > >>
>> > >>
>> > >> Honestly, I am no 100% sure that the
>> problem is that
>> > >> max_allowe_packet is too small on the
>> mysql client side,
>> > but that is
>> > >> my best guess.
>> > >>
>> > >> What I AM sure is that the CRL is in
>> mysql, but when I do
>> > a test
>> > >> with OCSP Responder, it cannot find
>> the cert serial number
>> > in the CRL.
>> > >>
>> > >> Jim
>> > >>
>> > >>
>> > >> On Saturday, June 29, 2019, 4:52:20
>> AM EDT,
>> > <oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya...
> <mailto:oh...@ya...>>>>> wrote:
>> > >>
>> > >>
>> > >> Hi Tomas,
>> > >>
>> > >> I already had the max_allowed_packet
>> configured on the Mysql
>> > >> server-side (in /etc/my.cnf).
>> > >>
>> > >> So as I said, when testing with mysql
>> (client) I had to
>> > include
>> > >> "--max_allowed_packet=100M" on the
>> mysql (client) command
>> > line, e.g.:
>> > >>
>> > >> mysql -u root -p
>> --max_allowed_packet=100M
>> > >>
>> > >> If I did NOT include the
>> "--max_allowed_packet=100M" on
>> > that command
>> > >> line, then when I did a "select" on
>> the field base64Crl, I
>> > would get
>> > >> an error.
>> > >>
>> > >> I suspect that EJBCA is acting as a
>> Mysql client, so, in
>> > order to
>> > >> work with such large CRLs, there
>> needs to be some way to
>> > set the
>> > >> equivalent of the
>> "--max_allowed_packet=100M" on the EJBCA
>> > side.
>> > >>
>> > >> How can I do that?
>> > >>
>> > >> If that can't be done, then I think
>> that when EJBCA tries to
>> > >> retrieve the CRL from the Mysql, it
>> (EJBCA) will see the
>> > same error.
>> > >>
>> > >> Thanks,
>> > >> Jim
>> > >>
>> > >> On Saturday, June 29, 2019, 3:25:47
>> AM EDT, Tomas Gustavsson
>> > >> <to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>> > <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>> <mailto:to...@pr...
> <mailto:to...@pr...>
>> <mailto:to...@pr...
> <mailto:to...@pr...>>>>> wrote:
>> > >>
>> > >>
>> > >>
>> > >> You configure max allowed packet size
>> in you MySQL
>> > configuration on the
>> > >> MySQL server.
>> > >>
>> > >> /Tomas
>> > >>
>> > >> On 2019-06-29 02:19, o haya via
>> Ejbca-develop wrote:
>> > >> > I am not 100% sure, but I think I
>> know what might be
>> > causing this
>> > >> problem.
>> > >> >
>> > >> > I did a test, where I just tried to
>> log into mysql and
>> > do the
>> > >> following
>> > >> > select:
>> > >> >
>> > >> > select base64Crl INTO OUTFILE
>> > '/var/lib/mysql-files/xx.out' from
>> > >> CRLData
>> > >> > where cRLNumber=3167;
>> > >> >
>> > >> > and I got an error saying that the
>> result was more
>> > >> than max_allowed_packet.
>> > >> >
>> > >> > So then I did the same thing, but
>> for the mysql command
>> > I used:
>> > >> >
>> > >> > mysql -u root -p
>> --max_allowed_packet=100M
>> > >> >
>> > >> > and then when I did that same
>> select, I was able to
>> > output the entire
>> > >> > CRL as a PEM.
>> > >> >
>> > >> > So, I am theorizing that that is
>> what is happening with
>> > EJBCA, i.e.,
>> > >> > when I run the "openssl ocsp"
>> command, EJBCA is querying the
>> > > base64CRL
>> > >> > but the result is too large and so
>> that is causing the
>> > query to fail.
>> > >> >
>> > >> > So the question now is: How can I
>> cause EJBCA OCSP
>> > Responder to run
>> > >> > with a larger max_allowed_packet so
>> that the query
>> > succeeds (and then
>> > >> > hopefully EJBCA OCSP Responder will
>> be able to check the
>> > revocation
>> > >> > properly)?
>> > >> >
>> > >> > Does anyone know how that can be done?
>> > >> >
>> > >> > Thanks,
>> > >> > Jim
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Friday, June 28, 2019, 07:16:39
>> PM UTC, o haya
>> > > <oh...@ya...
> <mailto:oh...@ya...> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>>
>> > >> <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>> > <mailto:oh...@ya...
> <mailto:oh...@ya...>
>> <m...
[truncated message content] |
