Menu
▾
▴
Re: [Ejbca-develop] EJBCA OCSP Responder does not appear to
|
From: Tomas G. <to...@pr...> - 2019-07-19 08:18:19
|
Ah, eap 7.2 corresponds to wildly 14 as base.
Yes, run that to convert, then you should be able to create the index.
Cheers,
Tomas
On July 19, 2019 10:07:01 AM GMT+02:00, ohaya--- via Ejbca-develop <ejb...@li...> wrote:
> Hi Tomas,
>No, I am not using Wildfly, but rather JBOSS... specifically
>"jboss-eap-7.2".
>So I restored my snapshot from just after I had deleted all the rows in
>CertificateData that were created before.
>And it looks like all the tables in the "ejbca" database are MyISAM.
>So I found a one-liner that will convert them all to InnoDB:
>mysql -u root -p ejbca -e "show table status where Engine='MyISAM';" |
>awk 'NR>1 {print "ALTER TABLE "$1" ENGINE = InnoDB;"}' | mysql -u
>root -p ejbca
>Should I run that and convert all the tables in the "ejbca" to
>InnoDB???
>
>And then try to run the create indexes SQL script?
>
>Thanks for all your help!
>Jim
>
>
>
>
>On Friday, July 19, 2019, 8:01:15 AM UTC, Tomas Gustavsson
><to...@pr...> wrote:
>
>
>Yes, MyISAM is horrible and can not be used.
>
>I'm not sure about the exact commands to change it. But I think there
>are good internet guides.
>
>The reason you have MyISAM is probably because you are using WildFly 14
>(are you?). There was a change in Hibernate that caused this issue with
>automatic database creation.
>
>One way to modify is to mysqldump the database, change enging in the
>sql
>file and restore it.
>
>Regards,
>Tomas
>
>On 2019-07-19 09:19, ohaya--- via Ejbca-develop wrote:
>> Hi,
>>
>> Mysql version: Server version: 5.7.26 MySQL Community Server (GPL)
>>
>>
>>
>>
>> mysql> show create table CertificateData;
>>
>>
>>
>> | CertificateData | CREATE TABLE `CertificateData` (
>> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `certificateProfileId` int(11) NOT NULL,
>> `endEntityProfileId` int(11) DEFAULT NULL,
>> `expireDate` bigint(20) NOT NULL,
>> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `notBefore` bigint(20) DEFAULT NULL,
>> `revocationDate` bigint(20) NOT NULL,
>> `revocationReason` int(11) NOT NULL,
>> `rowVersion` int(11) NOT NULL,
>> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `status` int(11) NOT NULL,
>> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
>NULL,
>> `type` int(11) NOT NULL,
>> `updateTime` bigint(20) NOT NULL,
>> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
>NULL,
>> `base64Cert` longtext,
>> `rowProtection` longtext,
>> PRIMARY KEY (`fingerprint`),
>> KEY `certificatedata_idx2` (`username`)
>> ) ENGINE=MyISAM DEFAULT CHARSET=utf8 |
>>
>>
>>
>> AH! Yes mine is showing as MyISAM and not InnoDB!
>>
>>
>>
>> Do I need to change the engine to InnoDB? And, if so, how do I do
>that?
>>
>> Do I need to do that for every table in the ejbca database?
>>
>>
>> Thanks,
>> Jim
>>
>>
>>
>>
>> On Friday, July 19, 2019, 6:20:23 AM UTC, Tomas Gustavsson
>> <to...@pr...> wrote:
>>
>>
>>
>> That looks about right. What version of database are you using? I
>just
>> tried the index command (on line 23 in the SQL file) myself and it
>works.
>>
>> CREATE INDEX certificatedata_idx4 ON CertificateData (subjectDN);
>>
>>
>> What does your database creation say?
>>
>>>show create table CertiifcateData;
>>
>> It should look something like this:
>>
>> | CertificateData | CREATE TABLE `CertificateData` (
>> `fingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `cAFingerprint` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `certificateProfileId` int(11) NOT NULL,
>> `crlPartitionIndex` int(11) DEFAULT NULL,
>> `endEntityProfileId` int(11) DEFAULT NULL,
>> `expireDate` bigint(20) NOT NULL,
>> `issuerDN` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `notBefore` bigint(20) DEFAULT NULL,
>> `revocationDate` bigint(20) NOT NULL,
>> `revocationReason` int(11) NOT NULL,
>> `rowVersion` int(11) NOT NULL,
>> `serialNumber` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `status` int(11) NOT NULL,
>> `subjectAltName` varchar(2000) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `subjectDN` varchar(400) CHARACTER SET utf8 COLLATE utf8_bin NOT
>NULL,
>> `subjectKeyId` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin
>> DEFAULT NULL,
>> `tag` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
>NULL,
>> `type` int(11) NOT NULL,
>> `updateTime` bigint(20) NOT NULL,
>> `username` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
>NULL,
>> `base64Cert` longtext,
>> `certificateRequest` longtext,
>> `rowProtection` longtext,
>> PRIMARY KEY (`fingerprint`)
>> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
>>
>> Is the database engine wrong? I.e. not InnoDB?
>>
>> I am importing the CRL right now and processing about 27 records per
>> second on my laptop.
>>
>> Regards,
>> Tomas
>>
>>
>> On 2019-07-18 18:09, ohaya--- via Ejbca-develop wrote:
>>> According to my notes I used these commands:
>>>
>>> |*/CREATE DATABASE ejbca CHARACTER SET utf8 COLLATE
>utf8_general_ci;/*|
>>>
>>> |*/ /*|
>>>
>>> |*/GRANT ALL PRIVILEGES ON ejbca.* TO 'ejbca'@'localhost' IDENTIFIED
>BY
>>> 'ejbca';/*|
>>>
>>>
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 3:37:51 PM UTC, Tomas Gustavsson
>>> <to...@pr... <mailto:to...@pr...>> wrote:
>>>
>>>
>>> What command did you use to create the database?
>>>
>>> On July 18, 2019 5:32:33 PM GMT+02:00, ohaya--- via Ejbca-develop
>>> <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi,
>>>
>>> This is really getting frustrating!!!
>>>
>>> [root@ejbca <mailto:root@ejbca> sql-scripts]# mysql -u root
>> -p'xxxxxx' ejbca <
>>> create-index-ejbca.sql
>>> mysql: [Warning] Using a password on the command line interface
>can
>>> be insecure.
>>> ERROR 1071 (42000) at line 23: Specified key was too long; max
>key
>>> length is 1000 bytes
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 2:59:36 PM UTC, oh...@ya...
>> <mailto:oh...@ya...>
>>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> The "machine" is a virtual machine under VirtualBox on a machine
>I
>>> have that has 64GB physical RAM. Drives are all 7200 RPM drives.
>>>
>>> I will restore the snapshot, then increase the memory and # of
>>> processors and try to run the create index again.
>>>
>>> Jim
>>>
>>> On Thursday, July 18, 2019, 2:32:13 PM UTC, Tomas Gustavsson
>>> <to...@pr... <mailto:to...@pr...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> Just run create index script.
>>>
>>> Create index should run pretty fast even with millions of records
>in
>>> the database. I suspect that your database is configured very
>slow,
>>> perhaps running everything on disl, without RAM, using slow
>disks?
>>>
>>> A standard installation of mysql/mariadb on a centos machine
>should
>>> run faster. Then you have pretty low amount of ram for running
>>> everything on one box, handing millions of records.
>>>
>>> Regards,
>>> Tomas
>>>
>>> On July 18, 2019 4:22:56 PM GMT+02:00, ohaya--- via Ejbca-develop
>>> <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi,
>>>
>>> I had taken a snapshot of the machine before I tried to do
>the
>>> index earlier, so I restored that snapshot.
>>>
>>> Then, I tried running the "optimize-ejbca-mysql.sql" and I
>got a
>>> different/new error:
>>>
>>> ERROR 1091 (42000) at line 27: Can't DROP
>>> 'certificatedata_idx5'; check that column/key exists
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 2:15:07 PM UTC, oh...@ya...
>> <mailto:oh...@ya...>
>>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> Oh oh, I just got this from the mysql execution:
>>>
>>> ERROR 1071 (42000) at line 23: Specified key was too long;
>max
>>> key length is 1000 bytes
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 2:09:26 PM UTC, oh...@ya...
>> <mailto:oh...@ya...>
>>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> FYI, I am running the create-index-ejbca.sql script now, but
>>> that seems to be taking FOREVER?
>>>
>>> I had stopped the earlier import, and then deleted all the
>rows
>>> for that CA DN in CertificateData table, and then bounced the
>>> machine, then I ran the create-index-ejbca.sql.
>>>
>>> Why is it taking so long? Should I have done something in
>>> addition to deleting the rows? I am guessing that the reason
>>> the sql script is taking so long is that it's somehow still
>>> trying to index the rows were deleted or something like that?
>>>
>>> Thanks,
>>> Jim
>>>
>>>
>>>
>>> On Thursday, July 18, 2019, 1:52:53 PM UTC, oh...@ya...
>> <mailto:oh...@ya...>
>>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> I notice that there is:
>>>
>>> - create-index-ejbca.sql
>>> - optimize-ejbca-mysql.sql
>>>
>>> Should I run BOTH of them, or just the
>"create-index-ejbca.sql "?
>>>
>>> Thanks,
>>> Jim
>>>
>>>
>>> On Thursday, July 18, 2019, 7:40:03 AM UTC, Tomas Gustavsson
>>> <to...@pr... <mailto:to...@pr...>> wrote:
>>>
>>>
>>> Yes, i could see from your output from top that mysql takes
>all
>>> CPU. This indicates that you don't have any indexes in your
>>> database, which makes it horribly slow as numbers grow.
>>>
>>> Apply database indexes from doc/sql-scripts.
>>>
>>> Regards,
>>> Tomas
>>>
>>>
>>> On July 18, 2019 4:43:45 AM GMT+02:00, ohaya--- via
>>> Ejbca-develop <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi,
>>>
>>> So far, processed only 185706 entries, and it looks like
>the
>>> processing is going to a crawl (really slow now).
>>>
>>> Which kind of makes me wonder if maybe part of reason for
>>> the slowness is that it is having to search the
>certificate
>>> database for the serial number, and now, as there are
>more
>>> dummy certs in the database, that search is taking longer
>>> and longer?
>>>
>>> Jim
>>>
>>>
>>> On Wednesday, July 17, 2019, 6:03:37 PM UTC, ohaya--- via
>>> Ejbca-develop <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> This is what top is showing now (the import is still
>running):
>>>
>>> top - 13:56:19 up 1 day, 1:09, 3 users, load average:
>>> 1.18, 1.11, 1.07
>>> Tasks: 167 total, 2 running, 165 sleeping, 0
>stopped,
>>> 0 zombie
>>> %Cpu(s): 47.4 us, 2.0 sy, 0.0 ni, 50.3 id, 0.0 wa,
>0.0
>>> hi, 0.3 si, 0.0 st
>>> KiB Mem : 4045880 total, 123192 free, 3053072 used,
>>> 869616 buff/cache
>>> KiB Swap: 3905532 total, 3753980 free, 151552 used.
>>> 688136 avail Mem
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU
>>> %MEM TIME+ COMMAND
>>> 1489 mysql 20 0 1580320 129624 4396 S 92.7
>3.2
>>> 1227:41 mysqld
>>> 3322 root 20 0 4031732 1.3g 7236 S 6.0 33.2
>>> 139:05.67 java
>>> 3617 root 20 0 3546412 1.1g 7256 S 1.3 29.3
>>> 176:26.68 java
>>> 22466 root 20 0 0 0 0 S 0.7
>0.0
>>> 0:00.73 kworker/u4:1
>>>
>>> That is running on 2-CPU machine with 4GB RAM.
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>> On Wednesday, July 17, 2019, 2:33:45 PM UTC, Tomas
>>> Gustavsson <to...@pr... <mailto:to...@pr...>>
>> wrote:
>>>
>>>
>>> Yes, like top.
>>>
>>>
>>> On July 17, 2019 3:52:53 PM GMT+02:00, ohaya--- via
>>> Ejbca-develop <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi Tomas,
>>>
>>> I'm sorry that I missed the message below, but I am
>not
>>> clear what you are asking me to check/monitor? Do
>you
>>> mean like running something like "top" command?
>>>
>>> Thanks,
>>> Jim
>>>
>>> On Tuesday, July 16, 2019, 7:43:45 PM UTC, Tomas
>>> Gustavsson <to...@pr...
>> <mailto:to...@pr...>> wrote:
>>>
>>>
>>> To verify that the changes actually happen...
>>>
>>> Do you monitor CPU usage to see what processes use
>CPU
>>> and memory?
>>>
>>>
>>> On July 16, 2019 8:52:37 PM GMT+02:00, ohaya--- via
>>> Ejbca-develop <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>> Hi,
>>>
>>> The import has been running for about an hour now
>>> and has imported 17836 entries.
>>>
>>> Per what I had said previously, this CRL file has
>>> 1002255 entries, that means it would take about
>56+
>>> hours to import this CRL.
>>>
>>> So, this import processing is still way too slow.
>>>
>>> I don't know what is involved, or even how much
>it
>>> would improve things, but I think that the next
>>> thing to try is to eliminated both the check for
>the
>>> serial number in the DB AND the dummy cert
>creation.
>>>
>>> Jim
>>>
>>>
>>> On Tuesday, July 16, 2019, 5:54:54 PM UTC,
>ohaya---
>>> via Ejbca-develop
>>> <ejb...@li...
>> <mailto:ejb...@li...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> Ok, I rebuilt the JAR using your Java class, and
>it
>>> is running now, and creating rows in the
>>> CertificateData, but it still SEEMS kind of
>slow.
>>>
>>> I am running now and will try to get a
>measurement
>>> of revoked entries/time and will post back.
>>>
>>> Jim
>>>
>>>
>>>
>>>
>>> On Tuesday, July 16, 2019, 5:25:32 PM UTC,
>>> oh...@ya... <mailto:oh...@ya...>
>> <oh...@ya... <mailto:oh...@ya...>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> So I just tried what I said (emptying the "if")
>and
>>> you are right, it doesn't create the rows in the
>>> CeritificateData table :(...
>>>
>>> Ok, I will try your version...
>>>
>>> Jim
>>>
>>> On Tuesday, July 16, 2019, 3:57:31 PM UTC, Tomas
>>> Gustavsson <to...@pr...
>> <mailto:to...@pr...>> wrote:
>>>
>>>
>>>
>>> Hi,
>>>
>>> I don't think so, no quite actually. In order to
>use
>>> the API to populate
>>> the database you need the certificate object it
>>> seems. Something like
>>> the attached file should work, it still creates a
>>> dummy certificate, but
>>> uses a static hard coded key, so no key
>generation.
>>>
>>> The file assumes you are on 6.15.x.
>>>
>>> Regards,
>>> Tomas
>>>
>>> On 2019-07-16 15:02, ohaya--- via Ejbca-develop
>wrote:
>>> > For testing (only...), I am thinking that if I
>>> modify this If clause in
>>> > CaImportCrlCommand.java:
>>> >
>>> > if (adaptive)
>>> > {
>>> > Date time = new Date();
>>> > KeyPair key_pair =
>>> KeyTools.genKeys("2048", "RSA");
>>> >
>>> > SubjectPublicKeyInfo pkinfo =
>>> >
>>>
>> SubjectPublicKeyInfo.getInstance(key_pair.getPublic().getEncoded());
>>> > X500Name dnName = new
>>> X500Name("CN=Dummy Missing in
>>> > Imported CRL, serialNumber=" + serialHex);
>>> > Date notAfter = new
>>> Date(time.getTime() + 315360000000L);
>>> > X509v3CertificateBuilder
>certbuilder
>>> = new
>>> >
>X509v3CertificateBuilder(X500Name.getInstance(cacert
>>> >
>>> .getSubjectX500Principal().getEncoded()),
>serialNr,
>>> > time, notAfter, dnName, pkinfo);
>>> > ContentSigner signer = new
>>> BufferingContentSigner(new
>>> >
>>>
>>
>JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(key_pair
>>> > .getPrivate()), 20480);
>>> > X509CertificateHolder certHolder
>=
>>> certbuilder.build(signer);
>>> > X509Certificate certificate =
>>> >
>>>
>>
>(X509Certificate)CertTools.getCertfromByteArray(certHolder.getEncoded(),
>>> > X509Certificate.class);
>>> >
>>> > String fingerprint =
>>> > CertTools.getFingerprintAsString(certificate);
>>> >
>>> > EndEntityInformation
>>> missingUserEndEntityInformation =
>>> >
>>>
>>
>((EndEntityAccessSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityAccessSessionRemote.class)).findUser(getAuthenticationToken(),
>>> > missing_user_name);
>>> > if
>(missingUserEndEntityInformation
>>> == null)
>>> > {
>>> > log.debug("Loading/updating
>user "
>>> + missing_user_name);
>>> >
>>> > EndEntityInformation
>userdataNew
>> = new
>>> > EndEntityInformation(missing_user_name,
>>> > CertTools.getSubjectDN(certificate),
>>> cainfo.getCAId(), null, null, 10,
>>> > new EndEntityType(EndEntityTypes.ENDUSER), 1,
>1,
>>> null, null, 1, 0, null);
>>> >
>>> >
>userdataNew.setPassword("foo123");
>>> >
>>> >
>>>
>>
>((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).addUser(getAuthenticationToken(),
>>> > userdataNew, false);
>>> >
>>> > log.info("User '" +
>>> missing_user_name + "' has been
>>> > added.");
>>> >
>>> >
>>>
>>
>((EndEntityManagementSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(EndEntityManagementSessionRemote.class)).setUserStatus(getAuthenticationToken(),
>>> > missing_user_name, 50);
>>> >
>>> > log.info("User '" +
>>> missing_user_name + "' has been
>>> > updated.");
>>> > }
>>> >
>>> >
>>>
>>
>((CertificateStoreSessionRemote)EjbRemoteHelper.INSTANCE.getRemoteSession(CertificateStoreSessionRemote.class)).storeCertificateRemote(getAuthenticationToken(),
>>> >
>>> > EJBTools.wrap(certificate),
>>> missing_user_name,
>>> > fingerprint, 20, 1, 1, 0, null, new Date()
>>> >
>>> > .getTime());
>>> > log.info("Dummy certificate '" +
>>> serialHex + "' has been
>>> > stored.");
>>> > }
>>> >
>>> > to (for example):
>>> >
>>> > if (adaptive)
>>> > {
>>> >
>>> > log.info("*** SKIPPING ***
>Dummy
>>> certificate '" +
>>> > serialHex + "' has been stored.");
>>> > }
>>> >
>>> >
>>> > Would THAT work??
>>> >
>>> > Thanks,
>>> > Jim
>>> >
>>> >
>>> >
>>> >
>>> > On Tuesday, July 16, 2019, 10:59:31 AM UTC,
>>> ohaya--- via Ejbca-develop
>>> > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>>> >
>>> >
>>> > Hi Tomas (or anyone else),
>>> >
>>> > Sorry - I was typing my last msg on phone, and
>it
>>> wasn't as detailed or
>>> > clear as I wanted.
>>> >
>>> > Basically what I was saying is I was thinking
>>> that, as a quick sanity
>>> > test, if there was one or two places where the
>> code in
>>> > the CaImportCRLCommand.java was maybe making a
>>> method call to create the
>>> > dummy cert and put it into the database, then
>>> maybe I could decompile
>>> > and modify the CaImportCRLCommand.java, and
>>> comment out those method
>>> > calls, then, re-do my large CRL import test and
>>> time it?
>>> >
>>> > If so, can you let me know specifically where
>>> those calls are, and if I
>>> > just comment them out, would that work?
>>> >
>>> > Ironically, that (not doing the dummy cert
>>> creation and writing) was
>>> > what I *thought* the "LENIENT" option was
>doing).
>>> >
>>> > Thanks,
>>> > Jim
>>> >
>>> > P. S. I have time today to try this if I
>figure
>>> out what needs to be done.
>>> >
>>> >
>>> >
>>> > On Monday, July 15, 2019, 6:24:53 PM UTC,
>ohaya---
>>> via Ejbca-develop
>>> > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>>> >
>>> >
>>> >
>>> > Yes, we are definitely interested/willing to
>test
>>> patches. I actually
>>> > was wondering if it would be a quick test to
>mod
>>> the code somewhere to
>>> > avoid a call that creates the dummy cert, to
>get
>>> an idea if that would
>>> > reduce the import time enough??
>>> >
>>> >
>>> > Sent from Yahoo Mail on Android
>>> >
>>>
>>
><https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
>>> >
>>> > On Mon, Jul 15, 2019 at 12:10 PM, Tomas
>Gustavsson
>>> > <to...@pr...
><mailto:to...@pr...>
>> <mailto:to...@pr... <mailto:to...@pr...>>>
>>> wrote:
>>> > Most large users, some very very large,
>>> typically uses on line
>>> > updates directly from the CA. Many CAs today
>>> don't accept only
>>> > importing CRLs, as they need to keep track
>of
>>> issued certificates as
>>> > well not only revoked (can't use "non
>existing
>>> is good").
>>> > Having said thst, it's interesting to be
>able
>>> to import large CRLs,
>>> > so improving it is of interest.
>>> >
>>> > Importing CRLs would actually not have to
>>> generate a dummy cert at
>>> > all, since a certificate object is not
>needed
>>> on the responder,
>>> > which should make it very fast.
>>> >
>>> > If you are willing to test patches, we can
>work
>>> with you on improving.
>>> >
>>> > Regards,
>>> > Tomas
>>> >
>>> >
>>> > On July 15, 2019 5:26:08 PM GMT+02:00,
>ohaya---
>>> via Ejbca-develop
>>> > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I am curious whether you have other
>users
>>> that import large CRLs
>>> > like this?
>>> >
>>> > Are we the only ones who would encounter
>>> these really long
>>> > import times, because the other users
>who
>>> might have large CRLs
>>> > use the EJBCA CA also, so they don't run
>>> into having the dummy
>>> > certs being created for every entry in
>the
>> CRL?
>>> >
>>> > Jim
>>> >
>>> > On Monday, July 15, 2019, 1:42:57 PM
>UTC,
>>> Tomas Gustavsson
>>> > <to...@pr...
><mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>>> wrote:
>>> >
>>> >
>>> >
>>> > I see...
>>> >
>>> > I can see that a new key pair is being
>>> generated for each dummy
>>> > cert.
>>> >
>>> > In CaImportCRLCommand.java:
>>> >
>>> > �� final KeyPair key_pair =
>>> KeyTools.genKeys("2048",
>>> > AlgorithmConstants.KEYALGORITHM_RSA);
>>> >
>>> > Chaning that to 1024 or 512 would be an
>>> easy improvement. A bit more
>>> > complex, but even larger speedup would
>be
>>> to use a "hard coded"
>>> > dummy
>>> > key, i.e. same key for all dummy certs.
>>> >
>>> > You can experiment by changing 2048 to
>512
>>> (not secure but it
>>> > doesn't
>>> > have to be for this dummy cert).
>>> >
>>> > Regards,
>>> > Tomas
>>> >
>>> > On 2019-07-11 14:19, ohaya--- via
>>> Ejbca-develop wrote:
>>> > > FYI, the is running, but it is REALLY
>>> slow. The CRL file has
>>> > 1002255
>>> > > entries in it, and the import is only
>up
>>> to 8899 now, after
>>> > about 40
>>> > > minutes... which would take about 75
>> hours!!
>>> > >
>>> > > In our production systems, we are
>>> required to update some CRLs
>>> > every
>>> > > hour, so that would be a problem.
>>> > >
>>> > > What can I do to speed that up??
>>> > >
>>> > > Thanks,
>>> > > Jim
>>> > >
>>> > >
>>> > > On Thursday, July 11, 2019, 11:09:33
>AM
>>> UTC, ohaya--- via
>>> > Ejbca-develop
>>> > > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>> wrote:
>>> > >
>>> > >
>>> > > AHHH... Ok, it is running now, and
>from
>>> checking the
>>> > CertificateData in
>>> > > the Mysql database, I can see that it
>is
>>> now populating the
>>> > > CertificateData!!
>>> > >
>>> > > Thanks!
>>> > >
>>> > > Jim
>>> > >
>>> > >
>>> > >
>>> > > On Thursday, July 11, 2019, 11:07:22
>AM
>>> UTC, ohaya--- via
>>> > Ejbca-develop
>>> > > <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>> wrote:
>>> > >
>>> > >
>>> > > I wasn't sure if we should use
>ADAPTIVE,
>>> since we are not
>>> > using the CA part.
>>> > >
>>> > >
>>> > > I will try with "-o ADAPTIVE" now.
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Thursday, July 11, 2019, 7:42:14 AM
>>> UTC, Tomas Gustavsson
>>> > > <to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>>>> wrote:
>>> > >
>>> > >
>>> > >
>>> > > It should be ADAPTIVE.
>>> > >
>>> > > The documentation says:
>>> > > bin/ejbca.sh ca importcrl --help
>>> > > ---
>>> > > ADAPTIVE means that missing
>certficates
>>> will be replaced by dummy
>>> > > certificates to cater for proper CRLs
>for
>>> missing certificates.
>>> > > ---
>>> > >
>>> > > Regards,
>>> > > Tomas
>>> > >
>>> > > On 2019-07-11 08:05, ohaya--- via
>>> Ejbca-develop wrote:
>>> > >> Hi,
>>> > >>
>>> > >> I've been looking at the code in
>>> CaImportCRLCommand, and I am
>>> > wondering
>>> > >> if, when the command is run in
>LENIENT
>>> mode and it doesn't
>>> > find the cert
>>> > >> in the database (and outputs the
>>> "missing in database" error
>>> > message),
>>> > >> what is supposed to happen?
>>> > >>
>>> > >> Mind you, the only error messages I
>am
>>> seeing are the "missing in
>>> > >> database" error messages. I don't
>see
>>> any of the other error
>>> > messages
>>> > >> that are in/from
>CaImportCRLCommand.java.
>>> > >>
>>> > >> I am wondering if, when the command
>is
>>> run with "-o LENIENT",
>>> > and when
>>> > >> it outputs the "missing in database"
>>> error message, then the
>>> > ejbca.sh is
>>> > >> basically not doing anything (and not
>>> storing anything into the
>>> > >> CertificateData table)?
>>> > >>
>>> > >> From the lack of any other error
>>> messages (vs. what I see in the
>>> > >> CaImportCRLCommand.java code), that
>is
>>> what it looks like is
>>> > happening.
>>> > >>
>>> > >> So given I am using EJBCA for OCSP
>>> responder only, is running the
>>> > >> ejbca.sh in the "-o LENIENT" mode
>correct?
>>> > >>
>>> > >> Jim
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Thursday, July 11, 2019, 5:43:37
>AM
>>> UTC, ohaya--- via
>>> > Ejbca-develop
>>> > >> <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>
>>> > >
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> I've been doing a bunch of testing
>with
>>> trying to import the
>>> > large CRL,
>>> > >> and wanted to summarize:
>>> > >>
>>> > >> - The ejbca.sh runs fine - it
>completes
>>> without outputting an
>>> > errors,
>>> > >> except for outputting a bunch of the
>>> "cert xxxx missing in
>>> > database"
>>> > >> messages.
>>> > >>
>>> > >> - When the import completes, and I
>check
>>> the Mysql database,
>>> > the crlData
>>> > >> (the full PEM of the CRL) is in the
>>> database, but there is no
>>> > rows in
>>> > >> the CertificateData table for that CA
>at
>>> all.
>>> > >>
>>> > >> - I even figured out how to get the
>>> ejbca.sh to log using
>>> > >> log4j.properties, and it output a
>bunch
>>> of logging, mainly
>>> > the missing
>>> > >> in database errors, but other than
>that
>>> I didn't see any
>>> > obvious errors.
>>> > >>
>>> > >>
>>> > >> So, bottom line is that even though
>the
>>> ejbca.sh seems to run
>>> > without
>>> > >> errors (other than the missing in
>>> database errors), the import is
>>> > >> causing the CRL PEM to be stored in
>the
>>> database, but the
>>> > >> CertificateData table is not being
>>> populated at all for that CA.
>>> > >>
>>> > >>
>>> > >> Jim
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Wednesday, July 10, 2019, 7:40:16
>PM
>>> UTC, ohaya--- via
>>> > Ejbca-develop
>>> > >> <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>
>>> > >
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> Ok, I found Tomas' original
>message...
>>> it was in my Spam
>>> > folder....
>>> > >>
>>> > >> I also tried re-running the
>"./ejbca.sh
>>> ca importcrl" command
>>> > a bunch of
>>> > >> ways. I think it is running without
>>> outputting any errors,
>>> > but still,
>>> > >> no rows are appearing with that
>issuerDN
>>> at all.
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Tuesday, July 9, 2019, 10:45:01 AM
>>> UTC, Tomas Gustavsson
>>> > >> <to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
><mailto:to...@pr...>>>
>>> > <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> OCSP responses does not have anything
>to
>>> do with the CRL
>>> > itself. Once
>>> > >> the CRL has been imported each
>>> certificate status is in its
>>> > own record
>>> > >> in the CertificateData table. This is
>>> where status is read
>>> > when reponses
>>> > >> are created.
>>> > >> If there are issues, it is in that
>case
>>> related to the import
>>> > of the CRL.
>>> > >>
>>> > >> Verify that certificate records are
>>> created in
>>> > CertificateData when
>>> > >> importing the CRL.
>>> > >> Also, a debug log when making OCSP
>>> request will tell exact
>>> > details of
>>> > >> what the responder finds in the
>database
>>> or not.
>>> > >>
>>> > >> Regards,
>>> > >> Tomas
>>> > >>
>>> > >>
>>> > >> On July 8, 2019 4:59:06 PM GMT+02:00,
>>> ohaya--- via Ejbca-develop
>>> > >> <ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>
>>> > >
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>
>>> >
>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>
>>> <mailto:ejb...@li...
>> <mailto:ejb...@li...>>>>> wrote:
>>> > >>
>>> > >> [Re-sending, as this (handling of
>>> large CRLs) is still a
>>> > problem and
>>> > >> is still not working.]
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Monday, July 1, 2019, 12:47:23
>PM
>>> UTC, <oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>>> > <mailto:oh...@ya...
>> <mailto:oh...@ya...>
>>> <mailto:oh...@ya...
><mailto:oh...@ya...>>>
>>> > > <mailto:oh...@ya...
>> <mailto:oh...@ya...>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
>>> <mailto:oh...@ya...
>> <mailto:oh...@ya...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> I think that I've found how to
>>> set/change the
>>> > max_allowed_packet on
>>> > >> the client (i.e., EJBCA) side, by
>>> editing the Mysql URL in the
>>> > >>
>>> standalone/configuration/standalone.xml to:
>>> > >>
>>> > >>
>>> > >
>>> >
>>>
>>
><connection-url>jdbc:mysql://127.0.0.1:3306/ejbca?maxAllowedPacket=104857600</connection-url>
>>> > >>
>>> > >> However, when I test the OCSP
>>> Responder with a serial
>>> > number (and CA
>>> > >> cert) from the CA with the large
>>> (>71MB) CRL file, I am still
>>> > >> getting "unknown" and EJBCA OCSP
>>> Responder is still NOT
>>> > FINDING the
>>> > >> cert serial number.
>>> > >>
>>> > >> NOTE, again, that I can test with
>>> certs (and CA cert) that
>>> > have
>>> > >> smaller CRL files, and those tests
>work.
>>> > >>
>>> > >> So it is only when testing with
>the
>>> large CRL file that is
>>> > failing
>>> > >> with the OCSP Responder.
>>> > >>
>>> > >> This seems like a potential BUG
>for
>>> EJBCA OCSP Responder?
>>> > >>
>>> > >> Please advise.
>>> > >>
>>> > >> Thanks,
>>> > >> Jim
>>> > >> On Saturday, June 29, 2019,
>3:36:18
>>> PM EDT,
>>> > <oh...@ya...
><mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>>> > > <mailto:oh...@ya...
>> <mailto:oh...@ya...>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
>>> <mailto:oh...@ya...
>> <mailto:oh...@ya...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi,
>>> > >>
>>> > >> BTW, just to check, I have
>actually
>>> gone through the
>>> > process of
>>> > >> getting the crlData from the MySQL
>>> using mysql client (I
>>> > had to add
>>> > >> the --max_allowed-packet=100M to
>the
>>> mysql command) and I
>>> > was able
>>> > >> to actually get the PEM from the
>>> Mysql database that way.
>>> > >>
>>> > >> So, I KNOW that the entire CRL is
>in
>>> the Mysql database,
>>> > and yet
>>> > >> when I run an "openssl ocsp" test
>>> against the EJBCA OCSP
>>> > Responder,
>>> > >> it says it cannot find the
>revocation.
>>> > >>
>>> > >> OCSP Request Data:
>>> > >> Version: 1 (0x0)
>>> > >> Requestor List:
>>> > >> Certificate ID:
>>> > >> Hash Algorithm: sha1
>>> > >> Issuer Name Hash:
>>> 37E3972FD58A03EEC31D8DF5F6F03F6AE4F5C600
>>> > >> Issuer Key Hash:
>>> 3929A80864E28F83AA816147119133F85A37A6E4
>>> > >> Serial Number: 2FFFF4
>>> > >> Request Extensions:
>>> > >> OCSP Nonce:
>>> > >>
>04103054A60A55E389B239A55228033D8F62
>>> > >> Response verify OK
>>> > >> 0x2FFFF4: unknown
>>> > >> This Update: Jun 29 19:31:02 2019
>GMT
>>> > >>
>>> > >>
>>> > >> In the server log:
>>> > >>
>>> > >> 2019-06-29 15:31:02,150 DEBUG
>>> > >>
>>> >
>>>
>> [org.cesecore.certificates.ocsp.OcspResponseGeneratorSessionBean]
>>> > >> (default task-1) Unable to find
>>> revocation information for
>>> > >> certificate with serial '2ffff4'
>from
>>> issuer 'CN=XXX....'
>>> > >>
>>> > >>
>>> > >> Honestly, I am no 100% sure that
>the
>>> problem is that
>>> > >> max_allowe_packet is too small on
>the
>>> mysql client side,
>>> > but that is
>>> > >> my best guess.
>>> > >>
>>> > >> What I AM sure is that the CRL is
>in
>>> mysql, but when I do
>>> > a test
>>> > >> with OCSP Responder, it cannot
>find
>>> the cert serial number
>>> > in the CRL.
>>> > >>
>>> > >> Jim
>>> > >>
>>> > >>
>>> > >> On Saturday, June 29, 2019,
>4:52:20
>>> AM EDT,
>>> > <oh...@ya...
><mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>
>> <mailto:oh...@ya... <mailto:oh...@ya...>>>
>>> > > <mailto:oh...@ya...
>> <mailto:oh...@ya...>
>>> <mailto:oh...@ya... <mailto:oh...@ya...>>
>> <mailto:oh...@ya... <mailto:oh...@ya...>
>>> <mailto:oh...@ya...
>> <mailto:oh...@ya...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >> Hi Tomas,
>>> > >>
>>> > >> I already had the
>max_allowed_packet
>>> configured on the Mysql
>>> > >> server-side (in /etc/my.cnf).
>>> > >>
>>> > >> So as I said, when testing with
>mysql
>>> (client) I had to
>>> > include
>>> > >> "--max_allowed_packet=100M" on the
>>> mysql (client) command
>>> > line, e.g.:
>>> > >>
>>> > >> mysql -u root -p
>>> --max_allowed_packet=100M
>>> > >>
>>> > >> If I did NOT include the
>>> "--max_allowed_packet=100M" on
>>> > that command
>>> > >> line, then when I did a "select"
>on
>>> the field base64Crl, I
>>> > would get
>>> > >> an error.
>>> > >>
>>> > >> I suspect that EJBCA is acting as
>a
>>> Mysql client, so, in
>>> > order to
>>> > >> work with such large CRLs, there
>>> needs to be some way to
>>> > set the
>>> > >> equivalent of the
>>> "--max_allowed_packet=100M" on the EJBCA
>>> > side.
>>> > >>
>>> > >> How can I do that?
>>> > >>
>>> > >> If that can't be done, then I
>think
>>> that when EJBCA tries to
>>> > >> retrieve the CRL from the Mysql,
>it
>>> (EJBCA) will see the
>>> > same error.
>>> > >>
>>> > >> Thanks,
>>> > >> Jim
>>> > >>
>>> > >> On Saturday, June 29, 2019,
>3:25:47
>>> AM EDT, Tomas Gustavsson
>>> > >> <to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
><mailto:to...@pr...>>>
>>> > <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>> <mailto:to...@pr...
>> <mailto:to...@pr...>
>>> <mailto:to...@pr...
>> <mailto:to...@pr...>>>>> wrote:
>>> > >>
>>> > >>
>>> > >>
>>> > >> You configure max allowed packet
>size
>>> in you MySQL
>>> > configuration on the
>>> > >> MySQL server.
>>> > >>
>>> > >> /Tomas
>>> > >>
>>> > >> On 2019-06-29 02:19, o haya via
>>> Ejbca-develop wrote:
>>> > >> > I am not 100% sure, but I think
>I
>>> know what might be
>>> > causing this
>>> > >> problem.
>>> > >> >
>>> > >> > I did a test, where I just tried
>to
>>> log into mysql and
>>> > do the
>>> > >> following
>>> > >> > select:
>>> > >> >
>>> > >> > select base64Crl INTO OUTFILE
>>> > '/var/lib/mysql-files/xx.out' from
>>> > >> CRLData
>>> > >> > where cRLNumber=3167;
>>> > >> >
>>> > >> > and I got an error saying that
>the
>>> result was more
>>> > >> than max_allowed_packet.
>>> > >> >
>>> > >> > So then I did the same thing,
>but
>>> for the mysql command
>>> > I used:
>>> > >> >
>>> > >> > mysql -u root -p
>>> --max_allowed_packet=100M
>>> > >> >
>>> > >> > and then when I did that same
>>> select, I was able to
>>> > output the entire
>>> > >> > CRL as a PEM.
>>> > >> >
>>> > >> > So, I am theorizing that that is
>>> what is happening with
>>> > EJBCA, i.e.,
>>> > >> > when I run the "openssl ocsp"
>>> command, EJBCA is querying the
>>> > > base64CRL
>>> > >> > but the result is too large and
>so
>>> that is causing the
>>> > query to fail.
>>> > >> >
>>> > >> > So the question now is: How can
>I
>>> cause EJBCA OCSP
>>> > Responder to run
>>> > >> > with a larger max_allowed_packet
>so
>>> that the query
>>> > ...
[truncated message content] |
