Menu
▾
▴
Re: [Ejbca-develop] Around 20K queries to the DB are being produced even if there is only one successful publishing event
|
From: Tomas G. <to...@pr...> - 2019-08-19 05:38:41
|
Hi Jaime,
I hope you saw the commits of the other issues you reported.
I still want to analyze this issue a little more. Are you available?
Regards,
Tomas
On 2019-08-05 12:47, Tomas Gustavsson wrote:
> Hi,
>
> I'm not sure I fully understand. The patch below simply removes the 20K
> limit right, so if there were 100K rows needing to be re-puplished they
> would all be right?
>
> The 20K limit existed also before r32384 though. This is what I don't
> see in the code, what change r32384 did to the 20K limit.
> Can you point me to that?
>
> Cheers,
> Tomas
>
> On 2019-07-26 01:36, Jaime Hablutzel wrote:
>> Since r32384, around 20k queries to the DB are being produced from
>> PublisherQueueSessionBean#plainFifoTryAlwaysLimit100EntriesOrderByTimeCreated
>> if it succeds at least once in publishing qeued data.
>>
>> Below is a patch where it is worth noting that the 20K limit has been
>> removed as it doesn't make sense anymore cause the transactions are
>> currently circumscribed to the chunks of 100 records processed with each
>> call to PublisherQueueSessionBean#doChunk:
>>
>>
>> ---
>> modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionLocal.java
>> (revision 32884)
>> +++
>> modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionLocal.java
>> (date 1564089225333)
>> @@ -102,12 +102,10 @@
>> /**
>> * Intended for use from PublishQueueProcessWorker.
>> *
>> - * Publishing algorithm that is a plain fifo queue, but limited to
>> selecting entries to republish at 100 records at a time. It will select
>> from the database for this particular publisher id, and process
>> + * Publishing algorithm that is a plain fifo queue, but limited to
>> selecting entries to publish at 100 records at a time. It will select
>> from the database for this particular publisher, and process
>> * the record that is returned one by one. The records are ordered
>> by date, descending so the oldest record is returned first.
>> * Publishing is tried every time for every record returned, with
>> no limit.
>> * Repeat this process as long as we actually manage to publish
>> something this is because when publishing starts to work we want to
>> publish everything in one go, if possible.
>> - * However we don't want to publish more than 20000 certificates
>> each time, because we want to commit to the database some time as well.
>> - * Now, the OCSP publisher uses a non-transactional data source so
>> it commits every time so...
>> */
>> PublishingResult
>> plainFifoTryAlwaysLimit100EntriesOrderByTimeCreated(AuthenticationToken
>> admin, BasePublisher publisher);
>>
>> ---
>> modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublishingResult.java
>> (revision 32884)
>> +++
>> modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublishingResult.java
>> (date 1564088515443)
>> @@ -18,8 +18,7 @@
>>
>> /**
>> * Represents a return type for publishing operations. Successes and
>> failures are stored in sets containing the fingerprints of what's being
>> stored
>> - * due to the fact that the publisher will by default retry 20,000
>> times on a failed attempt, so the "failures" will be per attempted
>> certificate
>> - * and not all attempts.
>> + * where the "failures" will be per attempted certificate and not all
>> attempts.
>> *
>> * @version $Id: PublishingResult.java 32392 2019-05-22 18:45:21Z
>> mikekushner $
>> *
>> ---
>> modules/ejbca-ejb/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionBean.java
>> (revision 32884)
>> +++
>> modules/ejbca-ejb/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionBean.java
>> (date 1564093297113)
>> @@ -278,12 +278,11 @@
>> PublishingResult result = new PublishingResult();
>> // Repeat this process as long as we actually manage to publish
>> something
>> // this is because when publishing starts to work we want to
>> publish everything in one go, if possible.
>> - // However we don't want to publish more than 20000
>> certificates each time, because we want to commit to the database some
>> time as well.
>> - int totalcount = 0;
>> + PublishingResult chunkResult;
>> do {
>> - result.append(publisherQueueSession.doChunk(admin, publisher));
>> - totalcount += result.getSuccesses();
>> - } while ((result.getSuccesses() > 0) && (totalcount < 20000));
>> + chunkResult = publisherQueueSession.doChunk(admin, publisher);
>> + result.append(chunkResult);
>> + } while (chunkResult.getSuccesses() > 0);
>> return result;
>> }
>>
>> @@ -296,7 +295,6 @@
>>
>> /**
>> * @param admin
>> - * @param publisherId
>> * @param publisher
>> *
>> * @return how many publishing operations that succeeded and failed */
>>
>>
>>
>> --
>>
>> Jaime Hablutzel - +51 994690880
>>
>>
>> _______________________________________________
>> Ejbca-develop mailing list
>> Ejb...@li...
>> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>>
>
>
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>
|
