Re: [Ejbca-develop] Around 20K queries to the DB are being produced even if there is only one successful publishing event

[Tomas G. <to...@pr...>]

Hi,

I'm not sure I fully understand. The patch below simply removes the 20K
limit right, so if there were 100K rows needing to be re-puplished they
would all be right?

The 20K limit existed also before r32384 though. This is what I don't
see in the code, what change r32384 did to the 20K limit.
Can you point me to that?

Cheers,
Tomas

On 2019-07-26 01:36, Jaime Hablutzel wrote:
> Since r32384, around 20k queries to the DB are being produced from
> PublisherQueueSessionBean#plainFifoTryAlwaysLimit100EntriesOrderByTimeCreated
> if it succeds at least once in publishing qeued data.
> 
> Below is a patch where it is worth noting that the 20K limit has been
> removed as it doesn't make sense anymore cause the transactions are
> currently circumscribed to the chunks of 100 records processed with each
> call to PublisherQueueSessionBean#doChunk:
> 
> 
> ---
> modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionLocal.java
> (revision 32884)
> +++
> modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionLocal.java
> (date 1564089225333)
> @@ -102,12 +102,10 @@
>      /**
>       * Intended for use from PublishQueueProcessWorker.
>       *
> -     * Publishing algorithm that is a plain fifo queue, but limited to
> selecting entries to republish at 100 records at a time. It will select
> from the database for this particular publisher id, and process
> +     * Publishing algorithm that is a plain fifo queue, but limited to
> selecting entries to publish at 100 records at a time. It will select
> from the database for this particular publisher, and process
>       * the record that is returned one by one. The records are ordered
> by date, descending so the oldest record is returned first.
>       * Publishing is tried every time for every record returned, with
> no limit.
>       * Repeat this process as long as we actually manage to publish
> something this is because when publishing starts to work we want to
> publish everything in one go, if possible.
> -     * However we don't want to publish more than 20000 certificates
> each time, because we want to commit to the database some time as well.
> -     * Now, the OCSP publisher uses a non-transactional data source so
> it commits every time so...
>       */
>      PublishingResult
> plainFifoTryAlwaysLimit100EntriesOrderByTimeCreated(AuthenticationToken
> admin, BasePublisher publisher);
>  
> ---
> modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublishingResult.java
> (revision 32884)
> +++
> modules/ejbca-ejb-interface/src/org/ejbca/core/ejb/ca/publisher/PublishingResult.java
> (date 1564088515443)
> @@ -18,8 +18,7 @@
>  
>  /**
>   * Represents a return type for publishing operations. Successes and
> failures are stored in sets containing the fingerprints of what's being
> stored
> - * due to the fact that the publisher will by default retry 20,000
> times on a failed attempt, so the "failures" will be per attempted
> certificate
> - * and not all attempts.
> + * where the "failures" will be per attempted certificate and not all
> attempts.
>   *
>   * @version $Id: PublishingResult.java 32392 2019-05-22 18:45:21Z
> mikekushner $
>   *
> ---
> modules/ejbca-ejb/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionBean.java
> (revision 32884)
> +++
> modules/ejbca-ejb/src/org/ejbca/core/ejb/ca/publisher/PublisherQueueSessionBean.java
> (date 1564093297113)
> @@ -278,12 +278,11 @@
>          PublishingResult result = new PublishingResult();
>          // Repeat this process as long as we actually manage to publish
> something
>          // this is because when publishing starts to work we want to
> publish everything in one go, if possible.
> -        // However we don't want to publish more than 20000
> certificates each time, because we want to commit to the database some
> time as well.
> -        int totalcount = 0;
> +        PublishingResult chunkResult;
>          do {
> -            result.append(publisherQueueSession.doChunk(admin, publisher));
> -            totalcount += result.getSuccesses();
> -        } while ((result.getSuccesses() > 0) && (totalcount < 20000));
> +            chunkResult = publisherQueueSession.doChunk(admin, publisher);
> +            result.append(chunkResult);
> +        } while (chunkResult.getSuccesses() > 0);
>          return result;
>      }
>  
> @@ -296,7 +295,6 @@
>  
>      /**
>       * @param admin
> -     * @param publisherId
>       * @param publisher
>       *  
>       * @return how many publishing operations that succeeded and failed */
> 
> 
> 
> -- 
> 
> Jaime Hablutzel -  +51 994690880
> 
> 
> _______________________________________________
> Ejbca-develop mailing list
> Ejb...@li...
> https://lists.sourceforge.net/lists/listinfo/ejbca-develop
>