Menu
▾
▴
[Dpr-development] [ dpr-Bugs-3612956 ] Job Normalisation step very slow for archive file
|
From: SourceForge.net <no...@so...> - 2013-05-16 01:06:56
|
Bugs item #3612956, was opened at 2013-05-08 22:07 Message generated for change (Comment added) made by terryoneill You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=676456&aid=3612956&group_id=116934 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Preservation Group: None Status: Open Resolution: Fixed Priority: 6 Private: No Submitted By: Terry O'Neill (terryoneill) Assigned to: Kirti Chennareddy (kirtic) Summary: Job Normalisation step very slow for archive file Initial Comment: The Transfer Job Normalisation step and Reprocessing Job Normalisation step are both very slow when processing a file that results in a lot of Archival Information Packages (i.e. Xena files) being creating. When running with show_sql set to true (hibernate option) it is apparent that there are multiple unnecessary updates of the aip_creation_record table that seem to increase in number the more AIPs that have been processed for a single input data object. ---------------------------------------------------------------------- >Comment By: Terry O'Neill (terryoneill) Date: 2013-05-15 18:06 Message: Fixed advanced normalisation issue on performance branch and updated bug tag to the commit of this fix. This change returns the cascading options to the ReprocessingJobAIPCreationRecord and TransferJobAIPCreationRecord classes but alters the PFDataObjectXenaProcessingRecord class so that the getAipCreationRecords() function only cascades for Delete and Replicate (rather than for All). This will mean that Updates as done in normalisation steps will cascade from the AIP Creation Records to the PF Data Object Xena Processing Record but will not cascade further. This is much slower than the original fix but still much faster than the code prior to any fixes. Each update of an AIP Creation Record will now take a roughly similar period of time. Prior to any changes for fixing this bug these updates would take progressively longer because the cascade from the PF Data Object Xena Processing Record would go back to all prevously processed AIP Creation Records for the input Data Object which would result in each successive AIP created for a single Data Object taking more and more time. Note that there is still a lot of scope for performance improvement in this area. For testing it may be desireable to turn on sql output from hibernate. This can be by adding the following line to hibernate.cfg.xml after the <session-factory> tag: <property name="show_sql">true</property> ---------------------------------------------------------------------- Comment By: Kirti Chennareddy (kirtic) Date: 2013-05-13 16:51 Message: Testing on performance branch- DPR 5.1.2 At the advanced normalization step, if you select an entry that has been normalized and click on normalize selected, DPR fails silently with below error in the console, Exception occurred during event dispatching: org.hibernate.TransientObjectException: object references an unsaved transient instance - save the transient instance before flushing: au.gov.naa.digipres.dpr.model.transferjob.PFDataObjectXenaProcessingRecord at org.hibernate.engine.ForeignKeys.getEntityIdentifierIfNotUnsaved(ForeignKeys.java:243) at org.hibernate.type.EntityType.getIdentifier(EntityType.java:456) at org.hibernate.type.ManyToOneType.isDirty(ManyToOneType.java:265) at org.hibernate.type.ManyToOneType.isDirty(ManyToOneType.java:275) at org.hibernate.type.TypeHelper.findDirty(TypeHelper.java:295) at org.hibernate.persister.entity.AbstractEntityPersister.findDirty(AbstractEntityPersister.java:3403) at org.hibernate.event.def.DefaultFlushEntityEventListener.dirtyCheck(DefaultFlushEntityEventListener.java:520) at org.hibernate.event.def.DefaultFlushEntityEventListener.isUpdateNecessary(DefaultFlushEntityEventListener.java:230) at org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:154) at org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:219) at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:99) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216) at au.gov.naa.digipres.dpr.dao.hibernate.HibernateArchivalInformationPackageDAO.saveAIPCreationRecord(HibernateArchivalInformationPackageDAO.java:93) at au.gov.naa.digipres.dpr.task.step.TransferJobNormalisationStep.createAIPCreationRecord(TransferJobNormalisationStep.java:558) at au.gov.naa.digipres.dpr.task.step.TransferJobNormalisationStep.saveSuccessfulNormalisationResults(TransferJobNormalisationStep.java:462) at au.gov.naa.digipres.dpr.task.step.TransferJobNormalisationStep.saveNormalisationResults(TransferJobNormalisationStep.java:407) at au.gov.naa.digipres.dpr.task.step.TransferJobNormalisationStep.normaliseDataObject(TransferJobNormalisationStep.java:347) at au.gov.naa.digipres.dpr.ui.task.step.AdvancedNormalisationDialog.normaliseSelection(AdvancedNormalisationDialog.java:682) at au.gov.naa.digipres.dpr.ui.task.step.AdvancedNormalisationDialog.access$100(AdvancedNormalisationDialog.java:95) at au.gov.naa.digipres.dpr.ui.task.step.AdvancedNormalisationDialog$3.actionPerformed(AdvancedNormalisationDialog.java:239) at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2012) at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2335) at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:404) at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259) at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252) at java.awt.Component.processMouseEvent(Component.java:6268) at javax.swing.JComponent.processMouseEvent(JComponent.java:3267) at java.awt.Component.processEvent(Component.java:6033) at java.awt.Container.processEvent(Container.java:2045) at java.awt.Component.dispatchEventImpl(Component.java:4629) at java.awt.Container.dispatchEventImpl(Container.java:2103) at java.awt.Component.dispatchEvent(Component.java:4455) at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4633) at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4297) at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4227) at java.awt.Container.dispatchEventImpl(Container.java:2089) at java.awt.Window.dispatchEventImpl(Window.java:2517) at java.awt.Component.dispatchEvent(Component.java:4455) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:649) at java.awt.EventQueue.access$000(EventQueue.java:96) at java.awt.EventQueue$1.run(EventQueue.java:608) at java.awt.EventQueue$1.run(EventQueue.java:606) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:105) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:116) at java.awt.EventQueue$2.run(EventQueue.java:622) at java.awt.EventQueue$2.run(EventQueue.java:620) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:105) at java.awt.EventQueue.dispatchEvent(EventQueue.java:619) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:275) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:200) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:194) at java.awt.Dialog$1.run(Dialog.java:1072) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:226) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:647) at java.awt.EventQueue.access$000(EventQueue.java:96) at java.awt.EventQueue$1.run(EventQueue.java:608) at java.awt.EventQueue$1.run(EventQueue.java:606) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:105) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:116) at java.awt.EventQueue$2.run(EventQueue.java:622) at java.awt.EventQueue$2.run(EventQueue.java:620) at java.security.AccessController.doPrivileged(Native Method) at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:105) at java.awt.EventQueue.dispatchEvent(EventQueue.java:619) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:275) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:200) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:185) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:177) at java.awt.EventDispatchThread.run(EventDispatchThread.java:138) ---------------------------------------------------------------------- Comment By: Terry O'Neill (terryoneill) Date: 2013-05-08 22:31 Message: Fixed on performance branch and assigned to Kirti to test. This fix involved removing all cascading options that were set for the ReprocessingJobAIPCreationRecord and TransferJobAIPCreationRecord classes on the getXenaProcessingRecord() function. Note that this function is used in a number of contexts so steps other than the Job Normalisation steps should also be tested to ensure no change in functionality at other steps. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=676456&aid=3612956&group_id=116934 |
