Trying to add ~50k PVs in one go fails in a surprising place
Brought to you by:
slac-mshankar
Trying to add ~50k PVs in one go fails in a surprising place. It seems to be trying to read my policies.py for each PV, which exhausts available FDs.
11840793 [MgmtArchivePVWorkflow1] ERROR org.epics.archiverappliance.mgmt.archivepv.ArchivePVState - Exception transitioning archive pv state for pv SR:C21-BI{BPM:4}Kx-SP in state METAINFO_OBTAINED java.io.FileNotFoundException: /etc/archappl/policies.py (Too many open files) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at org.epics.archiverappliance.config.DefaultConfigService.getPolicyText(DefaultConfigService.java:1384) at org.epics.archiverappliance.config.DefaultConfigService.computePolicyForPV(DefaultConfigService.java:1314) at org.epics.archiverappliance.mgmt.archivepv.ArchivePVState.run(ArchivePVState.java:96) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)
Ha! So this seems to be a consequence of leaking FDs from my policies.py, which is using the 'logging' module with RotatingFileHandler.
Perhaps a missing call to cleanup()?
http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html
Michael Davidsaver generated a patch for this that has been applied. This is a side-effect of Jython reusing a PySystemState instance which hangs on to the FD and the system eventually running out of FD's. Fix is to create a new PySystemState instance for each PythonInterpreter. Potential future enhancement is to cache the PythonInterpreter.