#10 Trying to add ~50k PVs in one go fails in a surprising place

[Michael Davidsaver]

Trying to add ~50k PVs in one go fails in a surprising place. It seems to be trying to read my policies.py for each PV, which exhausts available FDs.

11840793 [MgmtArchivePVWorkflow1] ERROR org.epics.archiverappliance.mgmt.archivepv.ArchivePVState  - Exception transitioning archive pv state for pv SR:C21-BI{BPM:4}Kx-SP in state METAINFO_OBTAINED
java.io.FileNotFoundException: /etc/archappl/policies.py (Too many open files)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at org.epics.archiverappliance.config.DefaultConfigService.getPolicyText(DefaultConfigService.java:1384)
        at org.epics.archiverappliance.config.DefaultConfigService.computePolicyForPV(DefaultConfigService.java:1314)
        at org.epics.archiverappliance.mgmt.archivepv.ArchivePVState.run(ArchivePVState.java:96)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

[Michael Davidsaver]

Ha! So this seems to be a consequence of leaking FDs from my policies.py, which is using the 'logging' module with RotatingFileHandler.

Perhaps a missing call to cleanup()?

http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html

[Murali Shankar]

Michael Davidsaver generated a patch for this that has been applied. This is a side-effect of Jython reusing a PySystemState instance which hangs on to the FD and the system eventually running out of FD's. Fix is to create a new PySystemState instance for each PythonInterpreter. Potential future enhancement is to cache the PythonInterpreter.