#3 OSError: [Errno 24] Too many open files

[Anonymous]

I'm running duplicity, which calls GnuPGInterface to encrypt its backup files. I'm running Python 2.5 and GnuPGInterface 0.3.2, so I think I have the latest releases. The underlying OS is Mac OS X 10.4.10.

The problem appears to be that a pipe call fails with an error OSError: [Errno 24] Too many open files. I'm guessing that duplicity or GnuPGInterface isn't closing pipes it should, which doesn't matter on most systems but does here, but I don't know python well enough to know where to start to look for this.

The program fails with the following error:

.....
Running 'scp -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no /tmp/tmp0uHsAr x@y:backup/hourly/duplicity-inc.2007-12-10T00:33:02-04:00.to.2007-12-13T10:02:06-04:00.vol1.difftar.gpg' (attempt #1)
Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 378, in <module>
if __name__ == "__main__": main()
File "/usr/local/bin/duplicity", line 372, in main
else: incremental_backup(sig_chain)
File "/usr/local/bin/duplicity", line 175, in incremental_backup
bytes_written = write_multivol("inc", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 83, in write_multivol
globals.gpg_profile,globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 193, in GPGWriteFile
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 100, in __init__
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/GnuPGInterface.py", line 357, in run
create_fhs, attach_fhs)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/GnuPGInterface.py", line 388, in _attach_fork_exec
pipe = os.pipe()
OSError: [Errno 24] Too many open files
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
(and so on...)

[Frank Tobin]

Logged In: YES
user_id=7577
Originator: NO

Too many open files usually requires many, many files to be open (e.g., maybe 4k on a standard linux box). Is GnuPGInterface being called many times, or just a few? Since it is running inside another process, I have to suspect that not only it, but others are all adding up to reach the process limit. You can check your file descriptor limit with "ulimit -aH"

[Kenneth Loafman]

Logged In: YES
user_id=30149
Originator: NO

I'm the maintainer for duplicity. This is happening on another Macintosh as well. The symptoms we're seeing is that multiple gpg processes are left behind, each requiring multiple fd's. This symptom is not being seen on Linux or Windows. Duplicity uses your process.wait() when closing. I looked and that wraps os.waitpid(process.pid, 0) and this seems to be working under other systems.

[Kevin Crowston]

Logged In: YES
user_id=274483
Originator: NO

(I was the initial bug reporter but forgot to login before submitting...) The default file limit on MacOS X is only 256, so a work around is to set the ulimit higher. But it really is just a work around, since the gpg processes don't seem to go away after they're used, so it ends up running out of file handles after spawning lots of processes (a hundred with the default setting or a couple hundred with a higher limit).

[Kenneth Loafman]

You can close this one. It has nothing to do with GnuPGInterface. The issue is with duplicity. A very long series of incremental backups will keep the backups open until completion.