Share

Python GnuPGInterface

Tracker: Bugs

5 OSError: [Errno 24] Too many open files - ID: 1850121
Last Update: Comment added ( crowston )

I'm running duplicity, which calls GnuPGInterface to encrypt its backup
files. I'm running Python 2.5 and GnuPGInterface 0.3.2, so I think I have
the latest releases. The underlying OS is Mac OS X 10.4.10.

The problem appears to be that a pipe call fails with an error OSError:
[Errno 24] Too many open files. I'm guessing that duplicity or
GnuPGInterface isn't closing pipes it should, which doesn't matter on most
systems but does here, but I don't know python well enough to know where to
start to look for this.

The program fails with the following error:

.....
Running 'scp -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no
/tmp/tmp0uHsAr
x@y:backup/hourly/duplicity-inc.2007-12-10T00:33:02-04:00.to.2007-12-13T10:
02:06-04:00.vol1.difftar.gpg' (attempt #1)
Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 378, in <module>
if __name__ == "__main__": main()
File "/usr/local/bin/duplicity", line 372, in main
else: incremental_backup(sig_chain)
File "/usr/local/bin/duplicity", line 175, in incremental_backup
bytes_written = write_multivol("inc", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 83, in write_multivol
globals.gpg_profile,globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 193,
in GPGWriteFile
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 100,
in __init__
File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packa
ges/GnuPGInterface.py", line 357, in run
create_fhs, attach_fhs)
File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packa
ges/GnuPGInterface.py", line 388, in _attach_fork_exec
pipe = os.pipe()
OSError: [Errno 24] Too many open files
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
gpg: Error writing to `-': Broken pipe
gpg: handle plaintext failed: file write error
(and so on...)


Nobody/Anonymous ( nobody ) - 2007-12-13 15:28

5

Open

None

Frank Tobin

None

None

Public


Comments ( 3 )




Date: 2007-12-14 15:10
Sender: crowston


(I was the initial bug reporter but forgot to login before submitting...)
The default file limit on MacOS X is only 256, so a work around is to set
the ulimit higher. But it really is just a work around, since the gpg
processes don't seem to go away after they're used, so it ends up running
out of file handles after spawning lots of processes (a hundred with the
default setting or a couple hundred with a higher limit).


Date: 2007-12-14 14:49
Sender: loafman


I'm the maintainer for duplicity. This is happening on another Macintosh
as well. The symptoms we're seeing is that multiple gpg processes are left
behind, each requiring multiple fd's. This symptom is not being seen on
Linux or Windows. Duplicity uses your process.wait() when closing. I
looked and that wraps os.waitpid(process.pid, 0) and this seems to be
working under other systems.



Date: 2007-12-13 22:12
Sender: ftobinProject Admin


Too many open files usually requires many, many files to be open (e.g.,
maybe 4k on a standard linux box). Is GnuPGInterface being called many
times, or just a few? Since it is running inside another process, I have
to suspect that not only it, but others are all adding up to reach the
process limit. You can check your file descriptor limit with "ulimit -aH"


Log in to comment.

Attached File

No Files Currently Attached

Change ( 1 )

Field Old Value Date By
assigned_to nobody 2007-12-13 22:13 ftobin