exporting can bring the server to its knees
Status: Beta
Brought to you by:
worden
Menu
▾
▴
#181 exporting can bring the server to its knees
When asked to export a project that has many prerequisite projects with very large working directories, it can grind for a long time running tar and gzip, ultimately fail, and never report a useful error message. It should fail more gracefully, providing an error message and making sure to clean up all the temp files it created.
Discussion
Anonymous
It seems like this tar operation should be time-limited in the same way make operations are. To do that for make operations, I ended up creating a C program (pe-make.c) to set a timer and kill its children if they run too long. So I could switch to using pe-make to make the tar files for export, maybe. That would provide the time limiting, and might also be sort of clean in some way. But pe-make is locked down by design such that it can only be used to invoke make, not anything else, and I'd rather keep it that way, so I'd have to construct make rules to do the tar/gzip operation.
This would probably require implementing an environment variable containing all the source filenames (https://sourceforge.net/tracker/index.php?func=detail&aid=3487370&group_id=366300&atid=1527388) and maybe one for all the project directories as well. Of course, that would also have other benefits.
It would also be helpful to provide a more flexible export interface (https://sourceforge.net/tracker/index.php?func=detail&aid=3103403&group_id=366300&atid=1527388) so people could choose to export the project without its prerequisites. And that form should ideally tell them how big each directory is, and include an estimate of what the upper size limit is.
In addition to a time limit, there should be a size limit. Given infinite speed, it would make sense to calculate the size of the files before doing the export and find out whether it's too big. This could really slow things down, on the other hand. Another option might be to use RLIMIT_FSIZE (see "man setrlimit") to make tar/gzip crash if the .tar.gz file gets too big.
So the way forward for this would be to experiment with writing make rules to reimplement the export operation; work out what environment variables they would need; implement those variables in WW/PE in such a way as to also satisfy the existing request for similar variables; switch to using the make rules for export; add RLIMIT_FSIZE to pe-make (but don't use it for make operations, only for exporting).