#79 sftp download leave .tmp files around when transfers fail.

[psilva]

There are files named "$file.tmp" appearing even when files are not successfully transferred (or at least, not deleted after try, perhaps a fix needed to the sftp method?)

In the subscribe config:

lock .tmp

Result:

ex:

-rw-rw-rw- 1 olduser oldgroup 0 Mar 8 18:46 testfile9999
-rw-rw-rw- 1 olduser oldgroup 0 Mar 8 20:18 testfile9999.tmp

in logs:

olduser@verona 76) grep testfile9999 sr_subscribe_ppp-eccc-cmod-policy_*2017-03-08
sr_subscribe_ppp-eccc-cmod-policy_0009.log.2017-03-08:2017-03-08 20:19:18,959 [INFO] msg_log received: 20170308185601.834 sftp://xferuser@myserver.mydomain//fs/site1/fs1/eccc/cmod/prod/testdir/testfile9999 topic=v02.post.fs.site1.fs1.eccc.cmod.prod.testdir.testfile9999 lag=4997.13 parts=1,0,1,0,0 mtime=20170308183445.151 atime=20170308183445.151 filename=testfile9999 to_clusters=DDSR.SCIENCE sum=0,74
sr_subscribe_ppp-eccc-cmod-policy_0009.log.2017-03-08:2017-03-08 20:19:18,968 [ERROR] Download failed sftp://xferuser@myserver.mydomain//fs/site1/fs1/eccc/cmod/prod/testdir/testfile9999. Type: <class 'filenotfounderror'="">, Value: [Errno 2] No such file
sr_subscribe_ppp-eccc-cmod-policy_0009.log.2017-03-08:2017-03-08 20:19:18,985 [ERROR] Download failed sftp://xferuser@myserver.mydomain//fs/site1/fs1/eccc/cmod/prod/testdir/testfile9999. Type: <class 'filenotfounderror'="">, Value: [Errno 2] No such file
sr_subscribe_ppp-eccc-cmod-policy_0009.log.2017-03-08:2017-03-08 20:19:19,037 [ERROR] Download failed sftp://xferuser@myserver.mydomain//fs/site1/fs1/eccc/cmod/prod/testdir/testfile9999. Type: <class 'filenotfounderror'="">, Value: [Errno 2] No such file
sr_subscribe_ppp-eccc-cmod-policy_0022.log.2017-03-08:2017-03-08 18:46:35,043 [INFO] msg_log received: 20170308173142.440 sftp://xferuser@myserver.mydomain//fs/site1/fs1/eccc/cmod/prod/testdir/testfile9999 topic=v02.post.fs.site1.fs1.eccc.cmod.prod.testdir.testfile9999 lag=4492.6 sum=0,77 mtime=20170308170505.902 atime=20170308170505.902 parts=1,0,1,0,0 to_clusters=DDSR.SCIENCE filename=testfile9999
sr_subscribe_ppp-eccc-cmod-policy_0022.log.2017-03-08:2017-03-08 18:46:35,789 [INFO] file_log downloaded to: /fs/cetus/fs2/mirror_hubs/science_migration/test/cmod/prod/testdir/testfile9999
sr_subscribe_ppp-eccc-cmod-policy_0036.log.2017-03-08:2017-03-08 20:18:06,334 [INFO] msg_log received: 20170308184414.106 sftp://xferuser@myserver.mydomain//fs/site1/fs1/eccc/cmod/prod/testdir/testfile9999 topic=v02.post.fs.site1.fs1.eccc.cmod.prod.testdir.testfile9999 lag=5632.23 to_clusters=DDSR.SCIENCE filename=testfile9999 sum=0,78 mtime=20170308183445.151 parts=1,0,1,0,0 atime=20170308183445.151</class></class></class>

@

sr_subscribe_ppp-eccc-cmod-policy_0036.log.2017-03-08:2017-03-08 20:18:06,516 [ERROR] Download failed sftp://xferuser@myserver.mydomain//fs/site1/fs1/eccc/cmod/prod/testdir/testfile9999. Type: <class 'filenotfounderror'="">, Value: [Errno 2] No such file
sr_subscribe_ppp-eccc-cmod-policy_0036.log.2017-03-08:2017-03-08 20:18:06,522 [ERROR] Download failed sftp://xferuser@myserver.mydomain//fs/site1/fs1/eccc/cmod/prod/testdir/testfile9999. Type: <class 'filenotfounderror'="">, Value: [Errno 2] No such file
sr_subscribe_ppp-eccc-cmod-policy_0036.log.2017-03-08:2017-03-08 20:18:06,570 [ERROR] Download failed sftp://xferuser@myserver.mydomain//fs/site1/fs1/eccc/cmod/prod/testdir/testfile9999. Type: <class 'filenotfounderror'="">, Value: [Errno 2] No such file</class></class></class>

My impression is that in the sftp method, you open the file for writing before try to download the file, and if the file doesn’t transfer properly, nothing cleans out the lock file used.

Dom

[psilva]

attempted patch... haven't done the work to test it, but the patch does no harm (passes existing test suite.)

[psilva]

I think this was fixed in March. (2.17.03aX)
have not really tested yet.

[psilva]

now: https://github.com/MetPX/sarracenia/issues/48