#46 sr_watch unsuccessful with some files

[Eric Funk]

I have an sr_watch on an 'incoming' directory which is fed various files via SFTP. I have an sr_sarra instance which grabs the files and then deletes them.

Theoretically the incoming directory should be empty or near empty with no old files in it, however I a bunch of old files sitting in the directory that the sr_sarra instance didn't seem to collect.

Looking in the sr_watch log I see the following:

2016-07-26 00:04:23,214 [DEBUG] sr_post cache_load
2016-07-26 00:04:23,214 [DEBUG] sr_poster cache_load
2016-07-26 00:04:23,214 [DEBUG] sr_poster cache_files_set
2016-07-26 00:04:23,214 [DEBUG] cache_file = /local/home/sarra/.cache/sarra/watch/urpx-files/_data_incoming_urp_209715200
2016-07-26 00:04:23,214 [DEBUG] load cache
2016-07-26 00:04:23,214 [DEBUG] sr_post watching
2016-07-26 00:04:23,215 [DEBUG] sr_post posting
2016-07-26 00:04:23,215 [DEBUG] sr_poster post_local_inplace
2016-07-26 00:04:23,215 [DEBUG] sr_poster post_local_file
2016-07-26 00:04:23,215 [DEBUG] sr_config set_sumalgo d
2016-07-26 00:04:23,215 [ERROR] [Errno 13] Permission denied: '//data/incoming/urp/201607260000~~CLOGZPPI,DOPVOL2,18,MPRATE:URP:WWW:RADAR:META'

It seems the sr_watch isn't gettting read permissions on the file and I guess thus can't create the checksum for the post call.

My guess is that sr_watch is seeing the file in between the time the file shows up and before the OS releases it for reading. sr_watch only tries to post a file once before it gives up and moves on and thus the file is left there.

Another error I am seeing in the logs that I am not sure is related or not, are a lot of messages like the following:

2016-07-26 08:40:16,765 [ERROR] File not found //data/incoming/urp/201607260840~~SVRWX,SvrWxParameters,AGL:URP:WHN:RADAR:META
2016-07-26 08:40:16,770 [ERROR] File not found //data/incoming/urp/201607260840~~CAPPI,1.0,AGL,MPRATE:URP:WHN:RADAR:META
2016-07-26 08:40:16,772 [ERROR] File not found //data/incoming/urp/201607260840~~PA,180,60,PA_CAPPI15_1HOURLY,MM_HR,MM:URP:WHN:RADAR:META
2016-07-26 08:40:16,773 [ERROR] File not found //data/incoming/urp/201607260840~~VVW,2.0,VIL,AGL,MPRATE:URP:XDR:RADAR:META
2016-07-26 08:40:16,775 [ERROR] File not found //data/incoming/urp/201607260840~~PA,1440,60,PA_CAPPI15_1HOURLY,MM_HR,MM:URP:WHN:RADAR:META
2016-07-26 08:40:16,776 [ERROR] File not found //data/incoming/urp/201607260840~~CAPPI,1.5,AGL,MPRATE:URP:WHK:RADAR:META
2016-07-26 08:40:16,778 [ERROR] File not found //data/incoming/urp/201607260840~~PA,60,10,PA_CAPPI10,CM_HR,CM:URP:WHN:RADAR:META

For referencemy sr_watch configuration is as follows:

broker amqp://LOCAL@some-server/
to DSR.WPG
url sftp://sarra@some-server/
path /data/incoming/urp
rename /RADAR/
events created|modified
loglevel debug

[Eric Funk]

Just to add. I set the sr_watch to trigger on just 'created' events to see what difference that made, and it seems like its missing a ton of files now. Thus it seems like its triggering a lot on 'modified' even though the files are new.

Also just a quick correction. The files are being transferred to the incoming directory via FTP and not SFTP

[Murray Rennie]

In my testing sr_watch only seems to pick up files when I run it in the foreground. When run as per normal there is no entry in the logs to indicate that files are being picked up (even with loglevel set to debug) and nothing shows up on the web site.

[Murray Rennie]

Okay. The watch does seem to upload (just retested that) but still no log entries. the loglevel is still debug.

[psilva]

now: https://github.com/MetPX/sarracenia/issues/51