#28 support umask option for pxSender config

[Tom Kralidis]

When pxSender creates destination files, the sender configuration applies the chmod option to both directories and files.

It would be valuable/precise to add a umask config option which would apply the umask to dirs/files instead. Setting, say, chmod 755 (which is required for our workflow, given we create directories) is fine for directories, but not for files.

[psilva]

Is this about pxsender or sr_sender?

[Tom Kralidis]

At time of OP pxSender

[psilva]

the underlying SFTP protocol does not support umask, one can only set specific permissions via chmod. so if we do it, we need to do it explicitly ourselves, and apply chmod. to effect the result.

[Tom Kralidis]

Possible options:

• do nothing
• apply logic that, given a chmod value for files (existing config), determine what the appropriate directory value would be (i.e. assume that chmod-664 would apply 775 for dirs)
• create chmod config value for dirs, i.e. chmod_dir=775

[psilva]

Hey Murray... Take a look at this one for fixing in sarra. conclusion of discussions:
add chmod_dir option that is just like the existing chmod option, but it will be used when creating directories.

So then once you have added chmod_dir to sr_config.py, then you need to look wherever there are mkdir's in the various senders, and apply this setting instead of the chmod setting.

[Murray Rennie]

looks like mkdir's are only in sr_ftp.py and sr_sftp.py. make's things possibly easier

[Murray Rennie]

found self.chmod in sr_config.py. made a self_chmod_dir and set it to 775.

I also duplicated the entry in the option function

elif words0 == 'chmod': # See: function not actually implemented, stub of ftp support.
self.chmod = int(words[1])
n = 2

elif words0 == 'chmod_dir': # See: function not actually implemented, stub of ftp support.
self.chmod_dir = int(words[1])
n = 2

[Murray Rennie]

the sftp modules handles this chmod differently. First off it uses the paramiko module for sftp. secondly it sets the permissions on newely created directories to 775 already.

Will the chmod_dir value be changing or will users be allowed to modify this value in config files. Right now sr_sftp.py already sets newly created directories t0 775.

# mkdir
def mkdir(self, remote_dir):
    self.logger.debug("sr_sftp mkdir %s" % remote_dir)
    self.sftp.mkdir(remote_dir,0o775)

[Murray Rennie]

The other thing I have noticed is that the file sr_config.py is only imported into sr_ftp.py and sr_sftp.py when in test mode. They are not imported is regular use and therefore it looks like chmod_dir would be unavailable to them without some more changes

[Murray Rennie]

looks like I'll have to adapt what's in the test section to what I require.

[Murray Rennie]

Eric pointed out the in sr_ftp.py the parent portion of the initialization conatins the sr_config. makes my life a bit easier.

[Murray Rennie]

modified the mkdir function in sr_ftp.py to

# mkdir
def mkdir(self, remote_dir):
self.logger.debug("sr_ftp mkdir %s" % remote_dir)
self.ftp.mkd(remote_dir)
self.ftp.chmod(self.parent.chmod_dir,remote_dir)

[Murray Rennie]

in sr_sftp.py, it appears that its version of mkdir puts permission on as the directory is made.

I'm also not wure why Michel used octal numbers for this. I swapped in chmod_dir in place of those entries.

[psilva]

because the numbers need to be octal? unix/linux traditional permission bits are always octal...
755 is an octal thing. the bits will be wrong if you don´t allow for it.

[Murray Rennie]

I have found explainations online regarding paramiko and sftp. While the mode is listed as octal the mode parameter in mkdir is an int.

This is from http://www.math.uiuc.edu/~gfrancis/illimath/windows/aszgard_mini/pylibs/paramiko/docs/paramiko.SFTPClient-class.html#mkdir

mkdir(self, path, mode=511)
Create a folder (directory) named path with numeric mode mode. The default mode is 0777 (octal). On some systems, mode is ignored. Where it is used, the current umask value is first masked out.

Parameters:
    path - name of the folder to create
               (type=str) 
    mode - permissions (posix-style) for the newly-created folder
               (type=int)

I went with this when I modified the sftp mkdir to use my new parameter.

Only sr_config.py has been committed so far.

[Murray Rennie]

in sr_ftp.py I'm not sure my solution would be the best. I eventually went with

# mkdir
def mkdir(self, remote_dir):
    self.logger.debug("sr_ftp mkdir %s" % remote_dir)
    self.ftp.mkd(remote_dir)
    self.ftp.voidcmd('SITE CHMOD ' + str(self.parent.chmod_dir) + ' ' + remote_dir)

[Murray Rennie]

Tried pushing my sr_*ftp.py changes up. Don't know if they got there.

[psilva]

these changes were in 2.16.07a*

in a1 .. the changes are as is.. .problem: use of 0755 without octal setting caused all manner of issues.

in a3 ... the octal thing is fixed by Jun... (commit 2b24c8b96eb16d8c6b4ea78b09fc67f5328714a9)

so confirmed to work feeding operations from ddi.cmc