[TCLCORE] TIP #259: Making 'exec' Optionally Binary Safe

[Andreas L. <av...@lo...>]

 TIP #259: MAKING 'EXEC' OPTIONALLY BINARY SAFE 
================================================
 Version:      $Revision: 1.1 $
 Author:       Andreas Leitgeb <avl_at_logic.at>
 State:        Draft
 Type:         Project
 Tcl-Version:  8.5
 Vote:         Pending
 Created:      Monday, 12 December 2005
 URL:          http://purl.org/tcl/tip/259.html
 WebEdit:      http://purl.org/tcl/tip/edit/259
 Post-History: 

-------------------------------------------------------------------------

 ABSTRACT 
==========

 A new option shall be added to the command *exec*, that allows the user 
 to specify that input redirected from immediate data (using *<<*) 
 and/or the data received from the external command shall not undergo 
 any translation within Tcl. 

 MOTIVATION 
============

 External programs may expect binary data, or write out binary data, or 
 neither or even both. Whether a program reads/writes binary data or 
 platform-encoded data is generally specific to the particular program 
 and known by the programmer who intends to *exec* it from a Tcl script. 

 For example, a hexdump-utility expectably reads binary data and outputs 
 text. 

 DEFICIENCIES OF CURRENT STATE OF EXEC 
=======================================

 *Problem 1:* For passing string-data to external programs, *exec* now 
 behaves arguably incorrect, because it does not pass through \0-bytes. 

 *Problem 2:* For returning the result of external programs, *exec* 
 applies translations based on system-default encoding, which is OK in 
 most cases, except, of course, for programs that output binary data. 

 Problem 1 is actually a bug, but for compatibility reasons some 
 internal function cannot be changed, because it is believed to be used 
 by some extensions (although it is not officially exported), thus 
 blocking the fixing of the bug. This TIP goes beyond that, by not 
 having it fixed to some particular consistent behaviour, but to let the 
 script-developer decide on what is the right behaviour for his needs. 

 Problem 2 prevents the output of binary-outputting programs from being 
 correctly retrieved. 

 PROPOSAL FOR A NEW OPTION 
===========================

 The *exec* command already has an interface for options, and currently 
 supports one option *-keepnewline* and the end-of-options marker *--*. 
 This means that adding a new option will not adversely affect any 
 existing scripts. 

 This TIP proposes a new option, *-binary* /arg/. /arg/ can be either a 
 single boolean value: 

       if a boolean true then both input (if a *<<* redirection is 
       present) and return value are passed verbatim between Tcl and the 
       external program. 

       if a boolean false (which is the default) then behaviour would be 
       like it is now, except for input being \0-safe and 
       system-translation taking place as appropriate (e.g. 
       line-endings). 

 /arg/ can also be one of the keywords *in*, *out*, *both* (which is 
 equivalent to 1) or an empty string (equivalent to 0) for more readable 
 code. The directions are to be seen from external programs perspective. 

 If /arg/ is any true boolean value, *both* or *out*, then the option 
 *-keepnewline* is implied. 

 If some usage of *exec* does not use *<<* string-redirection, then the 
 *in*-bit has no visible effect. 

 For now, no binary flag is defined for stderr. This might be subject of 
 a future TIP or left out due to lack of need. 

 IMPLEMENTATION 
================

 No implementation exists right now, although it is possible that this 
 will change in near future. 

 An implementation would have to change these functions: Those functions 
 that are not modifyable due to them being used elsewhere, need to be 
 replaced by an extended version, and the old function can be changed to 
 call the new one with appropriate extra arguments, and eventually be 
 phased out. 

 Tcl_ExecObjCmd:
       Handle the new option and set new bits for the flags argument of 
       /Tcl_OpenCommandChannel/ or add a new bitset argument. 

 Tcl_OpenCommandChannel:
       Deal with the new bits, or with new argument and pass them/it on. 

 TclCreatePipeline:
       needs new arguments /flags/ (for solving bug #768678 along the 
       way) and pass new arguments to a new variant of 
       /TclpCreateTempFile/. 

 TclpCreateTempFile:
       both Unix and Win version currently get a dumb C-style 
       char-pointer with no length-information. They need two extra 
       arguments, a length and the appropriate binary-bit. Since 
       /TclpCreateTempFile/ is in the stubs-table, we definitely need a 
       new Version of it, e.g. /TclpCreateTempFileEx/, which will take 
       and use these new arguments. 

 COPYRIGHT 
===========

 This document has been placed in the public domain. 

-------------------------------------------------------------------------

 TIP AutoGenerator - written by Donal K. Fellows