tcl-core

[TCLCORE] TIP #345: Kill the '''identity'''encoding

[Alexandre F. <ale...@gm...>]

 TIP #345: KILL THE '''IDENTITY'''ENCODING 
===========================================
 Version:      $Revision: 1.1 $
 Author:       Alexandre Ferrieux <alexandre.ferrieux_at_gmail.com>
 State:        Draft
 Type:         Project
 Tcl-Version:  8.7
 Vote:         Pending
 Created:      Thursday, 05 February 2009
 URL:          http://purl.org/tcl/tip/345.html
 WebEdit:      http://purl.org/tcl/tip/edit/345
 Post-History: 

-------------------------------------------------------------------------

 ABSTRACT 
==========

 This TIP proposes to remove the 'identity' encoding which is the 
 Pandora's Box of invalid UTF-8 string representations. 

 BACKGROUND 
============

 The contract of string representations in Tcl states that the /bytes/ 
 field (the *strep*) of a Tcl_Obj must be a valid UTF-8 byte sequence. 
 Violating it leads at best to inconsistent and shimmer-sensitive string 
 comparisons. Fortunately, nearly all of the Tcl code takes careful 
 steps to enforce it. With one exception: the 'identity' encoding. 
 Indeed, this encoding allows any byte sequence to be copied verbatim 
 into the strep of a value, as a side-effect of a strep computation on a 
 ByteArray with [*encoding system*]=="identity", or through [*encoding 
 convertfrom identity*]. Hence an invalid UTF-8 sequence can easily make 
 it to the strep and start wreaking havoc. 

 PROPOSED CHANGE 
=================

 This TIP proposes to simply close that single window to the dark side. 

 RATIONALE 
===========

 The risk of compatibility breakage is inordinately mild in that case, 
 since it has only ever been documented in tcltest. 

 REFERENCE EXAMPLE 
===================

 See Bug 2564363 
 [<URL:https://sourceforge.net/support/tracker.php?aid=2564363>] 

 COPYRIGHT 
===========

 This document has been placed in the public domain. 

-------------------------------------------------------------------------

 TIP AutoGenerator - written by Donal K. Fellows 

Re: [TCLCORE] TIP #345: Kill the '''identity'''encoding

[Donald G P. <dg...@ni...>]

Alexandre Ferrieux wrote:
>  TIP #345: KILL THE '''IDENTITY'''ENCODING 
> ===========================================

Much to say on this topic, but my time is limited today.

For now just a bit of additional background.  See my commits
to Tcl and Tk from last October.  The use of the identity encoding
has been completely purged from Tk and its test suite.  The
use of the identity encoding as a place holder for the system
encoding during Tcl startup has been purged as well.

The main place the identity encoding is still in use is in
the Tcl test suite, both directly and through the command
[tcltest::bytestring] which is defined in terms of
[encoding convert* identity].

I think the next logical step is to convert the identity
encoding from one implemented directly in C code and in
the default sources of Tcl into one of the encodings that
is loaded on demand from a *.enc file.  Next Tcl can stop
distributing the identity.enc file, and those extensions or test
suites that still insist on having the encoding around
can distribute it themselves and use [encoding dirs] to
force Tcl to find it.

Getting rid of the identity encoding is a good thing, but it's
only a part of the larger task of enforcing the "contract"
we've evolved ourselves into of demanding that strings which
find their way to an objPtr->bytes must conform to our internal
encoding.  We didn't start out demanding that, but over time
more and more of our internals assume it to be true.

-- 
| Don Porter          Mathematical and Computational Sciences Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|