Re: [ciphertool-devel] next steps

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Alex,

On Wed, 2004-03-31 at 07:04, Alex Griffing wrote:
> Well here are a couple related situations I'd like to avoid:
> 
> 1)  I'm adding a new k2-keyed patristocrat to the list.  I've entered 
> the k2 keyword, the plaintext, the ciphertext, and the type.  However, 
> when I run the program it complains that there is no 'bacontext' 
> associated with this cipher entry.
> 
> 2)  I've added a new cipher type with the new attribute 'cipherfoo' and 
> I've written new code to deal with it.  I don't want to have to go back 
> and add 'cipherfoo=' to every cipher entry (of all types) in the list.

I think these problems can be easily avoided.  If the attributes are all
optional (except for ciphertext), then the file loader wouldn't complain
if any property was missing unless the file loader explicitly requires
certain property combinations (such as 'period' for type=vigenere). 
Here are the steps that a "read_file" function would perform:

1) Open the file for reading
2) Read the file line by line.
    a) If a line ends with '\', string the '\' and append the next line
to it.
    b) If a line starts with '#', discard it.
4) Split each line into key/value pairs.  Throw an error if a '=' is not
found anywhere on the line.
3) Group the lines by cipher id (for files containing multiple ciphers).
5) Iterate through each group
    a) Look for the ciphertext.  Throw an error if the ciphertext isn't
found.
    b) Perform basic validation of values.  For example, all periods
must be integers, or the cipher type must be one of a specific set of
known cipher types.
    c) Look for the cipher type.  Perform any type-specific validation
of parameters.  For example, the width for a route cipher must evenly
divide into the length of the ciphertext.  Another example is that the
bacontext can only appear with the baconian cipher.
    d) Populate some sort of cipher data structure with the set values.
    e) return some sort of cipher data structure containing the found
properties

The validation checks only check for very basic inconsistencies.  This
lets us store only the known information in the cipher file.  It allows
for easy addition of new properties since the validation can simply
ignore any properties that it does not recognize.  The following would
all be valid files:

-----
# This is a basic aristocrat file that hasn't been solved
cipher1.type=aristocrat
cipher1.ciphertext=ab cde fgh ijklm

-----
# This file contains a solved vigenere and an unsolved aristocrat
cipher1.type=vigenere
cipher1.ciphertext=abcdefghijklmn
cipher1.period=3
cipher1.key=dog
cipher1.keyword=dog
# Here is the aristocrat in the same file
cipehr2.type=aristocrat
cipher2.ciphertext=ab cde fgh ijklm

-----
# This file contains unknown ciphertext
cipher1.ciphertext=abcdefghijkl

-----
# This file introduces a new attribute
cipher1.type=aristocrat
cipher1.ciphertext=ab cde fgh ijklm
cipher1.new_attribute=anything can go here

By separating the cipher data from the validation code we can write
strict or loose validators.  New attributes can be added without
breaking existing validation code.  Inconsistent attribute combinations
(route with length 12 and width=5) can be handled appropriately. 
Unnecessary attributes can be ignored (such as the period for an
aristocrat) or generate a non-fatal warning.

Does this sound like something that would suit your needs?

--Mike

> Wart wrote:
> 
> >On Tue, 2004-03-30 at 20:43, Alex Griffing wrote:
> >  
> >
> >>>Major tasks:
> >>>
> >>>1) Add an "encode pt key" subcommand to all cipher types to make it easy
> >>>to encrypt plaintext using a given key.  This will make it possible to
> >>>write scripts that encode the same plaintext or different plaintext
> >>>samples in a variety of ways to analyze the statistical characteristics
> >>>of the encryption methods.
> >>>
> >>>2) Implement a new more portable file format for cipher data.  Some
> >>>discussion has already begun on this.  We need to continue the
> >>>discussion and come to a conclusion on the final format.
> >>>
> >>>      
> >>>
> >>These sound like projects that could be helpful for lots of people 
> >>working on classical cryptology.
> >>    
> >>
> >
> >The first project was spawned out of laziness.  I got annoyed by
> >hand-encoding the sample bifid cipher included with ciphertool.  :)
> >
> >As I might have mentioned before, the 
> >  
> >
> >>cipher file format problem seems like an almost contrived example for 
> >>what xml is for.  I've never used xml though, and I don't know if this 
> >>would be a sledgehammer vs. fly situation.
> >>    
> >>
> >
> >Having worked a lot with XML, I think it would be a little bit of
> >overkill for this situation.  XML is much more suited for highly
> >structured and hierarchical data.  Here we just have a simple list of
> >key/value pairs with no strong structure.
> >
> >The big problem with using XML, though, is the amount of work required
> >to read/write it.  XML parsers do exist for Tcl.  But the API for using
> >XML parsers in general tends to be somewhat verbose.  And the use of XML
> >would add another dependency on an external software package.  If
> >possible, I like to keep the number of external dependencies to a
> >minimum.
> >
> >The Java property file syntax might be better suited for this data.  The
> >format for this file type is a set of key/value pairs separated by a
> >single '=' character.  Keys can only contain alphanumeric characters,
> >including '.', '-', and '_', but not '='.  Values can contain any
> >character.  A backslash '\' at the end of a line indicates that the
> >value continues on to the next line.  For example:
> >
> >type=aristocrat
> >ciphertext=ab cde fgh ijklm
> >plaintext=my dog \
> >has fleas
> >key=abcde fleas
> >title=Sample
> >author=wart
> >
> >If we want to support multiple ciphers in a single file then we can
> >prefix each key name with a unique, but arbitrary, cipher id:
> >
> >cipher1.type=aristocrat
> >cipher1.ciphertext=ab cde efghi
> >...
> >cipher2.type=vigenere
> >cipher2.period=6
> >
> >  
> >
> >> type
> >> period
> >> plaintext
> >> ciphertext
> >> key
> >> keyword
> >> author
> >> title
> >> language
> >>    
> >>
> >
> >I'll add to this:
> >
> >for aristocrat types:
> >
> >k1key
> >fixedkey
> >k2key
> >
> >for morse types (pollux, morbit, fractionated morse):
> >
> >morsetext
> >
> >for baconian types:
> >
> >bacontext
> >
> >for progressive types:
> >
> >progressionEncoding
> >progressionIncrement
> >
> >  
> >
> >>might all fit in well as attributes.  Some of these could be forbidden 
> >>from use in certain cipher types (eg. period/aristocrat).  Others could 
> >>be mandatory (eg. period/vigenere).  Author and title could, for 
> >>example, always be optional regardless of cipher type.
> >>    
> >>
> >
> >Instead of being forbidden, couldn't they just be ignored by the
> >function that reads the file?  For example, if an aristocrat is loaded,
> >then the period field would just get ignored.
> >
> >All of these 
> >  
> >
> >>conditions could be enforced by the xml DTD (document type definition)
> >>    
> >>
> >
> >xml dtds (and xml schema) introduce a little more complexity.  If you
> >don't use a validating xml parser then the DTD and schema aren't used
> >and are pretty much useless.  If you use a validating xml parser when
> >you read the files, then the DTD or schema must be made available to the
> >parser.  This means they have to be made available in a well-known
> >location.  This is all entirely possible, but I feel that it's a little
> >bit of overkill for ciphertool.
> >
> >The property file syntax can also be validated, but the validation would
> >have to be part of the function that reads the file.  I don't think
> >that's such a bad thing.  I'm not sure that there's too much we need to
> >validate anyway.  The only real required field that I see is the
> >'ciphertext' field.  I envision using the files to store everything from
> >unidentified ciphertext (type is 'unknown' or missing), to half-solved
> >ciphers (type is known, period is '0' or missing), to completed ciphers
> >(keyword is known). In some cases a field might be required but not
> >known (the period for an unsolved vigenere for example).
> >
> >On the other hand, having stricter validation of 
> > 
> >  
> >
> >>Or it could all be in plain text files which might make more sense if 
> >>not many people are using it.
> >>    
> >>
> >
> >I think the property file format would be a little better suited for our
> >purpose.  I could probably still be convinced to make more fields
> >required or make more fields required only for certain cipher types
> >(period, for example).  It's getting late, I'll sleep on it.
> >
> >--Wart
> >
> >
> >
> >-------------------------------------------------------
> >This SF.Net email is sponsored by: IBM Linux Tutorials
> >Free Linux tutorial presented by Daniel Robbins, President and CEO of
> >GenToo technologies. Learn everything from fundamentals to system
> >administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> >_______________________________________________
> >ciphertool-devel mailing list
> >cip...@li...
> >https://lists.sourceforge.net/lists/listinfo/ciphertool-devel
> >
> >
> >
> >  
> >
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> _______________________________________________
> ciphertool-devel mailing list
> cip...@li...
> https://lists.sourceforge.net/lists/listinfo/ciphertool-devel