Jython

On Thu, 03 Apr 2008 01:19:17 -0700, Michael March scripst:
> How am I supposed to convert that to string so I can write it to disk?

If you really want string, then what

outF = file("filename","wb")
while <some condition>:
   partialFile=docfile.read(sessionid, documentid, offset, 16384)
   print >>outF,str(partialFile)

does? If the class of docfile.read() is some reasonable class it should 
support __str__() method (which is what str() uses).

Otherwise if you rather more care for the most faithful persistent 
storage of the content, pickle module is your friend (in standard jython).

Matej

-- 
The content of this message is licensed under a Creative Commons 
Attribution 3.0 License, Some Rights Reserved.
http://creativecommons.org/licenses/by/3.0/us/

Hmm.. the array is supposed to be a fragment of a binary file (like an
Excel spreadsheet). Will the method you detail for that work for that?


>
> > How am I supposed to convert that to string so I can write it to disk?
>
>  If you really want string, then what
>
>  outF = file("filename","wb")
>  while <some condition>:
>
>    partialFile=docfile.read(sessionid, documentid, offset, 16384)
>    print >>outF,str(partialFile)
>
>  does? If the class of docfile.read() is some reasonable class it should
>  support __str__() method (which is what str() uses).
>
>  Otherwise if you rather more care for the most faithful persistent
>  storage of the content, pickle module is your friend (in standard jython).
>
>  Matej

To get a little more specific.. the docs say this about the method:

   Returns:
       Data read from the file as a byte array. Returns null on file end.

... which is supposed to be a part of of binary download..

When I 'print' the array I receive after calling that method it looks
something like this:

   array('b', [-48, -49, 17, -95, etc, etc....])

I guess I have to convert that to a string so I can write it to a binary file?


On Thu, Apr 3, 2008 at 7:29 AM, Michael March <mmarch@...> wrote:
> Hmm.. the array is supposed to be a fragment of a binary file (like an
> Excel spreadsheet). Will the method you detail for that work for that?
>
>
>
>
>
> >
> > > How am I supposed to convert that to string so I can write it to disk?
> >
> >  If you really want string, then what
> >
> >  outF = file("filename","wb")
> >  while <some condition>:
> >
> >    partialFile=docfile.read(sessionid, documentid, offset, 16384)
> >    print >>outF,str(partialFile)
> >
> >  does? If the class of docfile.read() is some reasonable class it should
> >  support __str__() method (which is what str() uses).
> >
> >  Otherwise if you rather more care for the most faithful persistent
> >  storage of the content, pickle module is your friend (in standard jython).
> >
> >  Matej
>

[possible resend.. Email issues .. sorry]

To get a little more specific.. the docs say this about the method:

  Returns:
      Data read from the file as a byte array. Returns null on file end.

... which is supposed to be a part of of binary download..

When I 'print' the array I receive after calling that method it looks
something like this:

  array('b', [-48, -49, 17, -95, etc, etc....])

I guess I have to convert that to a string so I can write it to a binary file?
- Hide quoted text -


On Thu, Apr 3, 2008 at 7:29 AM, Michael March <mmarch@...> wrote:
> Hmm.. the array is supposed to be a fragment of a binary file (like an
> Excel spreadsheet). Will the method you detail for that work for that?
>
>
>
>
>
> >
> > > How am I supposed to convert that to string so I can write it to disk?
> >
> >  If you really want string, then what
> >
> >  outF = file("filename","wb")
> >  while <some condition>:
> >
> >    partialFile=docfile.read(sessionid, documentid, offset, 16384)
> >    print >>outF,str(partialFile)
> >
> >  does? If the class of docfile.read() is some reasonable class it should
> >  support __str__() method (which is what str() uses).
> >
> >  Otherwise if you rather more care for the most faithful persistent
> >  storage of the content, pickle module is your friend (in standard jython).
> >
> >  Matej

[Michael]
> I'm using a Java API for SourceForge enterprise.. if you want to
>  'grab' a file from it you make a call that grabs the file
>  incrementally:
>
>       partialFile=docfile.read(sessionid, documentid, offset, 16384)
>
>  .. partialFile is suppose to be a Java byte array that contains up to
>  16384 bytes of that file I'm downloading..
>
>  How am I supposed to convert that to string so I can write it to disk?

This code should be informative

# -------------------------------
def the_python_way(byte_array, encoding="utf-8"):
  char_array = [chr(byte) for byte in byte_array]
  result = "".join(char_array)
  if encoding:
    return result.decode(encoding)
  else:
    return result

def the_java_way(binary_data, encoding='utf-8'):
  from java.lang import String
  if encoding:
    return String(binary_data)
  else:
    return String(binary_data, encoding)

def get_binary_data():
  # download from sourceforge
  # binary_array = docfile.read(sessionid, documentid, offset, 16384)
  import jarray
  return jarray.array([ord(b) for b in "My  binary data\n"], 'b')

def timeit(binary_data, method, iterations):
  import time
  start = time.time()
  for ix in xrange(iterations):
    data_string = method(binary_data)
  total_msec = time.time() - start
  print "Method %s, total %1.3lf msec, per iter %1.6lf msec" % \
         (str(method), total_msec, total_msec/iterations)

def save_data(filename, data_string, encoding="utf-8"):
  if encoding:
    import codecs
    outfile = codecs.open(filename, "wb", encoding)
  else:
    outfile = open(filename, "wb")
  outfile.write(data_string)
  outfile.close()

if __name__ == "__main__":
#  timeit(get_binary_data(), the_python_way, 100000)
#  timeit(get_binary_data(), the_java_way, 100000)
  save_data("bytes.dat", the_java_way(get_binary_data()))

# -------------------------------

As I expected, the java way (using java.lang.String, which is natively
implemented) is over three times faster than the standard python way.

I think the python way is easier to grok though.

HTH,

Alan.

[Alan]
>  As I expected, the java way (using java.lang.String, which is natively
>  implemented) is over three times faster than the standard python way.

Of course, I should have removed the import from the java version, it
was slowing things down.

When you write the java method this way

from java.lang import String

def the_java_way(binary_data, encoding='utf-8'):
  # from java.lang import String
  if encoding:
    return String(binary_data)
  else:
    return String(binary_data, encoding)

The java way is over 7 times faster.

Alan.

Two things:

1. I'm a moron.
2. You ppl are AWESOME!

That very simple function took care of everything.

Thanks!

On Thu, Apr 3, 2008 at 1:23 PM, Alan Kennedy <jython-users@...> wrote:
> [Michael]
>
> > I'm using a Java API for SourceForge enterprise.. if you want to
>  >  'grab' a file from it you make a call that grabs the file
>  >  incrementally:
>  >
>  >       partialFile=docfile.read(sessionid, documentid, offset, 16384)
>  >
>  >  .. partialFile is suppose to be a Java byte array that contains up to
>  >  16384 bytes of that file I'm downloading..
>  >
>  >  How am I supposed to convert that to string so I can write it to disk?
>
>  This code should be informative
>
>  # -------------------------------
>  def the_python_way(byte_array, encoding="utf-8"):
>   char_array = [chr(byte) for byte in byte_array]
>   result = "".join(char_array)
>   if encoding:
>     return result.decode(encoding)
>   else:
>     return result
>
>  def the_java_way(binary_data, encoding='utf-8'):
>   from java.lang import String
>   if encoding:
>     return String(binary_data)
>   else:
>     return String(binary_data, encoding)
>
>  def get_binary_data():
>   # download from sourceforge
>   # binary_array = docfile.read(sessionid, documentid, offset, 16384)
>   import jarray
>   return jarray.array([ord(b) for b in "My  binary data\n"], 'b')
>
>  def timeit(binary_data, method, iterations):
>   import time
>   start = time.time()
>   for ix in xrange(iterations):
>     data_string = method(binary_data)
>   total_msec = time.time() - start
>   print "Method %s, total %1.3lf msec, per iter %1.6lf msec" % \
>          (str(method), total_msec, total_msec/iterations)
>
>  def save_data(filename, data_string, encoding="utf-8"):
>   if encoding:
>     import codecs
>     outfile = codecs.open(filename, "wb", encoding)
>   else:
>     outfile = open(filename, "wb")
>   outfile.write(data_string)
>   outfile.close()
>
>  if __name__ == "__main__":
>  #  timeit(get_binary_data(), the_python_way, 100000)
>  #  timeit(get_binary_data(), the_java_way, 100000)
>   save_data("bytes.dat", the_java_way(get_binary_data()))
>
>  # -------------------------------
>
>  As I expected, the java way (using java.lang.String, which is natively
>  implemented) is over three times faster than the standard python way.
>
>  I think the python way is easier to grok though.

[Alan]
>  When you write the java method this way

>  from java.lang import String
>
>  def the_java_way(binary_data, encoding='utf-8'):
>   # from java.lang import String
>   if encoding:
>     return String(binary_data)
>   else:
>     return String(binary_data, encoding)

D'oh! Another bug in the java method. It should look like this

#-----------------------------------
from java.lang import String
def the_java_way(binary_data, encoding='utf-8'):
 if encoding:
   return String(binary_data, encoding)
 else:
   return String(binary_data)
#-----------------------------------

It didn't cause problems in my example because my sample binary data
contained only ASCII characters.

Alan.

On Fri, 04 Apr 2008 11:12:57 +0100, Alan Kennedy scripst:
> from java.lang import String
> def the_java_way(binary_data, encoding='utf-8'):
>  if encoding:
>    return String(binary_data, encoding)
>  else:
>    return String(binary_data)

This looks bad to me -- because encoding has default value, "else" branch 
of "if" will IMHO never be executed, and the string without encoding will 
be automagically converted to UTF-8 (which is a good thing IMHO, but 
probably not what was intended).

Matej

-- 
The content of this message is licensed under a Creative Commons 
Attribution 3.0 License, Some Rights Reserved.
http://creativecommons.org/licenses/by/3.0/us/

[Alan]
>> from java.lang import String
>> def the_java_way(binary_data, encoding='utf-8'):
>>  if encoding:
>>    return String(binary_data, encoding)
>>  else:
>>    return String(binary_data)

[Matej]
>  This looks bad to me -- because encoding has default value, "else" branch
>  of "if" will IMHO never be executed

No, it works exactly as expected; call it as follows

data_string = the_java_way(binary_data, None)

[Matej]
> and the string without encoding will
>  be automagically converted to UTF-8 (which is a good thing IMHO, but
>  probably not what was intended).

Yes, the default case is to treat the bytes as being encoded in utf-8.

Making it the default assumption that bytes have some encoding is the
right thing to do, because all bytes representing strings have some
encoding. Using the above function definition forces the user to
specifically state that there is NO encoding, which is rarely true;
thus the programmer is forced to do some serious thinking about where
the data comes from and what encoding it is in, which is exactly what
I intended for the function.

Making the default encoding UTF-8 is a sensible default, and also
exactly what I intended to do.

Alan.

On 2008-04-04, 12:57 GMT, Alan Kennedy wrote:
> data_string = the_java_way(binary_data, None)

Of course, you are right. I forgot this possibility.

Matej

Ah.. Is why my byte counts were correct but my files were corrupted?

On Fri, Apr 4, 2008 at 3:12 AM, Alan Kennedy <jython-users@...> wrote:
> [Alan]
>
> >  When you write the java method this way
>
>  >  from java.lang import String
>  >
>  >  def the_java_way(binary_data, encoding='utf-8'):
>  >   # from java.lang import String
>  >   if encoding:
>  >     return String(binary_data)
>  >   else:
>  >     return String(binary_data, encoding)
>
>  D'oh! Another bug in the java method. It should look like this
>
>  #-----------------------------------
>
> from java.lang import String
>  def the_java_way(binary_data, encoding='utf-8'):
>   if encoding:
>    return String(binary_data, encoding)
>   else:
>    return String(binary_data)
>  #-----------------------------------
>
>  It didn't cause problems in my example because my sample binary data
>  contained only ASCII characters.
>
>  Alan.
>

[Michael]
> Ah.. Is why my byte counts were correct but my files were corrupted?

Define corrupted. Was everything corrupted, or just certain characters?

Which version of the function did you use?

Give an example of some data which you put through the function.

Alan.

> [Michael]
>
> > Ah.. Is why my byte counts were correct but my files were corrupted?
>
>  Define corrupted. Was everything corrupted, or just certain characters?

I don't know exactly what character.. i just know that the size of the
files (remote and the d/l'd one) were exact but the md5sum of the two
were different.

>  Which version of the function did you use?

The first version (Java).

>  Give an example of some data which you put through the function.

An Excel and Word file(s).

>  Alan.
>

[Alan]
>>  Give an example of some data which you put through the function.

[Michael]
>  An Excel and Word file(s).

Ah, binary data.

In these cases, you must call both the decode and encode (i.e. the
save) functions with an encoding parameter of None.

That way the data will be treated as raw bytes.

Alan.