jython-checkins — For Jython checkin messages

[Jython-checkins] CVS: jython/Lib javapath.py,1.5,1.6

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython/Lib
In directory usw-pr-cvs1:/tmp/cvs-serv6706

Modified Files:
	javapath.py 
Log Message:
Replace forward slashes with os.sep when running on backward slash OS
(like windows).



Index: javapath.py
===================================================================
RCS file: /cvsroot/jython/jython/Lib/javapath.py,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** javapath.py	2000/01/12 04:05:13	1.5
--- javapath.py	2001/07/24 20:11:16	1.6
***************
*** 187,190 ****
--- 187,192 ----
      """Normalize path, eliminating double slashes, etc."""
      sep = os.sep
+     if sep == '\\':
+         path = path.replace("/", sep)
      curdir = os.curdir
      pardir = os.pardir

[Jython-checkins] CVS: bugtests test306.py,NONE,1.1

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/bugtests
In directory usw-pr-cvs1:/tmp/cvs-serv6553

Added Files:
	test306.py 
Log Message:
A windows test for a bug discovered by Ype Kingma.



--- NEW FILE: test306.py ---
"""
Test normcase.
"""

import support
import os

if os.sep == '\\': #only do this test on windows.
    p1 = os.path.normpath('e:\\someDir\\packag/modul.py')
    if p1 != 'e:\\someDir\\packag\\modul.py':
        raise support.TestError('Wrong normpath %s' % p1)

[Jython-checkins] CVS: jython README.txt,1.10,1.11

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython
In directory usw-pr-cvs1:/tmp/cvs-serv31736

Modified Files:
	README.txt 
Log Message:
Updated with important 2.1 incompatibilities.



Index: README.txt
===================================================================
RCS file: /cvsroot/jython/jython/README.txt,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** README.txt	2001/01/17 16:56:46	1.10
--- README.txt	2001/07/23 19:36:27	1.11
***************
*** 1,36 ****
! Welcome to Jython 2.0
  =======================
- 
- Jython is the successor to JPython. The Jython project was
- created in accordance with the CNRI JPython 1.1.x license, in
- order to ensure the continued existence and development of this
- important piece of Python software.  
- 
- 
- If you are a JPython-1.1 user, you'll want to take a look at some
- of the differences between JPython-1.1 and Jython 2.0.
- 
- In particulary the few places where backward compatibility have 
- been broken. 
- 
-     - The user configuration file is now called <user.home>/.jython
  
!     - The jar file containing all the jython classes is now called 
!       jython.jar.
! 
!     - Text files will pass data read and written through the default
!       codecs for the JVM. Binary files will write only the lower eight
!       bits of each unicode character.
! 
!     - arrays passed to java code will no longer autocoerce just 
!       because the elements can be autocoerced.
  
!     - The precedence of java loading have changed. Now the sys.path
!       is searched for python modules before the CLASSPATH and sys.path
!       is searched for java class and java packages.
  
!     - The \x escape have changed, now it will eat two hex characters
!       but never more. The behaviour matches CPython2.0
  
-     - The python.path property is appended to sys.path instead of
-       being inserted at position 1 in sys.path.
--- 1,13 ----
! Welcome to Jython 2.1
  =======================
  
! Important: The format of both compiled ($py.class) and frozen module
! has changed from version 2.0. It is necesary to delete existing
! $py.class files and recompile frozen applications when upgrading
! to Jython-2.1.
  
! Other backward incompatible changes include:
  
! - The case of module names are now important, even on case ignoring
!   filesystems like windows. This matches CPython behaviour.
  

[Jython-checkins] CVS: jython/org/python/core imp.java,2.48,2.49

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython/org/python/core
In directory usw-pr-cvs1:/tmp/cvs-serv30401

Modified Files:
	imp.java 
Log Message:
caseok(): Fix a bug where a foo.PY file could not be imported.



Index: imp.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/core/imp.java,v
retrieving revision 2.48
retrieving revision 2.49
diff -C2 -d -r2.48 -r2.49
*** imp.java	2001/07/17 20:34:35	2.48
--- imp.java	2001/07/23 19:30:44	2.49
***************
*** 274,277 ****
--- 274,278 ----
          if (o != null) return o;
  
+         int nlen = name.length();
          String pyName = name+".py";
          String className = name+"$py.class";
***************
*** 299,303 ****
              // First check for packages
              File dir = new File(dirName, name);
!             if (dir.isDirectory() && caseok(dir, name) &&
                  (new File(dir, "__init__.py").isFile() ||
                   new File(dir, "__init__$py.class").isFile()))
--- 300,304 ----
              // First check for packages
              File dir = new File(dirName, name);
!             if (dir.isDirectory() && caseok(dir, name, nlen) &&
                  (new File(dir, "__init__.py").isFile() ||
                   new File(dir, "__init__$py.class").isFile()))
***************
*** 318,323 ****
              Py.writeDebug("import", "trying source " + pyFile.getPath());
  
!             if (pyFile.isFile() && caseok(pyFile, pyName)) {
!                 if (classFile.isFile() && caseok(classFile, className)) {
                      Py.writeDebug("import", "trying precompiled " +
                                    classFile.getPath());
--- 319,324 ----
              Py.writeDebug("import", "trying source " + pyFile.getPath());
  
!             if (pyFile.isFile() && caseok(pyFile, pyName, nlen)) {
!                 if (classFile.isFile() && caseok(classFile, className, nlen)) {
                      Py.writeDebug("import", "trying precompiled " +
                                    classFile.getPath());
***************
*** 338,342 ****
              // If no source, try loading precompiled
              Py.writeDebug("import", "trying " + classFile.getPath());
!             if (classFile.isFile() && caseok(classFile, className)) {
                  return createFromPyClass(modName, makeStream(classFile),
                                           false, classFile.getPath());
--- 339,343 ----
              // If no source, try loading precompiled
              Py.writeDebug("import", "trying " + classFile.getPath());
!             if (classFile.isFile() && caseok(classFile, className, nlen)) {
                  return createFromPyClass(modName, makeStream(classFile),
                                           false, classFile.getPath());
***************
*** 346,355 ****
      }
  
!     private static boolean caseok(File file, String filename) {
          if (Options.caseok)
              return true;
          try {
              File canFile = new File(file.getCanonicalPath());
!             return filename.equals(canFile.getName());
          } catch (IOException exc) {
              return false;
--- 347,356 ----
      }
  
!     private static boolean caseok(File file, String filename, int namelen) {
          if (Options.caseok)
              return true;
          try {
              File canFile = new File(file.getCanonicalPath());
!             return filename.regionMatches(0, canFile.getName(), 0, namelen);
          } catch (IOException exc) {
              return false;

[Jython-checkins] CVS: bugtests test305.py,NONE,1.1 test305m.PY,NONE,1.1

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/bugtests
In directory usw-pr-cvs1:/tmp/cvs-serv30099

Added Files:
	test305.py test305m.PY 
Log Message:
Test whether the case of the .PY excetion matters for module import.



--- NEW FILE: test305.py ---
"""
Test that the case of the module extension does not matter.
"""

import support

import test305m


--- NEW FILE: test305m.PY ---

foo = 1

[Jython-checkins] CVS: CVSROOT syncmail,1.1,1.2

[Barry W. <bw...@us...>]

Update of /cvsroot/jython/CVSROOT
In directory usw-pr-cvs1:/tmp/cvs-serv28844

Modified Files:
	syncmail 
Log Message:
Sync'ing with Fred's latest version of syncmail from the Python
project.


Index: syncmail
===================================================================
RCS file: /cvsroot/jython/CVSROOT/syncmail,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -r1.1 -r1.2
*** syncmail	2000/10/29 16:59:33	1.1
--- syncmail	2001/07/23 16:48:01	1.2
***************
*** 40,43 ****
--- 40,53 ----
          Print this text.
  
+     --context=#
+     -C #
+         Include # lines of context around lines that differ (default: 2).
+ 
+     -c
+         Produce a context diff (default).
+ 
+     -u
+         Produce a unified diff (smaller, but harder to read).
+ 
      <%%S>
          CVS %%s loginfo expansion.  When invoked by CVS, this will be a single
***************
*** 78,82 ****
  
  
! def calculate_diff(filespec):
      try:
          file, oldrev, newrev = string.split(filespec, ',')
--- 88,92 ----
  
  
! def calculate_diff(filespec, contextlines):
      try:
          file, oldrev, newrev = string.split(filespec, ',')
***************
*** 86,96 ****
      if oldrev == 'NONE':
          try:
!             fp = open(file)
              lines = fp.readlines()
              fp.close()
!             lines.insert(0, '--- NEW FILE ---\n')
          except IOError, e:
              lines = ['***** Error reading new file: ',
!                      str(e)]
      elif newrev == 'NONE':
          lines = ['--- %s DELETED ---\n' % file]
--- 96,110 ----
      if oldrev == 'NONE':
          try:
!             if os.path.exists(file):
!                 fp = open(file)
!             else:
!                 update_cmd = 'cvs -fn update -r %s -p %s' % (newrev, file)
!                 fp = os.popen(update_cmd)
              lines = fp.readlines()
              fp.close()
!             lines.insert(0, '--- NEW FILE: %s ---\n' % file)
          except IOError, e:
              lines = ['***** Error reading new file: ',
!                      str(e), '\n***** file: ', file, ' cwd: ', os.getcwd()]
      elif newrev == 'NONE':
          lines = ['--- %s DELETED ---\n' % file]
***************
*** 98,103 ****
          # This /has/ to happen in the background, otherwise we'll run into CVS
          # lock contention.  What a crock.
!         diffcmd = '/usr/bin/cvs -f diff -kk -C 2 -r %s -r %s %s' % (
!             oldrev, newrev, file)
          fp = os.popen(diffcmd)
          lines = fp.readlines()
--- 112,121 ----
          # This /has/ to happen in the background, otherwise we'll run into CVS
          # lock contention.  What a crock.
!         if contextlines > 0:
!             difftype = "-C " + str(contextlines)
!         else:
!             difftype = "-u"
!         diffcmd = "/usr/bin/cvs -f diff -kk %s --minimal -r %s -r %s '%s'" % (
!             difftype, oldrev, newrev, file)
          fp = os.popen(diffcmd)
          lines = fp.readlines()
***************
*** 115,119 ****
  
  
! def blast_mail(mailcmd, filestodiff):
      # cannot wait for child process or that will cause parent to retain cvs
      # lock for too long.  Urg!
--- 133,137 ----
  
  
! def blast_mail(mailcmd, filestodiff, contextlines):
      # cannot wait for child process or that will cause parent to retain cvs
      # lock for too long.  Urg!
***************
*** 127,131 ****
          # append the diffs if available
          for file in filestodiff:
!             fp.write(calculate_diff(file))
              fp.write('\n')
          fp.close()
--- 145,149 ----
          # append the diffs if available
          for file in filestodiff:
!             fp.write(calculate_diff(file, contextlines))
              fp.write('\n')
          fp.close()
***************
*** 137,142 ****
  # scan args for options
  def main():
      try:
!         opts, args = getopt.getopt(sys.argv[1:], 'h', ['cvsroot=', 'help'])
      except getopt.error, msg:
          usage(1, msg)
--- 155,162 ----
  # scan args for options
  def main():
+     contextlines = 2
      try:
!         opts, args = getopt.getopt(sys.argv[1:], 'hC:cu',
!                                    ['context=', 'cvsroot=', 'help'])
      except getopt.error, msg:
          usage(1, msg)
***************
*** 148,151 ****
--- 168,178 ----
          elif opt == '--cvsroot':
              os.environ['CVSROOT'] = arg
+         elif opt in ('-C', '--context'):
+             contextlines = int(arg)
+         elif opt == '-c':
+             if contextlines <= 0:
+                 contextlines = 2
+         elif opt == '-u':
+             contextlines = 0
  
      # What follows is the specification containing the files that were
***************
*** 172,177 ****
      if specs[-3:] == ['-', 'New', 'directory']:
          del specs[-3:]
      print 'Generating notification message...'
!     blast_mail(mailcmd, specs[1:])
      print 'Generating notification message... done.'
  
--- 199,213 ----
      if specs[-3:] == ['-', 'New', 'directory']:
          del specs[-3:]
+     elif len(specs) > 2:
+         L = specs[:2]
+         for s in specs[2:]:
+             prev = L[-1]
+             if string.count(prev, ",") < 2:
+                 L[-1] = "%s %s" % (prev, s)
+             else:
+                 L.append(s)
+         specs = L
      print 'Generating notification message...'
!     blast_mail(mailcmd, specs[1:], contextlines)
      print 'Generating notification message... done.'
  

[Jython-checkins] CVS: htdocs users.ht,1.6,1.7

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/htdocs
In directory usw-pr-cvs1:/tmp/cvs-serv22746

Modified Files:
	users.ht 
Log Message:
Added Propylon 



Index: users.ht
===================================================================
RCS file: /cvsroot/jython/htdocs/users.ht,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -r1.6 -r1.7
*** users.ht	2001/06/20 20:25:55	1.6
--- users.ht	2001/07/23 16:17:28	1.7
***************
*** 24,26 ****
  exploring genomic sequence similarity.  It is written entirely in Jython.
  
! </dl>
\ No newline at end of file
--- 24,32 ----
  exploring genomic sequence similarity.  It is written entirely in Jython.
  
! <p><dt><b><a href="http://www.propylon.com">Propylon</a></b>
! <dd>Propylon use Jython to bring the power of Python to complex XML
! processing on the Java Virtual Machine. Propylon also use Jython to
! provide enhanced server-side scripting and integration in its pervasive
! portal server product: Mission Control. 
! 
! </dl>

[Jython-checkins] CVS: jython/org/python/core CachedJarsPackageManager.java,1.8,1.9 PyJavaPackage.java,2.14,2.15

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython/org/python/core
In directory usw-pr-cvs1:/tmp/cvs-serv19395

Modified Files:
	CachedJarsPackageManager.java PyJavaPackage.java 
Log Message:
Solve feature request [ #428582 ] Access to non-public classes.

The cachedir files must be deleted for this change to take effect.



Index: CachedJarsPackageManager.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/core/CachedJarsPackageManager.java,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -r1.8 -r1.9
*** CachedJarsPackageManager.java	2001/03/14 14:30:48	1.8
--- CachedJarsPackageManager.java	2001/07/23 16:10:42	1.9
***************
*** 110,124 ****
          if (filterByName(className,false)) return;
  
!         // An extra careful test, maybe should be ignored????
!         int access = checkAccess(zip);
! 
!         if ((access == -1) || filterByAccess(name,access)) return;
! 
!         Vector vec = (Vector)zipPackages.get(packageName);
          if (vec == null) {
!             vec = new Vector();
              zipPackages.put(packageName, vec);
          }
-         vec.addElement(className);
      }
  
--- 110,124 ----
          if (filterByName(className,false)) return;
  
!         Vector[] vec = (Vector[])zipPackages.get(packageName);
          if (vec == null) {
!             vec = new Vector[] { new Vector(), new Vector() };
              zipPackages.put(packageName, vec);
+         }
+         int access = checkAccess(zip);
+         if ((access != -1) && !filterByAccess(name,access)) {
+             vec[0].addElement(className);
+         } else {
+             vec[1].addElement(className);
          }
      }
  
***************
*** 138,143 ****
          for (Enumeration e = zipPackages.keys() ; e.hasMoreElements() ;) {
              Object key = e.nextElement();
!             Vector vec = (Vector)zipPackages.get(key);
!             zipPackages.put(key, vectorToString(vec));
          }
  
--- 138,147 ----
          for (Enumeration e = zipPackages.keys() ; e.hasMoreElements() ;) {
              Object key = e.nextElement();
!             Vector[] vec = (Vector[])zipPackages.get(key);
!             String classes = vectorToString(vec[0]);
!             if (vec[1].size() > 0) {
!                 classes += '@' + vectorToString(vec[1]);
!             }
!             zipPackages.put(key, classes);
          }
  
***************
*** 298,301 ****
--- 302,310 ----
              String pkg = (String)e.nextElement();
              String classes = (String)zipPackages.get(pkg);
+ 
+             int idx = classes.indexOf('@');
+             if (idx >= 0 && Options.respectJavaAccessibility) {
+                 classes = classes.substring(0, idx);
+             }
  
              makeJavaPackage(pkg, classes, jarfile);

Index: PyJavaPackage.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/core/PyJavaPackage.java,v
retrieving revision 2.14
retrieving revision 2.15
diff -C2 -r2.14 -r2.15
*** PyJavaPackage.java	2001/07/18 15:53:56	2.14
--- PyJavaPackage.java	2001/07/23 16:10:42	2.15
***************
*** 104,108 ****
       */
      public void addPlaceholders(String classes) {
!         StringTokenizer tok = new StringTokenizer(classes, ",");
          while  (tok.hasMoreTokens())  {
              String p = tok.nextToken();
--- 104,108 ----
       */
      public void addPlaceholders(String classes) {
!         StringTokenizer tok = new StringTokenizer(classes, ",@");
          while  (tok.hasMoreTokens())  {
              String p = tok.nextToken();

[Jython-checkins] CVS: bugtests test304.py,NONE,1.1 test304m.py,NONE,1.1

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/bugtests
In directory usw-pr-cvs1:/tmp/cvs-serv12662

Added Files:
	test304.py test304m.py 
Log Message:
Test for feature request [ #428582 ] Access to non-public classes



--- NEW FILE ---
"""
Test that non-public classes gets loaded with dir() when respectJavaAcc is
false.
Feature request [ #428582 ] Access to non-public classes
"""

import support

support.runJava("org.python.util.jython " +
                "-Dpython.security.respectJavaAccessibility=false test304m.py")


--- NEW FILE ---

import support

import java.lang
d = dir(java.lang)
#print d
if "FDBigInt" not in d:
    raise support.TestWarning("Non-public class should by visible when " +
                        "respectJava is false")

[Jython-checkins] CVS: bugtests test303.py,NONE,1.1 test303j.java,NONE,1.1

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/bugtests
In directory usw-pr-cvs1:/tmp/cvs-serv10621

Added Files:
	test303.py test303j.java 
Log Message:
[ #440660 ] using nested java cls @ level >2 fails



--- NEW FILE ---
"""
Test deeply nested classes
[ #440660 ] using nested java cls @ level >2 fails
"""

import support
support.compileJava("test303j.java")

import test303j.A
try:
    import test303j.A.B
except ImportError:
    raise support.TestWarning('It should be possible to import test303j.A.B')

--- NEW FILE ---
public class test303j { 
    public static class A { 
        public static class B { 
            public static class C { 
            } 
        } 
    } 
} 

[Jython-checkins] CVS: jython/org/python/parser PythonGrammarTokenManager.java,2.11,2.12

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython/org/python/parser
In directory usw-pr-cvs1:/tmp/cvs-serv498

Modified Files:
	PythonGrammarTokenManager.java 
Log Message:
Commit generated files.



Index: PythonGrammarTokenManager.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/parser/PythonGrammarTokenManager.java,v
retrieving revision 2.11
retrieving revision 2.12
diff -C2 -r2.11 -r2.12
*** PythonGrammarTokenManager.java	2001/07/20 13:43:03	2.11
--- PythonGrammarTokenManager.java	2001/07/21 09:16:12	2.12
***************
*** 177,180 ****
--- 177,183 ----
  }
  static final long[] jjbitVec0 = {
+    0xfffffffffffffffeL, 0xffffffffffffffffL, 0xffffffffffffffffL, 0xffffffffffffffffL
+ };
+ static final long[] jjbitVec2 = {
     0x0L, 0x0L, 0xffffffffffffffffL, 0xffffffffffffffffL
  };
***************
*** 237,240 ****
--- 240,246 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 244,252 ****
              {
                 case 0:
!                   if ((jjbitVec0[i2] & l2) != 0L && kind > 110)
                       kind = 110;
                    break;
                 case 2:
!                   if ((jjbitVec0[i2] & l2) != 0L && kind > 111)
                       kind = 111;
                    break;
--- 250,258 ----
              {
                 case 0:
!                   if (jjCanMove_0(hiByte, i1, i2, l1, l2) && kind > 110)
                       kind = 110;
                    break;
                 case 2:
!                   if (jjCanMove_0(hiByte, i1, i2, l1, l2) && kind > 111)
                       kind = 111;
                    break;
***************
*** 421,424 ****
--- 427,433 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 428,432 ****
              {
                 case 0:
!                   if ((jjbitVec0[i2] & l2) != 0L && kind > 106)
                       kind = 106;
                    break;
--- 437,441 ----
              {
                 case 0:
!                   if (jjCanMove_0(hiByte, i1, i2, l1, l2) && kind > 106)
                       kind = 106;
                    break;
***************
*** 492,495 ****
--- 501,507 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 559,562 ****
--- 571,577 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 856,859 ****
--- 871,877 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 863,871 ****
              {
                 case 0:
!                   if ((jjbitVec0[i2] & l2) != 0L && kind > 110)
                       kind = 110;
                    break;
                 case 2:
!                   if ((jjbitVec0[i2] & l2) != 0L && kind > 111)
                       kind = 111;
                    break;
--- 881,889 ----
              {
                 case 0:
!                   if (jjCanMove_0(hiByte, i1, i2, l1, l2) && kind > 110)
                       kind = 110;
                    break;
                 case 2:
!                   if (jjCanMove_0(hiByte, i1, i2, l1, l2) && kind > 111)
                       kind = 111;
                    break;
***************
*** 998,1001 ****
--- 1016,1022 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 1005,1009 ****
              {
                 case 4:
!                   if ((jjbitVec0[i2] & l2) == 0L)
                       break;
                    if (kind > 16)
--- 1026,1030 ----
              {
                 case 4:
!                   if (!jjCanMove_0(hiByte, i1, i2, l1, l2))
                       break;
                    if (kind > 16)
***************
*** 1948,1951 ****
--- 1969,1975 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 1955,1959 ****
              {
                 case 8:
!                   if ((jjbitVec0[i2] & l2) == 0L)
                       break;
                    if (kind > 16)
--- 1979,1983 ----
              {
                 case 8:
!                   if (!jjCanMove_0(hiByte, i1, i2, l1, l2))
                       break;
                    if (kind > 16)
***************
*** 2131,2134 ****
--- 2155,2161 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 2138,2142 ****
              {
                 case 0:
!                   if ((jjbitVec0[i2] & l2) != 0L && kind > 105)
                       kind = 105;
                    break;
--- 2165,2169 ----
              {
                 case 0:
!                   if (jjCanMove_0(hiByte, i1, i2, l1, l2) && kind > 105)
                       kind = 105;
                    break;
***************
*** 2202,2205 ****
--- 2229,2235 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 2336,2339 ****
--- 2366,2372 ----
        else
        {
+          int hiByte = (int)(curChar >> 8);
+          int i1 = hiByte >> 6;
+          long l1 = 1L << (hiByte & 077);
           int i2 = (curChar & 0xff) >> 6;
           long l2 = 1L << (curChar & 077);
***************
*** 2343,2347 ****
              {
                 case 4:
!                   if ((jjbitVec0[i2] & l2) == 0L)
                       break;
                    if (kind > 16)
--- 2376,2380 ----
              {
                 case 4:
!                   if (!jjCanMove_0(hiByte, i1, i2, l1, l2))
                       break;
                    if (kind > 16)
***************
*** 2371,2374 ****
--- 2404,2419 ----
     33, 34, 37, 38, 
  };
+ private static final boolean jjCanMove_0(int hiByte, int i1, int i2, long l1, long l2)
+ {
+    switch(hiByte)
+    {
+       case 0:
+          return ((jjbitVec2[i2] & l2) != 0L);
+       default : 
+          if ((jjbitVec0[i1] & l1) != 0L)
+             return true;
+          return false;
+    }
+ }
  public static final String[] jjstrLiteralImages = {
  "", null, null, null, null, null, null, null, null, null, null, null, null, 

[Jython-checkins] CVS: jython/org/python/parser python.jjt,2.14,2.15

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython/org/python/parser
In directory usw-pr-cvs1:/tmp/cvs-serv32519

Modified Files:
	python.jjt 
Log Message:
Forgot to enable UNICODE support in the lexer. This makes charsets like
"~[\n]" as expected.



Index: python.jjt
===================================================================
RCS file: /cvsroot/jython/jython/org/python/parser/python.jjt,v
retrieving revision 2.14
retrieving revision 2.15
diff -C2 -r2.14 -r2.15
*** python.jjt	2001/07/20 13:37:43	2.14
--- python.jjt	2001/07/21 09:15:24	2.15
***************
*** 11,14 ****
--- 11,15 ----
  //    DEBUG_TOKEN_MANAGER = true;
      USER_CHAR_STREAM = true;
+     UNICODE_INPUT = true;
  }
  

[Jython-checkins] CVS: jython/org/python/parser CharStream.java,NONE,1.1

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython/org/python/parser
In directory usw-pr-cvs1:/tmp/cvs-serv20606

Added Files:
	CharStream.java 
Log Message:
Fix for bug: [ #439688 ] Syntax error for non-ascii characters

It involves replacing the broken ASCII_CharStream with a CharStrem
which return the whole chars unmodified. ASCII_CharStream cuts away the
top 8-bits of each char.



--- NEW FILE ---
/* Generated By:JavaCC: Do not edit this line. CharStream.java Version 0.7pre6 */
package org.python.parser;

/**
 * This interface describes a character stream that maintains line and
 * column number positions of the characters.  It also has the capability
 * to backup the stream to some extent.  An implementation of this
 * interface is used in the TokenManager implementation generated by
 * JavaCCParser.
 *
 * All the methods except backup can be implemented in any fashion. backup
 * needs to be implemented correctly for the correct operation of the lexer.
 * Rest of the methods are all used to get information like line number,
 * column number and the String that constitutes a token and are not used
 * by the lexer. Hence their implementation won't affect the generated lexer's
 * operation.
 */

public interface CharStream {

  /**
   * Returns the next character from the selected input.  The method
   * of selecting the input is the responsibility of the class
   * implementing this interface.  Can throw any java.io.IOException.
   */
  char readChar() throws java.io.IOException;

  /**
   * Returns the column position of the character last read.
   * @deprecated 
   * @see #getEndColumn
   */
  int getColumn();

  /**
   * Returns the line number of the character last read.
   * @deprecated 
   * @see #getEndLine
   */
  int getLine();

  /**
   * Returns the column number of the last character for current token (being
   * matched after the last call to BeginTOken).
   */
  int getEndColumn();

  /**
   * Returns the line number of the last character for current token (being
   * matched after the last call to BeginTOken).
   */
  int getEndLine();

  /**
   * Returns the column number of the first character for current token (being
   * matched after the last call to BeginTOken).
   */
  int getBeginColumn();

  /**
   * Returns the line number of the first character for current token (being
   * matched after the last call to BeginTOken).
   */
  int getBeginLine();

  /**
   * Backs up the input stream by amount steps. Lexer calls this method if it
   * had already read some characters, but could not use them to match a
   * (longer) token. So, they will be used again as the prefix of the next
   * token and it is the implemetation's responsibility to do this right.
   */
  void backup(int amount);

  /**
   * Returns the next character that marks the beginning of the next token.
   * All characters must remain in the buffer between two successive calls
   * to this method to implement backup correctly.
   */
  char BeginToken() throws java.io.IOException;

  /**
   * Returns a string made up of characters from the marked token beginning 
   * to the current buffer position. Implementations have the choice of returning
   * anything that they want to. For example, for efficiency, one might decide
   * to just return null, which is a valid implementation.
   */
  String GetImage();

  /**
   * Returns an array of characters that make up the suffix of length 'len' for
   * the currently matched token. This is used to build up the matched string
   * for use in actions in the case of MORE. A simple and inefficient
   * implementation of this is as follows :
   *
   *   {
   *      String t = GetImage();
   *      return t.substring(t.length() - len, t.length()).toCharArray();
   *   }
   */
  char[] GetSuffix(int len);

  /**
   * The lexer calls this function to indicate that it is done with the stream
   * and hence implementations can free any resources held by this class.
   * Again, the body of this function can be just empty and it will not
   * affect the lexer's operation.
   */
  void Done();

}

[Jython-checkins] CVS: jython/org/python/parser PythonGrammar.java,2.12,2.13 PythonGrammarTokenManager.java,2.10,2.11

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython/org/python/parser
In directory usw-pr-cvs1:/tmp/cvs-serv20057

Modified Files:
	PythonGrammar.java PythonGrammarTokenManager.java 
Log Message:
Comitting generated files.



Index: PythonGrammar.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/parser/PythonGrammar.java,v
retrieving revision 2.12
retrieving revision 2.13
diff -C2 -r2.12 -r2.13
*** PythonGrammar.java	2001/04/25 18:56:18	2.12
--- PythonGrammar.java	2001/07/20 13:43:03	2.13
***************
*** 6600,6604 ****
  
    public PythonGrammarTokenManager token_source;
-   ASCII_CharStream jj_input_stream;
    public Token token, jj_nt;
    private int jj_ntk;
--- 6600,6603 ----
***************
*** 6617,6623 ****
    private int jj_gc = 0;
  
!   public PythonGrammar(java.io.InputStream stream) {
!     jj_input_stream = new ASCII_CharStream(stream, 1, 1);
!     token_source = new PythonGrammarTokenManager(jj_input_stream);
      token = new Token();
      jj_ntk = -1;
--- 6616,6621 ----
    private int jj_gc = 0;
  
!   public PythonGrammar(CharStream stream) {
!     token_source = new PythonGrammarTokenManager(stream);
      token = new Token();
      jj_ntk = -1;
***************
*** 6627,6654 ****
    }
  
!   public void ReInit(java.io.InputStream stream) {
!     jj_input_stream.ReInit(stream, 1, 1);
!     token_source.ReInit(jj_input_stream);
!     token = new Token();
!     jj_ntk = -1;
!     jjtree.reset();
!     jj_gen = 0;
!     for (int i = 0; i < 104; i++) jj_la1[i] = -1;
!     for (int i = 0; i < jj_2_rtns.length; i++) jj_2_rtns[i] = new JJCalls();
!   }
! 
!   public PythonGrammar(java.io.Reader stream) {
!     jj_input_stream = new ASCII_CharStream(stream, 1, 1);
!     token_source = new PythonGrammarTokenManager(jj_input_stream);
!     token = new Token();
!     jj_ntk = -1;
!     jj_gen = 0;
!     for (int i = 0; i < 104; i++) jj_la1[i] = -1;
!     for (int i = 0; i < jj_2_rtns.length; i++) jj_2_rtns[i] = new JJCalls();
!   }
! 
!   public void ReInit(java.io.Reader stream) {
!     jj_input_stream.ReInit(stream, 1, 1);
!     token_source.ReInit(jj_input_stream);
      token = new Token();
      jj_ntk = -1;
--- 6625,6630 ----
    }
  
!   public void ReInit(CharStream stream) {
!     token_source.ReInit(stream);
      token = new Token();
      jj_ntk = -1;

Index: PythonGrammarTokenManager.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/parser/PythonGrammarTokenManager.java,v
retrieving revision 2.10
retrieving revision 2.11
diff -C2 -r2.10 -r2.11
*** PythonGrammarTokenManager.java	2001/04/25 18:56:18	2.10
--- PythonGrammarTokenManager.java	2001/07/20 13:43:03	2.11
***************
*** 2419,2423 ****
     0x0L, 0xff8780000000L, 
  };
! private ASCII_CharStream input_stream;
  private final int[] jjrounds = new int[52];
  private final int[] jjstateSet = new int[104];
--- 2419,2423 ----
     0x0L, 0xff8780000000L, 
  };
! private CharStream input_stream;
  private final int[] jjrounds = new int[52];
  private final int[] jjstateSet = new int[104];
***************
*** 2426,2441 ****
  int lengthOfMatch;
  protected char curChar;
! public PythonGrammarTokenManager(ASCII_CharStream stream)
  {
-    if (ASCII_CharStream.staticFlag)
-       throw new Error("ERROR: Cannot use a static CharStream class with a non-static lexical analyzer.");
     input_stream = stream;
  }
! public PythonGrammarTokenManager(ASCII_CharStream stream, int lexState)
  {
     this(stream);
     SwitchTo(lexState);
  }
! public void ReInit(ASCII_CharStream stream)
  {
     jjmatchedPos = jjnewStateCnt = 0;
--- 2426,2439 ----
  int lengthOfMatch;
  protected char curChar;
! public PythonGrammarTokenManager(CharStream stream)
  {
     input_stream = stream;
  }
! public PythonGrammarTokenManager(CharStream stream, int lexState)
  {
     this(stream);
     SwitchTo(lexState);
  }
! public void ReInit(CharStream stream)
  {
     jjmatchedPos = jjnewStateCnt = 0;
***************
*** 2451,2455 ****
        jjrounds[i] = 0x80000000;
  }
! public void ReInit(ASCII_CharStream stream, int lexState)
  {
     ReInit(stream);
--- 2449,2453 ----
        jjrounds[i] = 0x80000000;
  }
! public void ReInit(CharStream stream, int lexState)
  {
     ReInit(stream);
***************
*** 2662,2666 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
              if (parens == 0) {
                  indent = 0;
--- 2660,2664 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
              if (parens == 0) {
                  indent = 0;
***************
*** 2676,2680 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
              indent = (indent/8+1)*8;
              if (indent == indentation[level])
--- 2674,2678 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
              indent = (indent/8+1)*8;
              if (indent == indentation[level])
***************
*** 2687,2691 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
              indent += 1;
              if (indent == indentation[level])
--- 2685,2689 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
              indent += 1;
              if (indent == indentation[level])
***************
*** 2698,2702 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
              indent = 0;
              if (indent == indentation[level])
--- 2696,2700 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
              indent = 0;
              if (indent == indentation[level])
***************
*** 2709,2713 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
              if (parens == 0 && single_input && indent==0) {
                  //System.out.println("force newline");
--- 2707,2711 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
              if (parens == 0 && single_input && indent==0) {
                  //System.out.println("force newline");
***************
*** 2730,2734 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
                                                     indent = 0;
           break;
--- 2728,2732 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
                                                     indent = 0;
           break;
***************
*** 2746,2750 ****
                image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen)));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen)));
           jjimageLen = 0;
                             image.setLength(image.length()-3);
--- 2744,2748 ----
                image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen)));
           else
!             image.append(input_stream.GetSuffix(jjimageLen));
           jjimageLen = 0;
                             image.setLength(image.length()-3);
***************
*** 2754,2758 ****
                image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen)));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen)));
           jjimageLen = 0;
                             image.setLength(image.length()-2);
--- 2752,2756 ----
                image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen)));
           else
!             image.append(input_stream.GetSuffix(jjimageLen));
           jjimageLen = 0;
                             image.setLength(image.length()-2);
***************
*** 2762,2766 ****
                image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen)));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen)));
           jjimageLen = 0;
          int l = image.length();
--- 2760,2764 ----
                image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen)));
           else
!             image.append(input_stream.GetSuffix(jjimageLen));
           jjimageLen = 0;
          int l = image.length();
***************
*** 2772,2776 ****
                image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen)));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen)));
           jjimageLen = 0;
               image.setCharAt(image.length()-1, '\n');
--- 2770,2774 ----
                image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen)));
           else
!             image.append(input_stream.GetSuffix(jjimageLen));
           jjimageLen = 0;
               image.setCharAt(image.length()-1, '\n');
***************
*** 2788,2792 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
                                matchedToken.kind = NEWLINE;
           break;
--- 2786,2790 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
                                matchedToken.kind = NEWLINE;
           break;
***************
*** 2795,2799 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
              Token t1 = matchedToken;
              //System.out.println("final_newline: "+level);
--- 2793,2797 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
              Token t1 = matchedToken;
              //System.out.println("final_newline: "+level);
***************
*** 2807,2811 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
              if (indent > indentation[level]) {
                  level++;
--- 2805,2809 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
              if (indent > indentation[level]) {
                  level++;
***************
*** 2874,2878 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
      matchedToken.image = image.toString();
           break;
--- 2872,2876 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
      matchedToken.image = image.toString();
           break;
***************
*** 2881,2885 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
      matchedToken.image = image.toString();
           break;
--- 2879,2883 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
      matchedToken.image = image.toString();
           break;
***************
*** 2888,2892 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
      matchedToken.image = image.toString();
           break;
--- 2886,2890 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
      matchedToken.image = image.toString();
           break;
***************
*** 2895,2899 ****
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
      matchedToken.image = image.toString();
           break;
--- 2893,2897 ----
              image = new StringBuffer(new String(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1))));
           else
!             image.append(input_stream.GetSuffix(jjimageLen + (lengthOfMatch = jjmatchedPos + 1)));
      matchedToken.image = image.toString();
           break;

[Jython-checkins] CVS: jython/org/python/core parser.java,2.9,2.10

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython/org/python/core
In directory usw-pr-cvs1:/tmp/cvs-serv18201/core

Modified Files:
	parser.java 
Log Message:
Fix for bug: [ #439688 ] Syntax error for non-ascii characters

It involves replacing the broken ASCII_CharStream with a CharStrem
which return the whole chars unmodified. ASCII_CharStream cuts away the
top 8-bits of each char.



Index: parser.java
===================================================================
RCS file: /cvsroot/jython/jython/org/python/core/parser.java,v
retrieving revision 2.9
retrieving revision 2.10
diff -C2 -r2.9 -r2.10
*** parser.java	2001/05/15 18:53:35	2.9
--- parser.java	2001/07/20 13:37:43	2.10
***************
*** 96,100 ****
          } catch (IOException exc) { }
  
!         PythonGrammar g = new PythonGrammar(bufreader);
          SimpleNode node = null;
          try {
--- 96,100 ----
          } catch (IOException exc) { }
  
!         PythonGrammar g = new PythonGrammar(new ReaderCharStream(bufreader));
          SimpleNode node = null;
          try {

[Jython-checkins] CVS: jython/org/python/parser ReaderCharStream.java,NONE,2.1 python.jjt,2.13,2.14 ASCII_CharStream.java,2.2,NONE

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/jython/org/python/parser
In directory usw-pr-cvs1:/tmp/cvs-serv18201/parser

Modified Files:
	python.jjt 
Added Files:
	ReaderCharStream.java 
Removed Files:
	ASCII_CharStream.java 
Log Message:
Fix for bug: [ #439688 ] Syntax error for non-ascii characters

It involves replacing the broken ASCII_CharStream with a CharStrem
which return the whole chars unmodified. ASCII_CharStream cuts away the
top 8-bits of each char.



--- NEW FILE ---
package org.python.parser;

/**
 * An implementation of interface CharStream, where the data is read from
 * a Reader.
 * This file started life as a copy of ASCII_CharStream.java.
 */

public final class ReaderCharStream implements CharStream
{
  public static final boolean staticFlag = false;
  int bufsize;
  int available;
  int tokenBegin;
  public int bufpos = -1;
  private int bufline[];
  private int bufcolumn[];

  private int column = 0;
  private int line = 1;

  private boolean prevCharIsCR = false;
  private boolean prevCharIsLF = false;

  private java.io.Reader inputStream;

  private char[] buffer;
  private int maxNextCharInd = 0;
  private int inBuf = 0;

  private final void ExpandBuff(boolean wrapAround)
  {
     char[] newbuffer = new char[bufsize + 2048];
     int newbufline[] = new int[bufsize + 2048];
     int newbufcolumn[] = new int[bufsize + 2048];

     try
     {
        if (wrapAround)
        {
           System.arraycopy(buffer, tokenBegin, newbuffer, 0, bufsize - tokenBegin);
           System.arraycopy(buffer, 0, newbuffer,
                                             bufsize - tokenBegin, bufpos);
           buffer = newbuffer;

           System.arraycopy(bufline, tokenBegin, newbufline, 0, bufsize - tokenBegin);
           System.arraycopy(bufline, 0, newbufline, bufsize - tokenBegin, bufpos);
           bufline = newbufline;

           System.arraycopy(bufcolumn, tokenBegin, newbufcolumn, 0, bufsize - tokenBegin);
           System.arraycopy(bufcolumn, 0, newbufcolumn, bufsize - tokenBegin, bufpos);
           bufcolumn = newbufcolumn;

           maxNextCharInd = (bufpos += (bufsize - tokenBegin));
        }
        else
        {
           System.arraycopy(buffer, tokenBegin, newbuffer, 0, bufsize - tokenBegin);
           buffer = newbuffer;

           System.arraycopy(bufline, tokenBegin, newbufline, 0, bufsize - tokenBegin);
           bufline = newbufline;

           System.arraycopy(bufcolumn, tokenBegin, newbufcolumn, 0, bufsize - tokenBegin);
           bufcolumn = newbufcolumn;

           maxNextCharInd = (bufpos -= tokenBegin);
        }
     }
     catch (Throwable t)
     {
        throw new Error(t.getMessage());
     }


     bufsize += 2048;
     available = bufsize;
     tokenBegin = 0;
  }

  private final void FillBuff() throws java.io.IOException
  {
     if (maxNextCharInd == available)
     {
        if (available == bufsize)
        {
           if (tokenBegin > 2048)
           {
              bufpos = maxNextCharInd = 0;
              available = tokenBegin;
           }
           else if (tokenBegin < 0)
              bufpos = maxNextCharInd = 0;
           else
              ExpandBuff(false);
        }
        else if (available > tokenBegin)
           available = bufsize;
        else if ((tokenBegin - available) < 2048)
           ExpandBuff(true);
        else
           available = tokenBegin;
     }

     int i;
     try {
        if ((i = inputStream.read(buffer, maxNextCharInd,
                                    available - maxNextCharInd)) == -1)
        {
           inputStream.close();
           throw new java.io.IOException();
        }
        else
           maxNextCharInd += i;
        return;
     }
     catch(java.io.IOException e) {
        --bufpos;
        backup(0);
        if (tokenBegin == -1)
           tokenBegin = bufpos;
        throw e;
     }
  }

  public final char BeginToken() throws java.io.IOException
  {
     tokenBegin = -1;
     char c = readChar();
     tokenBegin = bufpos;

     return c;
  }

  private final void UpdateLineColumn(char c)
  {
     column++;

     if (prevCharIsLF)
     {
        prevCharIsLF = false;
        line += (column = 1);
     }
     else if (prevCharIsCR)
     {
        prevCharIsCR = false;
        if (c == '\n')
        {
           prevCharIsLF = true;
        }
        else
           line += (column = 1);
     }

     switch (c)
     {
        case '\r' :
           prevCharIsCR = true;
           break;
        case '\n' :
           prevCharIsLF = true;
           break;
        case '\t' :
           column--;
           column += (8 - (column & 07));
           break;
        default :
           break;
     }

     bufline[bufpos] = line;
     bufcolumn[bufpos] = column;
  }

  public final char readChar() throws java.io.IOException
  {
     if (inBuf > 0)
     {
        --inBuf;
        return buffer[(bufpos == bufsize - 1) ? (bufpos = 0) : ++bufpos];
     }

     if (++bufpos >= maxNextCharInd)
        FillBuff();

     char c = buffer[bufpos];

     UpdateLineColumn(c);
     return (c);
  }

  /**
   * @deprecated 
   * @see #getEndColumn
   */

  public final int getColumn() {
     return bufcolumn[bufpos];
  }

  /**
   * @deprecated 
   * @see #getEndLine
   */

  public final int getLine() {
     return bufline[bufpos];
  }

  public final int getEndColumn() {
     return bufcolumn[bufpos];
  }

  public final int getEndLine() {
     return bufline[bufpos];
  }

  public final int getBeginColumn() {
     return bufcolumn[tokenBegin];
  }

  public final int getBeginLine() {
     return bufline[tokenBegin];
  }

  public final void backup(int amount) {

    inBuf += amount;
    if ((bufpos -= amount) < 0)
       bufpos += bufsize;
  }

  public ReaderCharStream(java.io.Reader dstream)
  {
    inputStream = dstream;
    line = 1;
    column = 0;

    available = bufsize = 4096;
    buffer = new char[bufsize];
    bufline = new int[bufsize];
    bufcolumn = new int[bufsize];
  }

  public final String GetImage()
  {
     if (bufpos >= tokenBegin)
        return new String(buffer, tokenBegin, bufpos - tokenBegin + 1);
     else
        return new String(buffer, tokenBegin, bufsize - tokenBegin) +
                              new String(buffer, 0, bufpos + 1);
  }

  public final char[] GetSuffix(int len)
  {
     char[] ret = new char[len];

     if ((bufpos + 1) >= len)
        System.arraycopy(buffer, bufpos - len + 1, ret, 0, len);
     else
     {
        System.arraycopy(buffer, bufsize - (len - bufpos - 1), ret, 0,
                                                          len - bufpos - 1);
        System.arraycopy(buffer, 0, ret, len - bufpos - 1, bufpos + 1);
     }

     return ret;
  }

  public void Done()
  {
     buffer = null;
     bufline = null;
     bufcolumn = null;
  }

  /**
   * Method to adjust line and column numbers for the start of a token.<BR>
   */
  public void adjustBeginLineColumn(int newLine, int newCol)
  {
     int start = tokenBegin;
     int len;

     if (bufpos >= tokenBegin)
     {
        len = bufpos - tokenBegin + inBuf + 1;
     }
     else
     {
        len = bufsize - tokenBegin + bufpos + 1 + inBuf;
     }

     int i = 0, j = 0, k = 0;
     int nextColDiff = 0, columnDiff = 0;

     while (i < len &&
            bufline[j = start % bufsize] == bufline[k = ++start % bufsize])
     {
        bufline[j] = newLine;
        nextColDiff = columnDiff + bufcolumn[k] - bufcolumn[j];
        bufcolumn[j] = newCol + columnDiff;
        columnDiff = nextColDiff;
        i++;
     } 

     if (i < len)
     {
        bufline[j] = newLine++;
        bufcolumn[j] = newCol + columnDiff;

        while (i++ < len)
        {
           if (bufline[j = start % bufsize] != bufline[++start % bufsize])
              bufline[j] = newLine++;
           else
              bufline[j] = newLine;
        }
     }

     line = bufline[j];
     column = bufcolumn[j];
  }

}

Index: python.jjt
===================================================================
RCS file: /cvsroot/jython/jython/org/python/parser/python.jjt,v
retrieving revision 2.13
retrieving revision 2.14
diff -C2 -r2.13 -r2.14
*** python.jjt	2001/04/25 18:47:19	2.13
--- python.jjt	2001/07/20 13:37:43	2.14
***************
*** 10,13 ****
--- 10,14 ----
  //    DEBUG_PARSER = true;
  //    DEBUG_TOKEN_MANAGER = true;
+     USER_CHAR_STREAM = true;
  }
  

--- ASCII_CharStream.java DELETED ---

[Jython-checkins] CVS: bugtests test302.py,NONE,1.1

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/bugtests
In directory usw-pr-cvs1:/tmp/cvs-serv14617

Added Files:
	test302.py 
Log Message:
test for bug #439688.



--- NEW FILE ---
"""
Test the cp1542 encoding (euro-centric console)
In bug #439688 the value 0x99 does not survive the JavaCC parser.
"""

import support

f = open("test302s.py", "wb")
f.write('v = "\x99"\n')
f.close()

try:
    import test302s
except SyntaxError:
    raise support.TestWarning('Should not raise a Syntaxerror')

f = open("test302.out", "wb")
f.write("\x99")
f.close();

import java
readerval = java.io.FileReader("test302.out").read()

if ord(test302s.v) != readerval:
    raise support.TestError("Module source was not decoded correctly %x %x" % 
                (ord(test302s.v), ord(readerval)))

[Jython-checkins] CVS: htdocs .cvsignore,1.1,1.2

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/htdocs
In directory usw-pr-cvs1:/tmp/cvs-serv22502

Modified Files:
	.cvsignore 
Log Message:
The .tar files was renamed when moving to ant.



Index: .cvsignore
===================================================================
RCS file: /cvsroot/jython/htdocs/.cvsignore,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -r1.1 -r1.2
*** .cvsignore	2000/11/13 10:28:55	1.1
--- .cvsignore	2001/07/19 15:18:04	1.2
***************
*** 1 ****
! *.html htdocs.tar.gz *.bat
--- 1 ----
! *.html htdocs.tgz htdocs.tar *.bat

[Jython-checkins] CVS: htdocs README,1.1,1.2

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/htdocs
In directory usw-pr-cvs1:/tmp/cvs-serv18642

Modified Files:
	README 
Log Message:
Updated docs to mention build.xml and ant instead of make.



Index: README
===================================================================
RCS file: /cvsroot/jython/htdocs/README,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -r1.1 -r1.2
*** README	2000/11/13 10:08:52	1.1
--- README	2001/07/19 15:03:16	1.2
***************
*** 1,15 ****
  This directory contains the "source" for the jython website. The html 
! pagaes are generated by the ht2html tool which must be installed 
  separately:
  
     http://www.python.org/~bwarsaw/software/pyware.html
  
! See the Makefile for details on how the pages are generated.
  
  (Each page must be generated separately. Giving multiple ht-files as argument 
  to ht2html does not work for me)
  
! Makefile        Generates the .html files from .ht files. Works on 
!                 windows (nmake) and should also work on unix.
  
  links.h         List of links to the sidebar on each webpage.
--- 1,16 ----
  This directory contains the "source" for the jython website. The html 
! pages are generated by the ht2html tool which must be installed 
  separately:
  
     http://www.python.org/~bwarsaw/software/pyware.html
  
! See the build.xml for details on how the pages are generated.
  
  (Each page must be generated separately. Giving multiple ht-files as argument 
  to ht2html does not work for me)
  
! build.xml       Generates the .html files from .ht files. It is
!                 necessary to install bsf in the ant/lib directory
!                 for this to work.
  
  links.h         List of links to the sidebar on each webpage.

[Jython-checkins] CVS: htdocs/docs README,1.1,1.2

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/htdocs/docs
In directory usw-pr-cvs1:/tmp/cvs-serv18642/docs

Modified Files:
	README 
Log Message:
Updated docs to mention build.xml and ant instead of make.



Index: README
===================================================================
RCS file: /cvsroot/jython/htdocs/docs/README,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -r1.1 -r1.2
*** README	2000/11/13 10:14:40	1.1
--- README	2001/07/19 15:03:16	1.2
***************
*** 1,5 ****
  This is a placeholder for the generated documentation .html file.
  The files here are generated a little bit differently than the .html
! files in jython/Doc. See the Makefile for details.
  
  The differences are:
--- 1,5 ----
  This is a placeholder for the generated documentation .html file.
  The files here are generated a little bit differently than the .html
! files in jython/Doc. See the ../build.xml for details.
  
  The differences are:

[Jython-checkins] CVS: htdocs/applets Makefile,1.1,NONE

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/htdocs/applets
In directory usw-pr-cvs1:/tmp/cvs-serv16385/applets

Removed Files:
	Makefile 
Log Message:
Moved to ant building.



--- Makefile DELETED ---

[Jython-checkins] CVS: htdocs/docs Makefile,1.3,NONE

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/htdocs/docs
In directory usw-pr-cvs1:/tmp/cvs-serv16385/docs

Removed Files:
	Makefile 
Log Message:
Moved to ant building.



--- Makefile DELETED ---

[Jython-checkins] CVS: htdocs Makefile,1.3,NONE

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/htdocs
In directory usw-pr-cvs1:/tmp/cvs-serv16385

Removed Files:
	Makefile 
Log Message:
Moved to ant building.



--- Makefile DELETED ---

[Jython-checkins] CVS: htdocs NEWS.ht,1.9,1.10 license.ht,1.3,1.4

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/htdocs
In directory usw-pr-cvs1:/tmp/cvs-serv16075

Modified Files:
	NEWS.ht license.ht 
Log Message:
Fixed a minor bug in the generated .ht files.



Index: NEWS.ht
===================================================================
RCS file: /cvsroot/jython/htdocs/NEWS.ht,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -r1.9 -r1.10
*** NEWS.ht	2001/07/18 20:23:07	1.9
--- NEWS.ht	2001/07/19 14:52:11	1.10
***************
*** 344,346 ****
  indent-tabs-mode: nil
  End:
! </pre>'
--- 344,346 ----
  indent-tabs-mode: nil
  End:
! </pre>

Index: license.ht
===================================================================
RCS file: /cvsroot/jython/htdocs/license.ht,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -r1.3 -r1.4
*** license.ht	2000/11/25 21:39:15	1.3
--- license.ht	2001/07/19 14:52:11	1.4
***************
*** 142,144 ****
                                 [ACCEPT BUTTON]
  
! </pre>'
--- 142,144 ----
                                 [ACCEPT BUTTON]
  
! </pre>

[Jython-checkins] CVS: htdocs/applets links.h,1.2,1.3

[Finn B. <bc...@us...>]

Update of /cvsroot/jython/htdocs/applets
In directory usw-pr-cvs1:/tmp/cvs-serv15953

Modified Files:
	links.h 
Log Message:
Updated to 2.1a2 download link.



Index: links.h
===================================================================
RCS file: /cvsroot/jython/htdocs/applets/links.h,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -r1.2 -r1.3
*** links.h	2001/03/14 15:44:36	1.2
--- links.h	2001/07/19 14:51:07	1.3
***************
*** 4,8 ****
  <li><a href="../license.html">License</a>
  <li><a href="../download.html">Jython 2.0</a>
! <li><a href="http://sourceforge.net/project/showfiles.php?group_id=12867&release_id=27187">Jython 2.1a1</a>
  <li><a href="../install.html">Installing</a>
  <li><a href="../platform.html">JVM Compatibility</a>
--- 4,8 ----
  <li><a href="../license.html">License</a>
  <li><a href="../download.html">Jython 2.0</a>
! <li><a href="http://sourceforge.net/project/showfiles.php?group_id=12867&release_id=44111">Jython 2.1a2</a>
  <li><a href="../install.html">Installing</a>
  <li><a href="../platform.html">JVM Compatibility</a>
***************
*** 20,22 ****
  <h3>Applet Problems</h3>
  <li><a href="problems.html">Here's what to do</a>
! <li><a href="issues.html">Other applet issues</a>
\ No newline at end of file
--- 20,22 ----
  <h3>Applet Problems</h3>
  <li><a href="problems.html">Here's what to do</a>
! <li><a href="issues.html">Other applet issues</a>