Re: [Roundup-devel] Some Python 3 preparation patches

[Joseph M. <js...@po...>]

Here's a 36th patch similar to the previous 35 (and relative to a tree 
with those 35 applied), for an issue I missed when preparing those 
patches.  The same general description of testing etc. applies as for the 
previous 35 (description quoted below since the message with those patches 
is pending list moderator review as over 40 KB).

36. Python 3 preparation: update HTMLParser / htmlentitydefs imports.

Manual patch.

-- 
Joseph S. Myers
js...@po...

On Sat, 9 Jun 2018, Joseph Myers wrote:

> The attached patches fix various Python 3 compatibility issues in Roundup.  
> This specifically excludes anything for text handling and character set 
> conversions (for which we've previously discussed the most appropriate 
> approach for making things Python 3 compatible, see emails quoted in 
> 2to3-done.txt) - such as anything relating to uses of the unicode and 
> unichr functions and the StringIO module as well as the expected need for 
> explicit conversions in various places.  Thus, it does not get to a point 
> where Roundup code can actually run under Python 3, but should still be 
> useful incremental progress towards such a point, and I hope that these 
> patches or others for the same issues can go into the Roundup sources 
> (after 1.6 is out if too risky before then) as such incremental progress.
> 
> The attached tarball contains 35 patches.  Each patch aims to address one 
> Python 3 compatibility issue, wherever that issue occurs in the source 
> tree.  Many of the patches are either generated by tools such as 2to3 or 
> python-modernize and then included unchanged after a review of the 
> changes, or assisted by such tools (initial version generated by a tool 
> but I made further edits to the changes proposed by the tool); some are 
> manually created.  It is quite possible that some of the fixes are not the 
> optimal or most idiomatic ways of fixing the issues concerned, or that 
> there are mistakes in some of the changes (whether incorrect changes from 
> the tools that I didn't spot, or incorrect manual edits).
> 
> Each patch applies to a source tree with the previous patches applied, but 
> where there are any dependencies between patches they should only be 
> simple textual dependencies where they happen to change the same or nearby 
> code (e.g. code which has more than one issue in the same place), and the 
> ordering is fairly arbitrary.  Each patch (on top of the previous ones) 
> has been validated using Roundup's testsuite (under Python 2).  As I don't 
> have all the modules installed require to run the whole testsuite, some 
> tests are skipped automatically, and as parts of the Roundup code are not 
> covered by the testsuite, this may not be effective validation for some 
> changes or parts of changes (and so particular attention should be paid in 
> review to the manually written changes to code not covered by the 
> testsuite).
> 
> Some patches include changes to .py files (detectors etc.) that are copied 
> into people's instances.  There are no release notes added telling people 
> to update those files on upgrade, although as and when we actually have 
> working Python 3 support (in a form that allows moving an existing 
> instance from Python 2 to Python 3), release notes will certainly need to 
> mention the need to update such files if moving an instance to Python 3.
> 
> The 35 patches are as follows:
> 
> 1. Python 3 preparation: convert print to a function.
> 
> Tool-assisted patch.  It is possible that some "from __future__ import 
> print_function" are not in fact needed, if a file only uses print() with a 
> single string as an argument and so would work fine in Python 2 without 
> that import.
> 
> 2. Python 3 preparation: use repr() instead of ``.
> 
> Tool-generated patch.
> 
> 3. Python 3 preparation: "raise" syntax.
> 
> Changing "raise Exception, value" to "raise Exception(value)".  
> Tool-assisted patch.  Particular cases to check carefully are the one 
> place in frontends/ZRoundup/ZRoundup.py where a string exception needed to 
> be fixed, and the one in roundup/cgi/client.py involving raising an 
> exception with a traceback (requires three-argument form of raise in 
> Python 2, which as I understand it requires exec() to avoid a Python 3 
> syntax error),
> 
> 4. Python 3 preparation: "except" syntax.
> 
> Changing "except Exception, var" to "except Exception as var".  
> Tool-generated patch.
> 
> 5. Python 3 preparation: numeric literal syntax.
> 
> Fixes octal constants to use leading 0o, and removes 'L' suffixes.  
> Tool-assisted patch.
> 
> 6. Python 3 preparation: change "x.has_key(y)" to "y in x".
> 
> (Also likewise "not in" where appropriate.)  Tool-generated patch.
> 
> 7. Python 3 preparation: use != instead of <>.
> 
> Tool-generated patch.
> 
> 8. Python 3 preparation: use open() instead of file().
> 
> Tool-assisted patch.  Note one case where a simple substitution did not 
> suffice because the change was in a class that defined its own open() 
> method earlier, and thus needed to use builtins.open (respectively 
> __builtin__.open in Python 2).
> 
> 9. Python 3 preparation: convert exec to a function.
> 
> Tool-generated patch.
> 
> 10. Python 3 preparation: use exec(compile(open().read())) instead of 
> execfile().
> 
> Tool-assisted patch.
> 
> 11. Python 3 preparation: use sys.maxsize instead of sys.maxint.
> 
> Tool-generated patch.
> 
> 12. Python 3 preparation: use Exception instead of StandardError.
> 
> Tool-generated patch.
> 
> 13. Python 3 preparation: make relative imports explicit.
> 
> Tool-generated patch.
> 
> 14. Python 3 preparation: update function attribute names.
> 
> Tool-assisted patch.
> 
> 15. Python 3 preparation: use parentheses in list comprehension.
> 
> Tool-generated patch.
> 
> 16. Python 3 preparation: use range() instead of xrange().
> 
> Tool-assisted patch.  None of the existing range() uses seem to need to be 
> wrapped in list().  Note that range() may be less efficient than xrange() 
> in Python 2.
> 
> 17. Python 3 preparation: use sum() instead of reduce().
> 
> reduce() has changed from a built-in function to something in functools in 
> Python 3.  However, since both uses were with operator.add / 
> operator.__add__, it seemed to make more sense to use sum() instead of 
> functools.reduce().
> 
> 18. Python 3 preparation: avoid implicit tuple parameter unpacking.
> 
> Tool-assisted patch.
> 
> 19. Python 3 preparation: handle absence of "long" type.
> 
> Manual patch.
> 
> 20. Python 3 preparation: update calls to dict methods.
> 
> Tool-assisted patch.  Changes of iterkeys / itervalues / iteritems to keys 
> / values / items are fully automated, but may make things less efficient 
> in Python 2.  Automated tools want to add list() around many calls to keys 
> / values / items, but I thought most such list() additions were 
> unnecessary because it seemed the result of keys / values / items was just 
> iterated over while the set of dict keys remained unchanged, rather than 
> used in a way requiring an actual list, or used while the set of keys in 
> the dict could change.  It's quite possible I missed some cases where 
> list() was really needed, or left in some unnecessary list() calls.
> 
> In cases where list() was only needed because the resulting list was then 
> sorted in-place, I changed the code to use calls to sorted().
> 
> 21. Python 3 preparation: update next() usage for iterators.
> 
> Tool-assisted patch.  Note that various classes in TAL code with next() 
> methods are not actually Python iterators and so are not changed in this 
> patch, but roundup/cgi/ZTUtils/Iterator.py includes the IterIter class 
> which converts between the two styles of iterator.
> 
> 22. Python 3 preparation: use list() around zip() as needed.
> 
> Tool-assisted patch.
> 
> 23. Python 3 preparation: use list() around filter() as needed.
> 
> Tool-assisted patch.
> 
> 24. Python 3 preparation: update map() calls as needed.
> 
> Tool-assisted patch.  Note one place using map with first argument None 
> (not supported in Python 3) which needed to change to using 
> itertools.zip_longest (respectively itertools.izip_longest in Python 2).
> 
> 25. Python 3 preparation: use // and __truediv__ as needed.
> 
> Tool-assisted patch.  Those divisions that I thought must be integer floor 
> divisions and rely on Python 2 integer floor division semantics are 
> changed to use // (if any are actually meant to be floating-point 
> divisions, that would break things).  One __div__ method is changed to 
> __truediv__ (with __div__ = __truediv__ for Python 2 compatibility).
> 
> 26. Python 3 preparation: replace raw_input uses.
> 
> The existing my_input in roundup/admin.py is moved to a new 
> roundup/anypy/my_input.py, which is then used in more places.  Manual 
> patch.
> 
> 27. Python 3 preparation: update urllib / urllib2 / urlparse imports.
> 
> The existing roundup/anypy/urllib_.py is extended to cover more imports 
> and used in more places.  Manual patch.
> 
> 28. Python 3 preparation: remove unused ConfigParser import.
> 
> Manual patch (trivial, removing an unused import of a module that's 
> changed its name in Python 3).
> 
> 29. Python 3 preparation: use __bool__ instead of __nonzero__.
> 
> Tool-assisted patch.  __nonzero__ = __bool__ included for Python 2 
> compatibility.
> 
> 30. Python 3 preparation: use imp.reload instead of reload as needed.
> 
> Manual patch.
> 
> 31. Python 3 preparation: use isinstance(x, collections.Callable) instead of callable(x).
> 
> Tool-assisted patch.
> 
> 32. Python 3 preparation: update httplib imports.
> 
> Existing roundup.anypy.http_ used in another place.  Manual patch.
> 
> 33. Python 3 preparation: update xmlrpclib / SimpleXMLRPCServer imports.
> 
> New roundup/anypy/xmlrpc_.py added.  Manual patch.
> 
> 34. Python 3 preparation: update BaseHTTPServer imports.
> 
> roundup/anypy/http_.py extended and used in more places.  Manual patch.
> 
> 35. Python 3 preparation: update SocketServer import.
> 
> Manual patch.
> 
> -- 
> Joseph S. Myers
> js...@po...