#279 Installation 25 times slower for Python 3

[Hugo]

Pillow is tested on Travis CI (Ubuntu 12.04 LTS Server Edition 64 bit). We install some libraries that depend on docutils.

Installation is particularly slow for Python 3, due to docutils.

A minimal example running pip install docutils on Travis:

pypy    5s
pypy3   64s
2.6     3s
2.7     2s
3.2     98s
3.3     78s
3.4     102s

https://travis-ci.org/hugovk/test/builds/60198476

Python 3 installation is something like 25 times slower than with Python 2.

We're skipping the installation/test on Python 3, but is there any way to speed up Python 3 installation?

[Dmitry Shachnev]

That is because of 2to3 tool, which is used during Python 3 build.

The way to fix this is using the unified codebase for Python 2 and Python 3, but this can happen only when Docutils drops support for Python 2.5.

You can also try a patch similar to what we have in Debian that makes 2to3 run in parallel (don't forget to replace our logic to get num_processes with your own one).

[Hugo]

You can also try a patch similar to what we have in Debian that makes 2to3 run in parallel (don't forget to replace our logic to get num_processes with your own one).

Thanks, I tested a bit more.

So with no change, Python 3 installs take from around 60-63 seconds:
https://travis-ci.org/hugovk/test/builds/61182850

Then, changing number of processes like this:
https://github.com/hugovk/docutils/blob/parallel-2to3/setup.py#L28-34

With 2 processes, it takes from around 27-47 seconds:
https://travis-ci.org/hugovk/test/builds/61189573

With 3 processes, it takes from around 29-49 seconds:
https://travis-ci.org/hugovk/test/builds/61190202

With 4 processes, it takes from around 35-64 seconds:
https://travis-ci.org/hugovk/test/jobs/61186386

A definite improvement, but I think we'll stick to limiting this one test to Python 2 for now.

Thanks for the help!

[Günter Milde]

While a common codebase for 2.x and 3.x is a long term goal, this is not the only way to speedup installation under Python 3.

Feature request 44 suggests wheels to pre-converse the code for Py3k.
This gives similar installation time for Py 2.x and 3.x.

[Hugo]

For reference, a link to feature request "Make setup.py build wheels": https://sourceforge.net/p/docutils/feature-requests/43/

Wheels would be good. Any chance to include them for the next release? When is the next release due?

Or better still, added for the current release at PyPI?

Thank you!

[Tony Narlock]

Related: https://sourceforge.net/p/docutils/bugs/277/ / [bugs:#277]

This is the tip of the iceberg with supporting old python versions, Günter, another developer on the project, has seems to be inclined to milk out python 2.4 and 2.5 as long as he can, despite being explained why at length:

https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/mhthr6%24k6r%241%40ger.gmane.org/#msg34072787

I will follow up in 277.

[Marcelo Huerta]

2015-05-05 1:41 GMT-03:00 Tony Narlock gitpull@users.sf.net:

Related: https://sourceforge.net/p/docutils/bugs/277/ / [bugs:#277]

This is the tip of the iceberg with supporting old python versions, Günter,
another developer on the project, has seems to be inclined to milk out
python 2.4 and 2.5 as long as he can, despite being explained why at length:

Once upon a time, this was a civil place, at least towards long-time
contributors to the project (I wouldn't ask that much consideration
for me, as I haven't added much besides a language file for
esperanto).

However, the tone of recent interactions and promotion of ideas is
getting increasingly aggressive, and the quoted text reads as plainly
insulting.

Are you intent on alienating developers or this is just the way you
usually interact in development lists?

[Tony Narlock]

On the matter of keeping things civil, please keep threads on topic. Thanks :)

[Günter Milde]

There is now a Python-3 wheel at PyPi. Please test and close this bug, if this solves your issue.

[Hugo]

And the wheel does it!

Times for pip install docutils:

Python 3.2 1.22s
Python 3.3 0.95s
Python 3.4 1.36s
PyPy3 3.90s
Python 2.6 2.70s
Python 2.7 1.55s
PyPy 5.10s

https://travis-ci.org/hugovk/test/builds/61676353

Thank you!

PS I can't see anywhere that I can close this ticket, please can someone else do so?

[Günter Milde]

The link appears if you log in to SF (and are either the creator/owner of the bug or a Docutils maintainer.
Closing this now. Thanks for the report.