winpysetup

version 1.0 dd. 2015-05-25

Clean testing on Windows:

• install Windows (or clone your clean Windows VM)
• Download winpysetup.exe and start it.
• Once the program is done (now is a good time to make another snapshot if you run Windows in a VM), open a new Command Prompt (to get the change in PATH) and change to a directory without spaces in the path (e.g. C:\src).
• Run hg clone --insecure https://hg@bitbucket.org/ruamel/minimal
• Change to the minimal directory and run tox.
• Watch how tox invokes py.test succesfully against Python 2.7/2.6/3.5/3.4/3.3 and pypy (you can of course run tox immediately on your own, more interesting, code as well).

Installed items include: pip, easy_install, mercurial, tox, detox, py.test and the Visual C compiler for Python 2.7

Special hooks can change the behavior of the installer. In particular: - the list of python versions to install can be easily extended/reduced - the list of default packages to install can be extended/reduced

Winpysetup has been testen on:

• WinXP with Service Pack 2 (32 and 64 bit)
• Windows 7 (32 and 64 bit)

Other details:

• the default environment that runs 'tox', etc., is CPython 2.7
• the winpysetup.exe is a python2.3 based exe that can run on WinXP without preinstalling any other files (like VC runtime environments)
• all necessary files are downloaded and saved, on future runs internet access is only necessary for get_pip.py (unless you scratch your Windows, or delete any of those files).
• winpysetup first retrieves winpysetup.py and runs it.
• vcredist.exe is installed (as is necessary for some python/pypy installers and windows version combinations.
• updates/installs pip and setuptools for all versions (first using get_pip.py, taken from the link on the pip documentation page.)
• adds the directory of the executables and scripts/utils to the PATH for selected python versions (normally only python 2.7)
• make a pythonX.Y executable for each python.exe (same for pypy), in case multiple paths are inserted, to be able to invoke the right python without full path.
• it is possible to intercept the download and retrieve larger files from some alternative local server. Running from a mounted drive might be more difficult (except on WinXP) because of security limitations. A simpler "solution" is to save the directory with the MSI files on a shared drive and copy these files back in, to prevent re-downloading.
• On Windows 7 you have to run this from a local drive, directly starting from a network drive doesn't seem to work. I usually copy the file from a network drive, along with a populated msi subdirectory, as described in the simpler solution of the previous point.
• On Windows 7 you might have to run as administrator (right click). Or run from a command prompt that was started with admin rights.
• hosted on bitbucket the installer files are taken from their "normal" download sites.
• Explicitly install 32 bit version by appending -32 to the version (needed for pypy on Windows 64 bit, as well as 64/32 bit capable installers.
• winpysetup can be used to create the python2.3 environment to generate itself.
• Extended functionality for the Python 2.3 environment with argparse and ctypes (used e.g. for notifying programs that PATH has been updated).

Thanks to Holger Krekel for confirming that automating a Windows box setup would be a good idea, and to the VirtualBox developers for making the execution of 250+ test installs less time consuming.

Paths with spaces

Due to multiple problems with paths that have spaces in them, the python versions are instaleld in C:\python and not in C:\Program Files\python. You should also create your projects in a path without spaces (e.g. C:\users\yourname\src , on WinXP your home directory normally does have spaces, so use C:\usr or similar).

The setup includes a fixutilpath utility to correct "Failed to create process" which happens when pip installs a source (not when it installs from wheel). This is automatically invoked at the end of the install to fix e.g. py.test (easy_install doesn't have this problem, but currently doesn't download from pypy on none of my windows) This bug has been fixed in revision 738 of distlib (2015-02-28) but has not made it into pip yet.

Influencing installer actions

You can influence the installer actions by creating a pre_winpysetup.py in the same directory as winpysetup.exe (this one will not be downloaded as other missing files are).

This file should follow the following skeleton (please note that this has to be Python 2.3 conformant):

import sys
import os

# call sys.exit if something goes wrong

def pre_call(self):
    # to add/change self._versions, self.packages
    pass

def pre_versions(self):
    pass

def pre_download_url(self, version_info):
    # here you can download from some local repository and store in
    # self._msi_dir. The normal download will then see it in the cache
    # the url to download is in the url attribute: version_info.url
    return False  # if not returning False, version is not installed at all

def pre_install(self, path, res):
    return False # if not returning False install is not called

def pre_install_pip(self, exe, extra_modules):
    return False # if not returning False install_pip is not called

def main(verbose, installer):
    print 'loading pre_winpysetup'
    # can comment out what does not need to be called
    installer.pre_call = pre_call
    installer.pre_versions = pre_versions
    installer.pre_download_url = pre_download_url
    installer.pre_install = pre_install
    installer.pre_install_pip = pre_install_pip

An real example is given below for adding pypy3 to the installation.

Specifying which versions to install

By default the versions consists of "2.7*", "2.6", "3.4", "3.3", "py 2-32" This causes the latest versions for each of those series, that is not a release candidate or beta, to be downloaded and installed.

If you specify a matching version for a release candidate (e.g. currently 2.7.10 you will get that release candidate. Version numbers are preseeded by either nothing (implying cp for CPython, p2 (for pypy) p3 (for pypy3) version respectively. Please note that the versions you specify for pypy are their release numbers, not the ones of the CPython they are compatible with.

Additional packages (specified in self.packages) are installed for 2.7 because it is the starred (*) entry. Multiple entries can be starred and their order determines the order they will be insterted in the PATH (last one ends up being first in the path. Because of the copy to PythonX.Y.exe you can still easily access multiple executables without providing the full path, as long as they were all starred during installation.

You can comment out an entry by having it start with a hash #.

The list of installers and where to get them is in win_py_version.lst this currently not automatically download when changed on the web (you have to delete any previously downloaded version (but you can do that from pre_winpysetup.py if you want to be up-to-date all the time).

You can edit that list, generate it yourself with generate_version_list.py., or add to the list as from the pre_winpysetup.exe as is done in the example that regenerates winpysetup.exe itself.

Adapted environments

You can change many aspect of the setup by creating a pre_winpysetup.py file next to the winpysetup.exe.

Often it is enough to only set the installer.pre_call method, and set/change self.versions and/or adapt self.packages

If you only need to test on windows against Python 2.7, and don't need mercurial you can use the following pre_winpysetup.py:

""" this is python 2.3 code!"""

def pre_call(self):
    self.versions = [
        'vc 2008-32',  # needed for Python 2.7 on XP
        '2.7*', # last to be installed therefore first in %PATH%
    ]
    self.packages.remove('mercurial')
    self.packages.remove('detox')

def main(verbose, installer):
    print 'loading pre_winpysetup'
    installer.pre_call = pre_call

You can leave out the VCForPython27 compiler ('vc 2.7-32', as is only needed for the install of mercurial, and the Windows Installer ('wi 3.1') as it is not needed for the 2.7 installer.

In the following if only the body of pre_call() is given, the main from the example above is assumed.

You probably want the python that has pip,``tox``, et all installed (2.7), to be the last entry. So that version will have the file associations for .py.This is not absolutely necessary, but it explains some of the extra effort in the examples.

def pre_call(self):
    new_versions = []
    for v in self.versions:
        b32 = False
        for v32 in ['2.6', '3.4', '3.3', '2.7', 'vc 2010']:
            if v.startswith(v32):
                b32 = True
                break
        # install 32 bit version after the python which drives the whole
        # this is necessary for CPython 3.4 as installing 32 before 64
        # results in the 32 bit python.dll being removed
        if b32 and v[-1] == '*':
            new_versions.append(v32 + '-32')
        new_versions.append(v)
        # install 32 bit version before the full python that drives
        # the whole (usually 2.7)
        if b32 and v[-1] != '*':
            new_versions.append(v32 + '-32')
    self.versions = new_versions

tox -r --32

print("\rpython executable is " + platform.architecture()[0])

def pre_call():
    self.versions = [
        'wi 3.1',
        'vc 2.7-32',
        'vc 2008',
        'vc 2008-32',
        'vc 2010',
        'vc 2010-32',
        '3.4-32',
        '3.4',
        '2.7*'
    ]

def pre_call():
    self.versions = [
        'wi 3.1',
        'vc 2.7-32',
        'vc 2008',
        'vc 2008-32',
        'vc 2010',
        'vc 2010-32',
        '3.4',
        '3.4-32',
        '2.7*'
    ]

import os

def pre_call(self):
    # to add/change self._versions, self.packages
    if self.verbose > 0:
        print 'appending pypy3'
    self.versions.append('p3 2-32')  # pypy 3

def pre_install_pip(self, exe, extra_modules):
    # write out a batch file
    if '\\pypy3-' in exe:
        print '---->>> exe', exe
        file_name = os.path.join(os.path.dirname(exe), 'pip.bat')
        fp = open(file_name, 'w')
        fp.write('pypy -m pip %1 %2 %3 %4 %5 %6 %7 %8 %9\n')
        fp.close()
    return False # if not returning False install_pip is not called

def main(verbose, installer):
    print 'loading pre_winpysetup'
    installer.pre_call = pre_call
    installer.pre_install_pip = pre_install_pip

Creating winpysetup.exe

winpysetup.exe can be used to create itself. Download the winpysetup.exe and the pre_winpysetup.py in one directory on a clean windows machine. Rename pre_winpysetup.exe_gen-winpysetup.py to pre_winpysetup.py and start winpysetup.exe. After installing python 2.3, py2exe and other files necessary, the new C:\\winpysetup directory contains the newly generated winpysetup.exe file.

Calling from another program

winpysetup.py prompts for user interaction to close the window after an install, if not run from a Command Shell.

If you want to run winpysetup.exe from another program (not batch file) and have it not prompt for continuation at the end, set the PROMPT environment variable before starting the program to some non-empty value.

Background

Why not a real installer?

I reused a program that I wrote to drive the installation process some years ago. I created that not although I knew about windows installers, InstallShield and others, but because I knew about them (having supervised the creation of such installers for commercial programs that have been used by millions of people). MSI installers try to do away with the need for programs by defining actions, but you still need to run code for complex, real-life decisions, than can be handled by table entry manipulation. Hence InstallShield runs their script code fromm the .msi, where they used to do so more directly from an executable.

What I wanted is to write what I needed to do in Python, not in some installer scripting language. So I made a program that was wrapped with py2exe, where the output, when run, would read the .exe itself for attached zip data, extract that and take some action based on directory and name. I had to reinstall python 2.3 and redo this at some point, so this .exe would run on WinXP. Later versions of Python would require you to install a runtime library on WinXP first (this was before 2010).

This program, as used for winpysetup, only gets one file attached:

import sys
import os
import urllib2

_base_dir = os.path.dirname(os.path.abspath(sys.argv[0]))

module_file_name = 'winpysetup.py'
module_full_path = os.path.join(_base_dir, module_file_name)


def main():
    # I would have to update the utility to add additional packages
    # to be used by the 2.3 code (not the final install), so write out the
    # data here
    for k in modules:
        file_name = os.path.join(sys.path[0], k)
        fp = open(file_name, 'wb')
        fp.write(modules[k])
        fp.close()

    # only download if not there, updating to a new version can be done
    # by deleting winpysetup.py
    if not os.path.exists(module_full_path):
        url = "https://bitbucket.org/ruamel/winpysetup/raw/tip/" + module_file_name
        fp = open(module_full_path, 'w')
        fp.write(urllib2.urlopen(url).read())
        fp.close()

    sys.path.insert(0, _base_dir)
    import winpysetup

modules = {
# stripped
}

main()

(what is missing here are some modules that were not included in the original python2.3 executable: gettext, argparse)

It downloads the latest version of winpysetup.py from bitbucket, if not already on the disc next to winpysetup.exe and then executes that code.

It is nice to be able to do everything in Python. Even though 2.3 now feels somewhat restricted nowadays, I like it way more than using some installer scripting language.

Particular problems encountered developing winpysetup

#!C:\Program Files\Python\2.7\python.exe

ToDo

Things are working but of course there are several areas where things can be enhanced:

• make winpysetup.py runnable on Linux (again), so it can be used to populate a MSI subdirectory on a VM host.
• allow 64 bit version number below 32 bit (vcredist.exe 2008), now has to have the same minor number