#278 Ruamel should declare a dependency to setuptools

[Anthon van der Neut]

Currently ruamel seems to use pkg_resources to declare namespaces:

https://bitbucket.org/ruamel/yaml/src/862f1c2bdfd8ac3acb7e4ae9ee45741025ff0b45/setup.py?at=default&fileviewer=file-view-default#setup.py-490

pkg_resources is provided by setuptools but that is missing from install_requires in setup.py.

This causes for instance pex (https://github.com/pantsbuild/pex/) to complain:

/home/apollo13/bap_services(1).exe/.bootstrap/pex/environment.py:330: UserWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by ruamel.yaml 0.15.87. The ruamel.yaml 0.15.87 distribution should fix its `install_requires` to include `setuptools`

The same/similar issue occured for gunicorn: https://github.com/benoitc/gunicorn/issues/1716 -- one cannot rely on the fact that setuptools is available, it is not necessarily part of the python installation.

(originally posted on 2019-02-12 at 15:37:58 by Florian Apolloner <apollo13@bitbucket>)

[Anthon van der Neut]

If I cannot make it so that setuptools gets added to the pkg_resources only when installing from tar.gz this is not going to happen as the wheels don't have setup.py, and hence no need. ruamel.yaml is not dependent on setuptools, only its installer.

(originally posted on 2019-02-12 at 16:45:27)

[Anthon van der Neut]

None
(originally posted on 2019-02-12 at 16:46:03)

[Anthon van der Neut]

Intresting, I thought the generated file would end up in the installation -- but indeed there is no pkg_resources import in the installed code. I'll double check to see where the import comes from. Thanks so far!

(originally posted on 2019-02-12 at 17:30:23 by Florian Apolloner <apollo13@bitbucket>)

[Anthon van der Neut]

If you install with pip and cannot use the wheel, you'll get the tar.gz, run setup.py and that uses pkg_resources. So I know where it comes from. It is just that I don't want to burden wheel based installs with an (unused) dependency on setuptools (and after installing the tar.gz, you could potentially do away with setuptools.

This is rather nasty, as the wheel builder starts with the .tar.gz. I'll have to dig a bit, haven't had the need for non-trivial changes to setup.py (which is common to all the ruamel.xyz packages) for a while.

(originally posted on 2019-02-12 at 17:36:19)

[Anthon van der Neut]

I understand that, the thing is I am seeing this notice when executing a pex binary, which should be post installation…

Either way, please don't waste any time on it. I will dig in and report back once I have a clear picture of what pex is doing!

(originally posted on 2019-02-12 at 17:38:02 by Florian Apolloner <apollo13@bitbucket>)

[Anthon van der Neut]

Hrmpf, I can no longer reproduce. Apparently something in my build chain must have changed the last few hours :(

What I did note though is that the wheel files contain ruamel/yaml/data which contains a setup.py -- this seems rather weird to include in a wheel file. That said, I got a bit afraid when looking at the setup.py :D

(originally posted on 2019-02-12 at 17:56:46 by Florian Apolloner <apollo13@bitbucket>)

[Anthon van der Neut]

That data subdir should not be in there. Which wheel did you download (version, architecture), where you found this? (I could remove that setup.py at least for the time being, but the whole directory should not be there in the first place).

The setup.py interprets data defined in the __init__.py, it is (or should be) generic for the 100+ packages I maintain (not all open source). The data is read and the AST interpreted into a Python datastructure (so no importing), the that is used to populate setup(). The data section in the __init__.py can be programmatically updated/extended/inspected, something not easily done with mosts non-ruamel setup.py set ups.

(originally posted on 2019-02-12 at 18:45:29)

[Anthon van der Neut]

The file I downloaded is https://files.pythonhosted.org/packages/d5/37/54020dea78a2df39ff1dcb0efa5f78de58e865ac913828305f70f06662fe/ruamel.yaml-0.15.87-cp37-cp37m-manylinux1_x86_64.whl -- so the manylinux version for py3.7 and ruamel version 0.15.87. No need for a quickfix from my side, just wanted to inform you about it.

Interesting approach you took with your installations, no wonder it looked like pure magic on a first glance!

(originally posted on 2019-02-12 at 18:59:00 by Florian Apolloner <apollo13@bitbucket>)

[Anthon van der Neut]

Ah, thank you for that. I of course looked at one of the other wheels. The Windows and MacOS wheels are build from check outs, and there ruamel/yaml/data is not available. But the manylinux version runs in my source tree and there there several subdirs.
There is key/val nested:=True that needs to be set for these subpackages to be excluded and that was not available in the data/__init__.py. Because I am actually not building that subtree, but using it for seperately stored tests, it could have taken quite a bit of time to notice.

(originally posted on 2019-02-12 at 20:51:20)

[Anthon van der Neut]

• status set to resolved

This should be fixed in the wheels of 0.15.88, the tar.gz still had the data, but will be fixed in the next release

(originally posted on 2019-02-12 at 22:19:04)

[Anthon van der Neut]

Thank you, this did do the trick!

(originally posted on 2019-02-13 at 07:35:26 by Florian Apolloner <apollo13@bitbucket>)

[Anthon van der Neut]

• status set to closed

(originally posted on 2019-02-13 at 11:23:43)

[Julius Kreuzer]

Hi,
It seems the issue is still not resolved:

~/.pex/unzipped_pexes/e72a54eb68eb133ffaca563338c61cbee65ce582/.bootstrap/pex/environment.py:399: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:


1. ruamel.yaml==0.16.10 namespace packages:
  ruamel


2. ruamel.yaml.clib==0.2.0 namespace packages:
  ruamel
  ruamel.yaml

These distributions should fix their `install_requires` to include `setuptools`
  "should fix their `install_requires` to include `setuptools`".format(dists=dists)

I haven't yet looked into it. Any idea, why the problem still surfaces?

[Anthon van der Neut]

The installed package doesn't have any dependencies on pkg_resources (you can try that by installing from the wheel and search), so the install_requires should not have a dependency on setuptools. During build, a (temporary) __init__.py file is created that is dependend on pkg_resources (and thus setuptools).

Brief tests (with a virtualenv without setuptools installed) and with setuptools in the requires.txt file (as a result from adding it temporarilly to install_requires) shows that, you still get an error ModuleNotFoundError: No module named 'setuptools'. But maybe I should just make it a (documentation) requirement to run not just pip install, but also require to run this command in a virtual env set up with python -m venv (which AFAICT always has setuptools installed, and thereby also resolving issues with other "degraded" environments).

I can add setuptools to setup_requires, if you can confirm that that would satisfy that environment.py program you are running.

I am open to other solutions, as long as the wheel files generated don't get the setuptools dependency added.