Thread: [Shinken-devel] Recent concerns about setup.py
Status: Beta
Brought to you by:
naparuba
From: Rémi P. <re...@re...> - 2014-02-25 16:24:21
|
Hi there, There has been many discussions on IRC channel lately and some commits about the setup.py in order to fix some bugs regarding pip and also complains about its inner complexity. As a base statement to the following proposals, I think both distutils and setuptools are clearly NOT designed to install complex distributed daemons like Shinken. This has never been part of the goals of these tools. Trying to deeply hacking them for such purpose is totally error prone. Here are 2 things I think Shinken setup.py SHOULDN'T do: - Installing default init scripts: init scripts (and also systemd unit files) must be considered as distributions and OS specifics. Paths, shell functions, daemons launchers are not common on all distributions. Requirements and constraints also change from one release of a distribution to another. Writing working init scripts is hard, this responsability should be let to packagers. - Managing files owners: users and groups management should be done under the control of the package management system. Advanced package management systems are able to track groups/owners of installed files unless chown is done by scripts. I think that supporting 'pip install' or 'setup.py install' methods for production use of Shinken is a bad idea. Some /magical stuff/ should be removed from current setup.py so that RPM/DEB/whatever packaging effort could be done easier with these limitations in mind. This is only my own personal point of view. I'm looking forward to reading your feedback about these thoughts! -- Rémi Palancher |
From: Peter W. <pe...@sh...> - 2014-02-25 17:53:04
|
full agreement here. > On Feb 25, 2014, at 10:59 AM, Rémi Palancher <re...@re...> wrote: > > Hi there, > > There has been many discussions on IRC channel lately and some commits > about the setup.py in order to fix some bugs regarding pip and also > complains about its inner complexity. > > As a base statement to the following proposals, I think both distutils > and setuptools are clearly NOT designed to install complex distributed > daemons like Shinken. This has never been part of the goals of these > tools. Trying to deeply hacking them for such purpose is totally error > prone. > > Here are 2 things I think Shinken setup.py SHOULDN'T do: > > - Installing default init scripts: init scripts (and also systemd unit > files) must be considered as distributions and OS specifics. Paths, > shell functions, daemons launchers are not common on all distributions. > Requirements and constraints also change from one release of a > distribution to another. Writing working init scripts is hard, this > responsability should be let to packagers. > > - Managing files owners: users and groups management should be done > under the control of the package management system. Advanced package > management systems are able to track groups/owners of installed files > unless chown is done by scripts. > > I think that supporting 'pip install' or 'setup.py install' methods for > production use of Shinken is a bad idea. Some /magical stuff/ should be > removed from current setup.py so that RPM/DEB/whatever packaging effort > could be done easier with these limitations in mind. > > This is only my own personal point of view. I'm looking forward to > reading your feedback about these thoughts! > > -- > Rémi Palancher > > > ------------------------------------------------------------------------------ > Flow-based real-time traffic analytics software. Cisco certified tool. > Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer > Customize your own dashboards, set traffic alerts and generate reports. > Network behavioral analysis & security monitoring. All-in-one tool. > http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk > _______________________________________________ > Shinken-devel mailing list > Shi...@li... > https://lists.sourceforge.net/lists/listinfo/shinken-devel |
From: nap <nap...@gm...> - 2014-02-25 20:32:40
|
On Tue, Feb 25, 2014 at 4:59 PM, Rémi Palancher <re...@re...> wrote: > Hi there, > Hi :) > > There has been many discussions on IRC channel lately and some commits > about the setup.py in order to fix some bugs regarding pip and also > complains about its inner complexity. > yep, it's over complex for an easy task indeed. > > As a base statement to the following proposals, I think both distutils > and setuptools are clearly NOT designed to install complex distributed > daemons like Shinken. This has never been part of the goals of these > tools. Trying to deeply hacking them for such purpose is totally error > prone. > Yes, pip is designed for lib, setuptools can manage data_files, there is even some setup() parameters especially for this purpose, and it works quite well for easy cases like us (bunch of files with the same users and no real modifications into the files). > > Here are 2 things I think Shinken setup.py SHOULDN'T do: > > - Installing default init scripts: init scripts (and also systemd unit > files) must be considered as distributions and OS specifics. Paths, > shell functions, daemons launchers are not common on all distributions. > Requirements and constraints also change from one release of a > distribution to another. Writing working init scripts is hard, this > responsability should be let to packagers. > Writing init scripts that will be accepted by both debian AND redhat mainteners is not hard, it's impossible :) (why redhat/debian: they are the main distro we manage by default). It's possible to write scripts that work well on both without problems (and maintenable). But I already drop this fight years ago. We wrote init.d scripts that works well for upstream and make them usable. If packagers want to write new one that are fitting better the "distro" habits, that's their job indeed, and it's a really hard work. But I think you are wrong when you say we should not provide init.d script in upstream. It should be runnable without asking packager to do another part of the job or be forced to be run with packages (why if you run the master version? Like I do? :) ). Of course lot of the users will be running with packages, and so the init.d scripts from the package, and so we will provide help to packager to write init.d scripts that won't hurt the daemons or the user experience :) > - Managing files owners: users and groups management should be done > under the control of the package management system. Advanced package > management systems are able to track groups/owners of installed files > unless chown is done by scripts. > > I think that supporting 'pip install' or 'setup.py install' methods for > production use of Shinken is a bad idea. Some /magical stuff/ should be > removed from current setup.py so that RPM/DEB/whatever packaging effort > could be done easier with these limitations in mind. > yes, should be done by package when they are available. But one time again, upstream is not done to be run only by packages. And hopefully we only need files with shinken:shinken, so the user management is quite trivial. I don't plan to add such silly thing in the setup.py that will create the user if not existing or things like that. Because there of course it will complexify a LOT the package creation :p But running a recursive chown of the data_files with the shinken:shinken user/group is not so terrible or hard to manage, especially that the directories are quite easy to find (hum... etc, lib and log... I think we will manage this :) ). We can have a hard case if the user install with setup.py and then use shinken as package, that will mix all; but I think admin are not stupid and won't mix installation methods :) One last point: the update. I plan to only update the .py (the python lib and the default cli modules) and scripts with a setup.py update, and do not touch at the configuration or hard things like this, that only packages can manage in a good way (.rpmnew and co). > > This is only my own personal point of view. I'm looking forward to > reading your feedback about these thoughts! > No thanks for sharing your point of view :) As a packager I think I understand your thoughts, but one important point is that we should be not rely only on the packages only. As we are going quite quickly on the features, we should also be able to run in a "quite a good" way with upstream. The fact that I personnaly run my servers in this mode can make me not so objective of course :) One last important point is that the setup.py should be still usable to build a package in a good way, and currently the setup.py is just too complex to handle new files or filterings (like no init.d files option or thing like this). We will have to rewrite it for the 2.2 version, and this time I hope it will automatically make it possible to run with pip :) Jean > > -- > Rémi Palancher > > > > ------------------------------------------------------------------------------ > Flow-based real-time traffic analytics software. Cisco certified tool. > Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer > Customize your own dashboards, set traffic alerts and generate reports. > Network behavioral analysis & security monitoring. All-in-one tool. > > http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk > _______________________________________________ > Shinken-devel mailing list > Shi...@li... > https://lists.sourceforge.net/lists/listinfo/shinken-devel > |
From: Rémi P. <re...@re...> - 2014-02-26 10:14:56
|
On Tue, 25 Feb 2014 21:32:32 +0100, nap <nap...@gm...> wrote: > On Tue, Feb 25, 2014 at 4:59 PM, Rémi Palancher wrote: > There has been many discussions on IRC channel lately and some > commits > about the setup.py in order to fix some bugs regarding pip and also > complains about its inner complexity. > > yep, it's over complex for an easy task indeed. Cool, we all agree that setup.py is too complex! > As a packager I think I understand your thoughts, but one important > point is that we should be not rely only on the packages only. As we > are going quite quickly on the features, we should also be able to > run > in a "quite a good" way with upstream. The fact that I personnaly run > my servers in this mode can make me not so objective of course :) For particular early adopters or developer needs, it is not very hard to set up infrastructure to automatically build upstream rpm/deb/whatever packages at every commits or every nights. For that particular users, using last features would as simple as updating distribution packages. > One last important point is that the setup.py should be still usable > to build a package in a good way, and currently the setup.py is just > too complex to handle new files or filterings (like no init.d files > option or thing like this). We will have to rewrite it for the 2.2 > version, and this time I hope it will automatically make it possible > to run with pip :) OK but I can't get how you intend to significantly simplify the current setup.py logic _without_ removing all the included magical features. What's the plan? -- Rémi Palancher http://rezib.org |
From: nap <nap...@gm...> - 2014-02-26 14:47:55
|
Hi, On Wed, Feb 26, 2014 at 11:14 AM, Rémi Palancher <re...@re...> wrote: > [...]> yep, it's over complex for an easy task indeed. > > Cool, we all agree that setup.py is too complex! > Oh yes I'm more than agree! :D In fact, if I'm not wrong, the current setup.py allow some specific build with just config or just other part. We don't need so much things. We got just 2 main usercase for setup.py : * early adoter/tarball lover installation. We need to create some dir, regexp few lines in the ini files (soon no more need), and chmod shinken:shinken them * packagers : quite the same with just the python lib in fact if I'm not wrong, and maybe just some configirations files (just the cfg?) But currently It's hard, very very hard to separate the two. I don't understand the magic in the setup.py about the directory or variables. That's really specific to distutils, and we don't need so much things. We got a setup.cfg files with current arg (or even sys.argv args) and that's all. We don't need sub-class for some poor regexp, cp and chown... > > [...] > For particular early adopters or developer needs, it is not very hard > to set up infrastructure to automatically build upstream > rpm/deb/whatever packages at every commits or every nights. For that > particular users, using last features would as simple as updating > distribution packages. > I prefer to have a simple setup.py file and take time to code on features, instead of building packages that won't be used by packagers for distro. In fact for early adopters updates, a simple pip upgrade shinken or python setup.py update will only update all .py files, and that's all. We don't touch configuration as we don't break configuration files compat between update (even a 1.4 configuration will work on 2.0, it's just that the default module path will be /var/lib/shinken/modules). so not a big deal, even with a setup.py. (a concise one :) ) > > [...] > OK but I can't get how you intend to significantly simplify the current > setup.py logic _without_ removing all the included magical features. > What's the plan? > But I'll remove all the magic :) We got no choice there, we will have to mainly rewrite it. We did got a simple but working setup.py in the 0.8 version if I'm not wrong, we will take the god part of it and add the missing features we need (not a lot in fact). Was the acceptation of the setup.py enhancement pull-request an error? Yes, and believe me, now I'll look closer at this part of the project :) Jean > > -- > Rémi Palancher > http://rezib.org > |
From: Rémi P. <re...@re...> - 2014-02-26 16:42:10
|
On Wed, 26 Feb 2014 15:47:48 +0100, nap <nap...@gm...> wrote: > On Wed, Feb 26, 2014 at 11:14 AM, Rémi Palancher wrote: > [...] For particular early adopters or developer needs, it is not > very hard > to set up infrastructure to automatically build upstream > rpm/deb/whatever packages at every commits or every nights. For > that > particular users, using last features would as simple as updating > distribution packages. > > I prefer to have a simple setup.py file and take time to code on > features, instead of building packages that won't be used by > packagers > for distro. Hey, I am not asking _you_ to develop and build packages! Obviously, this effort could (and should) be done by distribution packagers themselves! Why are you pretending packagers won't use them? If they are part of the global effort, package maintainers will of course try to share the maximum amount of code between stable distribution packages and upstream bleeding-edge packages! Current deb/rpm/whatever package maintainers: what do you guys think about this? > We got no choice there, we will have to mainly rewrite it. We did got > a simple but working setup.py in the 0.8 version if I'm not wrong, we > will take the god part of it and add the missing features we need > (not > a lot in fact). Well, here is the setup.py in 0.8 according to github: https://github.com/naparuba/shinken/blob/e8825a56ae3f88a9cd71d7841c60100404947a6f/setup.py Not that much simpler! :) Maybe it was bit older? > Was the acceptation of the setup.py enhancement pull-request an > error? > Yes, and believe me, now I'll look closer at this part of the project > :) I guess you're talking about PR #1091. If yes, this is not what I'm talking about in this thread. -- Rémi Palancher http://rezib.org |
From: Hermann L. <Her...@iw...> - 2014-02-27 11:31:59
|
On Wed, Feb 26, 2014 at 05:41:55PM +0100, Rémi Palancher wrote: > On Wed, 26 Feb 2014 15:47:48 +0100, nap <nap...@gm...> wrote: > > On Wed, Feb 26, 2014 at 11:14 AM, Rémi Palancher wrote: > > [...] For particular early adopters or developer needs, it is not > > very hard > > to set up infrastructure to automatically build upstream > > rpm/deb/whatever packages at every commits or every nights. For > > that > > particular users, using last features would as simple as updating > > distribution packages. > > > > I prefer to have a simple setup.py file and take time to code on > > features, instead of building packages that won't be used by > > packagers > > for distro. > > Hey, I am not asking _you_ to develop and build packages! Obviously, > this effort could (and should) be done by distribution packagers > themselves! Why are you pretending packagers won't use them? > > If they are part of the global effort, package maintainers will of > course try to share the maximum amount of code between stable > distribution packages and upstream bleeding-edge packages! > > Current deb/rpm/whatever package maintainers: what do you guys think > about this? I'm only a system administrator, but from time to time we build debs with new software. And there a good setup.py helps realy, as then the build boils down to: $ python setup.py --command-packages=stdeb.command debianize $ dpkg-buildpackage -b -rfakeroot If there is an (recent) maintainer build in debian testing/unstable etc., sometimes the debian/ dir could also be copied and simply used with a newer version - but I have not found something good for an actual shinken version in the debian stuff yet. Please tell me if I missed something here. Just 2 more cents... Thanks for your work and greetings Hermann -- Netzwerkadministration/Zentrale Dienste, Interdiziplinaeres Zentrum fuer wissenschaftliches Rechnen der Universitaet Heidelberg IWR; INF 368; 69120 Heidelberg; Tel: (06221)54-8236 Fax: -5224 Email: Her...@iw... |
From: David D. <d.d...@si...> - 2014-02-28 02:16:03
|
Hello, with the real complexity of setup.py, I have rewriten it few hours ago and commited it in repository. Can people test it and say me if he have problems? For informations, the setup.py is compatible Linux, BSD and Windows (in fact can't test windows, I haven't have it and don't want it :p) David Durieux ++ Le Tue, 25 Feb 2014 16:59:13 +0100 Rémi Palancher <re...@re...> a écrit: > Hi there, > > There has been many discussions on IRC channel lately and some > commits about the setup.py in order to fix some bugs regarding pip > and also complains about its inner complexity. > > As a base statement to the following proposals, I think both > distutils and setuptools are clearly NOT designed to install complex > distributed daemons like Shinken. This has never been part of the > goals of these tools. Trying to deeply hacking them for such purpose > is totally error prone. > > Here are 2 things I think Shinken setup.py SHOULDN'T do: > > - Installing default init scripts: init scripts (and also systemd > unit files) must be considered as distributions and OS specifics. > Paths, shell functions, daemons launchers are not common on all > distributions. Requirements and constraints also change from one > release of a distribution to another. Writing working init scripts is > hard, this responsability should be let to packagers. > > - Managing files owners: users and groups management should be done > under the control of the package management system. Advanced package > management systems are able to track groups/owners of installed files > unless chown is done by scripts. > > I think that supporting 'pip install' or 'setup.py install' methods > for production use of Shinken is a bad idea. Some /magical stuff/ > should be removed from current setup.py so that RPM/DEB/whatever > packaging effort could be done easier with these limitations in mind. > > This is only my own personal point of view. I'm looking forward to > reading your feedback about these thoughts! > |
From: Hermann L. <Her...@iw...> - 2014-02-28 11:42:33
|
Hello, On Fri, Feb 28, 2014 at 02:56:01AM +0100, David DURIEUX wrote: > Hello, > > with the real complexity of setup.py, I have rewriten it few hours ago > and commited it in repository. > > Can people test it and say me if he have problems? > > For informations, the setup.py is compatible Linux, BSD and Windows (in > fact can't test windows, I haven't have it and don't want it :p) Just for curiosity a few results from my tries with "python setup.py --command-packages=stdeb.command debianize": NameError: name 'etc_root' is not defined going back one revision: KeyError: 'getpwnam(): name not found: shinken' KeyError: 'getgrnam(): name not found: shinken' OSError: [Errno 2] No such file or directory: '/var/run/shinken' So simple stupid packaging is only a dream at the moment ;-) Please feel free to ignore this comments and many thanks for all your work, greetings Hermann -- Netzwerkadministration/Zentrale Dienste, Interdiziplinaeres Zentrum fuer wissenschaftliches Rechnen der Universitaet Heidelberg IWR; INF 368; 69120 Heidelberg; Tel: (06221)54-8236 Fax: -5224 Email: Her...@iw... |
From: David D. <d.d...@si...> - 2014-02-28 11:48:52
|
Le Fri, 28 Feb 2014 12:42:23 +0100 Hermann Lauer <Her...@iw...> a écrit: >Hello, > >On Fri, Feb 28, 2014 at 02:56:01AM +0100, David DURIEUX wrote: >> Hello, >> >> with the real complexity of setup.py, I have rewriten it few hours >> ago and commited it in repository. >> >> Can people test it and say me if he have problems? >> >> For informations, the setup.py is compatible Linux, BSD and Windows >> (in fact can't test windows, I haven't have it and don't want it :p) > >Just for curiosity a few results from my tries with >"python setup.py --command-packages=stdeb.command debianize": > > NameError: name 'etc_root' is not defined I have fixed it this morning, but with commit of naparuba, it's know briken :/ >going back one revision: > KeyError: 'getpwnam(): name not found: shinken' > KeyError: 'getgrnam(): name not found: shinken' > OSError: [Errno 2] No such file or directory: '/var/run/shinken' > >So simple stupid packaging is only a dream at the moment ;-) > >Please feel free to ignore this comments and many >thanks for all your work, > greetings > Hermann > |
From: Rémi P. <re...@re...> - 2014-02-28 13:21:54
|
On Fri, 28 Feb 2014 02:56:01 +0100, David DURIEUX <d.d...@si...> wrote: > Hello, > > with the real complexity of setup.py, I have rewriten it few hours > ago > and commited it in repository. I have to admit I find quite weird and confusing you have rewritten from scratch a new setup.py and pushed it on master branch during RC freeze without even noticing package maintainers. Needless to say I feel a bit worried. Maybe you are an incredible genius and you have developed a both simple and fully-featured script everybody will embrace, but I seriously doubt that. Can you briefly explain what were your design goals? Is your script strictly compatible with the previous one? Are there known regressions? What are the planed evolutions? Can we at least determine here (together!) on some points regarding this setup.py: - what the script should/shoudn't do? - what the package maintainers should manager himself/herself? - how distributed setups are handled? - documentation packaging? Can we discuss about this? It shouldn't be hard to agree on these points. At the end, hopefuly both upstream developers and package maintainers will clearly understand what's their job and how to not bother others job. -- Rémi Palancher http://rezib.org |
From: David D. <d.d...@si...> - 2014-02-28 13:26:52
|
Le Fri, 28 Feb 2014 14:21:36 +0100 Rémi Palancher <re...@re...> a écrit: > On Fri, 28 Feb 2014 02:56:01 +0100, David DURIEUX > <d.d...@si...> wrote: >> Hello, >> >> with the real complexity of setup.py, I have rewriten it few hours >> ago >> and commited it in repository. > > I have to admit I find quite weird and confusing you have rewritten > from scratch a new setup.py and pushed it on master branch during RC > freeze without even noticing package maintainers. Needless to say I > feel a bit worried. It's simple the setup.py script not install correctly and may require be very very more time than rewrite it. > Maybe you are an incredible genius and you have developed a both > simple and fully-featured script everybody will embrace, but I > seriously doubt that. Can you briefly explain what were your design > goals? Is your script strictly compatible with the previous one? Are > there known regressions? What are the planed evolutions? I'm only crazy :p > Can we at least determine here (together!) on some points regarding > this setup.py: > > - what the script should/shoudn't do? > - what the package maintainers should manager himself/herself? > - how distributed setups are handled? > - documentation packaging? > > Can we discuss about this? It shouldn't be hard to agree on these > points. At the end, hopefuly both upstream developers and package > maintainers will clearly understand what's their job and how to not > bother others job. > Jean has add code for packagers, you can try with dev version and say if forget something. David ++ |
From: Rémi P. <re...@re...> - 2014-02-28 15:44:10
|
On Fri, 28 Feb 2014 14:26:20 +0100, David DURIEUX <d.d...@si...> wrote: > Le Fri, 28 Feb 2014 14:21:36 +0100 > Rémi Palancher <re...@re...> a écrit: >> Maybe you are an incredible genius and you have developed a both >> simple and fully-featured script everybody will embrace, but I >> seriously doubt that. Can you briefly explain what were your design >> goals? Is your script strictly compatible with the previous one? Are >> there known regressions? What are the planed evolutions? > > I'm only crazy :p Fine, good for you. Would you mind answering my questions? -- Rémi Palancher http://rezib.org |
From: David D. <d.d...@si...> - 2014-02-28 16:14:48
|
Le Fri, 28 Feb 2014 16:43:55 +0100 Rémi Palancher <re...@re...> a écrit: > On Fri, 28 Feb 2014 14:26:20 +0100, David DURIEUX > <d.d...@si...> wrote: >> Le Fri, 28 Feb 2014 14:21:36 +0100 >> Rémi Palancher <re...@re...> a écrit: >>> Maybe you are an incredible genius and you have developed a both >>> simple and fully-featured script everybody will embrace, but I >>> seriously doubt that. Can you briefly explain what were your design >>> goals? Is your script strictly compatible with the previous one? Are >>> there known regressions? What are the planed evolutions? >> >> I'm only crazy :p > > Fine, good for you. Would you mind answering my questions? Main goal is to be able to install on many different environments, like on Linux like on FreeBSD with specifications of each. The regression we have is the --root removed, it's little hard to re-add it. Is possible packagers can build (setup.py build) and after copy files in the right folders no without use this --root argument? David Durieux ++ |