From: Keith M. <kei...@us...> - 2010-02-08 19:00:33
Attachments:
profile.xml
mingw32-base.xml.lzma
|
With Cesar's recent efforts to progress a new GCC release, now seems like a good time to review progress on mingw-get, with a view to use it as the installer for GCC-4.5. Currently, I have it to the stage where, using the CLI client [*] built from current CVS head with the attached XML files to set up the configuration, I have successfully downloaded and installed GCC-3.4.5 (C, C++ and F77) into a clean VirtualBox hosted WinXP, with the command: $ mingw-get install g++ g77 [*] I first installed Console2 and MSYS-1.0.11, and used those to provide the command line environment, and used MSYS' tar initially, to extract mingw-get-0.1-mingw32-alpha-1-bin.tar.gz into an empty C:/MinGW directory; I guess we should consider other packaging options, to facilitate bootstrapping for users who don't want to install MSYS. Some comments on current status: 1) While current CVS will download and install packages, under the direction of the XML configuration files, it lacks a complementary uninstall capability; (I intend to work on this, when I've finished item (2)). 2) Installation tracking currently records only the names of the packages being installed; I have a partially developed and as yet untested patch which is intended to add a full files manifest. 3) I'm still considering the possibility of extending the tracking to not only record a per-package manifest of files as installed by each package, but also a system-wide files manifest with per-file records of providing packages; this would furnish a form of reference count for each file, so that we could accommodate multiple providers for common files, (at a cost of approximately doubling the size of the manifest, as it would be generated by item (2)). 4) I think that the installation manifest will need to include cross referenced dependency information, such that the dependants for each installed package are readily identifiable, (this is the reverse of the requirements specifications in current XML); this feature has not yet been developed. 5) Currently, only package lists which are explicitly identified in the user's local profile.xml will be processed; my idea is that we will eventually provide a master package-list.xml.lzma catalogue, as a single default download, and that this may recursively specify, download and merge any arbitrary number of secondary package list files, so permitting more granular package list maintenance. I'm sure I'll think of more as mingw-get matures. I'm wondering if we can now consider it ready for a first alpha release (CLI only), before I tackle the above? Any thoughts? -- Regards, Keith. P.S. If anyone would like a preview copy of the binary mingw-get tarball, as I've tested it so far, (it's 183K), please let me know, and I'll copy it to you privately. |
From: <ea...@us...> - 2010-02-08 20:15:55
|
Keith Marshall wrote: > With Cesar's recent efforts to progress a new GCC release, now seems > like a good time to review progress on mingw-get, with a view to use > it as the installer for GCC-4.5. Currently, I have it to the stage > where, using the CLI client [*] built from current CVS head with the > attached XML files to set up the configuration, I have successfully > downloaded and installed GCC-3.4.5 (C, C++ and F77) into a clean > VirtualBox hosted WinXP, with the command: > > $ mingw-get install g++ g77 > > [*] I first installed Console2 and MSYS-1.0.11, and used those to > provide the command line environment, and used MSYS' tar initially, > to extract mingw-get-0.1-mingw32-alpha-1-bin.tar.gz into an empty > C:/MinGW directory; I guess we should consider other packaging > options, to facilitate bootstrapping for users who don't want to > install MSYS. > > Some comments on current status: > > 1) While current CVS will download and install packages, under the > direction of the XML configuration files, it lacks a complementary > uninstall capability; (I intend to work on this, when I've finished > item (2)). > > This is fine for alpha release. > 2) Installation tracking currently records only the names of the > packages being installed; I have a partially developed and as yet > untested patch which is intended to add a full files manifest. > > This is fine for alpha release. > 3) I'm still considering the possibility of extending the tracking to > not only record a per-package manifest of files as installed by each > package, but also a system-wide files manifest with per-file records > of providing packages; this would furnish a form of reference count > for each file, so that we could accommodate multiple providers for > common files, (at a cost of approximately doubling the size of the > manifest, as it would be generated by item (2)). > > I think the benefit would be worth the increased size. > 4) I think that the installation manifest will need to include cross > referenced dependency information, such that the dependants for each > installed package are readily identifiable, (this is the reverse of > the requirements specifications in current XML); this feature has > not yet been developed. > > Yes needed eventually but not for alpha. > 5) Currently, only package lists which are explicitly identified in > the user's local profile.xml will be processed; my idea is that we > will eventually provide a master package-list.xml.lzma catalogue, as > a single default download, and that this may recursively specify, > download and merge any arbitrary number of secondary package list > files, so permitting more granular package list maintenance. > > I can see this being used by third party vendors to provide their own version of a mingw-package.xml. Would be beneficial for them to not have to worry with the burden of carrying a full distribution of mingw packages. > I'm sure I'll think of more as mingw-get matures. I'm wondering if > we can now consider it ready for a first alpha release (CLI only), > before I tackle the above? Any thoughts? > I think it is up to you mainly but I can see the benefit of doing so. Perhaps unannounced alpha release would allow for the benefit of asking trusted users to give it a whirl. Earnie. |
From: Keith M. <kei...@us...> - 2010-02-17 20:39:00
|
On Monday 08 February 2010 20:15:40 ea...@us... wrote: > > I'm sure I'll think of more as mingw-get matures. I'm wondering > > if we can now consider it ready for a first alpha release (CLI > > only), before I tackle the above? Any thoughts? > > I think it is up to you mainly but I can see the benefit of doing > so. Perhaps unannounced alpha release would allow for the > benefit of asking trusted users to give it a whirl. Okay. I've uploaded an alpha-1 release to SF. You may find it in the "Automated MinGW Installer/mingw-get" folder. The distribution manifest is "package-index.xml.lzma". Currently, this mimics (approximately) the installation footprint of the old MinGW-5.1.6.exe installer, but with the advantage that it may be updated to deliver more recent versions, INCLUDING DEPENDENCIES ON ADDITIONAL PACKAGES, by simply editing the XML content, and then uploading the modified version to replace the existing file. If any of you would like to "give it a whirl", the release notes explain how to install and invoke it. If any of you would care to hack the distribution manifest, to add additional packages, (GDB, Chris? GCC-4, Cesar), please do so; I think it is fairly self-explanatory, but in case of doubt, do please ask. (At present, that distribution manifest does need to be one aggregate of all supported packages; once I've got the uninstaller functionality in place, I'll look into segregating it into more easily maintained, logically distributed subsets, each with its own manifest file). -- Regards, Keith. |
From: Keith M. <kei...@us...> - 2010-02-09 21:23:56
|
On Monday 08 February 2010 20:15:40 Earnie Boyd wrote: > > Some comments on current status: > > > > 1) While current CVS will download and install packages, under > > the direction of the XML configuration files, it lacks a > > complementary uninstall capability; (I intend to work on this, > > when I've finished item (2)). > > This is fine for alpha release. Okay. My first thought was to re-read the original package tarball to generate the files manifest for the uninstaller. Many users may keep it around anyway, but if not, it would have to be downloaded again for the uninstall process. Either way, that would incur a greater overhead than writing a full files manifest at install time, but more critically, it would preclude the option of reference counting -- at least easily. > > 2) Installation tracking currently records only the names of the > > packages being installed; I have a partially developed and as > > yet untested patch which is intended to add a full files > > manifest. > > This is fine for alpha release. I've just tested that patch in Wine, (repeating the installation but without downloading again); seems to DTRT, so I may as well roll it into the first alpha. > > 3) I'm still considering the possibility of extending the > > tracking to not only record a per-package manifest of files as > > installed by each package, but also a system-wide files manifest > > with per-file records of providing packages; this would furnish > > a form of reference count for each file, so that we could > > accommodate multiple providers for common files, (at a cost of > > approximately doubling the size of the manifest, as it would be > > generated by item (2)). > > I think the benefit would be worth the increased size. I'm leaning towards that opinion too, so are we great minds thinking alike, or are we the fools who seldom differ? :-) > > 4) I think that the installation manifest will need to include > > cross referenced dependency information, such that the > > dependants for each installed package are readily identifiable, > > (this is the reverse of the requirements specifications in > > current XML); this feature has not yet been developed. > > Yes needed eventually but not for alpha. Okay. My only residual reticence stems from the need to be able retrospectively add information into to an existing installation manifest, as it becomes necessary, but without requiring a new installation. However, I expect that an ability to validate and reconstruct the database on demand will be a useful feature, which should be provided eventually anyway. > > 5) Currently, only package lists which are explicitly identified > > in the user's local profile.xml will be processed; my idea is > > that we will eventually provide a master package-list.xml.lzma > > catalogue, as a single default download, and that this may > > recursively specify, download and merge any arbitrary number of > > secondary package list files, so permitting more granular > > package list maintenance. > > I can see this being used by third party vendors to provide their > own version of a mingw-package.xml. There is nothing in the current design which would preclude that. > Would be beneficial for them to not have to worry with the burden > of carrying a full distribution of mingw packages. They would customise profile.xml, to point to their own repository. There, they would provide their own package-list.xml.lzma, which would identify their own package collection(s); no need for them to host *any* MinGW specific content. > > I'm sure I'll think of more as mingw-get matures. I'm wondering > > if we can now consider it ready for a first alpha release (CLI > > only), before I tackle the above? Any thoughts? > > I think it is up to you mainly but I can see the benefit of doing > so. Perhaps unannounced alpha release would allow for the > benefit of asking trusted users to give it a whirl. I'll proceed on that basis, in the next few days. -- Regards, Keith. |
From: Charles W. <cwi...@us...> - 2010-02-10 01:05:50
|
Keith Marshall wrote: > On Monday 08 February 2010 20:15:40 Earnie Boyd wrote: >> This is fine for alpha release. > > I've just tested that patch in Wine, (repeating the installation but > without downloading again); seems to DTRT, so I may as well roll it > into the first alpha. > >>> 3) I'm still considering the possibility of extending the >>> tracking to not only record a per-package manifest of files as >>> installed by each package, but also a system-wide files manifest >>> with per-file records of providing packages; this would furnish >>> a form of reference count for each file, so that we could >>> accommodate multiple providers for common files, (at a cost of >>> approximately doubling the size of the manifest, as it would be >>> generated by item (2)). >> I think the benefit would be worth the increased size. I'm confused. Naturally, for uninstall, you need to keep a list of all files associated with a specific package/tarball. Once you have that data, then you can process it in multiple ways (e.g. derive the reverse lookup: file->package fairly easily). So, I don't really see the functional need for including the reverse lookup in the actual manifest, except as a speed optimization. Furthermore, the idea of merging all of the file lists from all of the packages into one giant manifest.xml just sounds...icky. One file gets scrogged, and your entire installation is hosed (shades of the rpmdb corruption issue). (Plus, it's a maintainance nightmare for US, to keep that file up-to-date on sourceforge) I could see instead storing the "primary" data in a bunch of separate manifests (/mingw/manifests/gcc-4.3.4-3-mingw32-bin.xml and the like), with a single manifest-of-manifests that refers to each of those other ones. The reverse lookup info is less critical; if it's missing or corrupted it can always be regenerated from the "primary" data. For speed, I could see this additional info being stored in a mysqlite db or something, or a different .xml file separate from all the ${PKG}-${VER}-${SUBSYS}-${TYPE}.xml ones. In any case, parsing a giant xml with 10000 entries (my msys+mingw installation, which I "manage" using a collection of custom scripts, currently tracks 9399 installed filed) is going to be quite slow, so for this "generated" database, I could see using a different storage medium, like an actual db engine. > I'm leaning towards that opinion too, so are we great minds thinking > alike, or are we the fools who seldom differ? :-) I think the *data* will be useful merely as a speed optimization, since all of the information COULD be derived on-the-fly at runtime from the primary manifest information (but then again, xml-string parsing of that portion of a single-giant-manifest may eat up all the time savings from precomputing the reverse-file-provenance info!) However, as I said above, I'm more concerned that the larger this single giant manifest gets, the more susceptible it becomes to corruption issues. If the manifest info is spread out into multiple files, then a single pkg manifest file becoming corrupted is easily fixed simply by re-installing that single package. Not so if it's a system-wide rpmdb kind of thing. >>> 4) I think that the installation manifest will need to include >>> cross referenced dependency information, such that the >>> dependants for each installed package are readily identifiable, >>> (this is the reverse of the requirements specifications in >>> current XML); this feature has not yet been developed. Again, as above, this reverse-lookup info can be derived on-the-fly from the "primary" dependency info. Pre-computing the reverse dependencies is just a speed optimization. >> Yes needed eventually but not for alpha. Agreed. > Okay. My only residual reticence stems from the need to be able > retrospectively add information into to an existing installation > manifest, as it becomes necessary, but without requiring a new > installation. Easily obtained if you use the manifest-of-manifests approach. > However, I expect that an ability to validate and > reconstruct the database on demand will be a useful feature, which > should be provided eventually anyway. Again, if these two reverse-lookup entities are first computed on-the-fly (during mingw-get's initial startup) for the alpha/beta release, and only later (beta/gamma) you add the ability to persist it in some form, then reconstructing THOSE pieces becomes a simple matter of: rm file-provenance.db rm pkg-required-by.db ./mingw-get --persist-dbs >>> 5) Currently, only package lists which are explicitly identified >>> in the user's local profile.xml will be processed; my idea is >>> that we will eventually provide a master package-list.xml.lzma >>> catalogue, as a single default download, and that this may >>> recursively specify, download and merge any arbitrary number of >>> secondary package list files, so permitting more granular >>> package list maintenance. hmm... master package-lists-xml.tar.lzma? >>> I'm sure I'll think of more as mingw-get matures. I'm wondering >>> if we can now consider it ready for a first alpha release (CLI >>> only), before I tackle the above? Any thoughts? >> I think it is up to you mainly but I can see the benefit of doing >> so. Perhaps unannounced alpha release would allow for the >> benefit of asking trusted users to give it a whirl. > > I'll proceed on that basis, in the next few days. Looking forward to it. Aside: whether mingw-get uses a single manifest, or a collection of them, where would it be stored? If you use mingw-get to install both mingw packages (into c:\MinGW, for instance) and msys packages (into c:\msys\1.0) -- or only one, or only the other -- you'd like to have just one manifest location, not two. But...what if I want to maintain three different MinGW's and two different MSYS's (which I do, on my personal machine)? It makes sense for the manifest info to be stored "with" the stuff it describes. But where IS that, in this scenario? Or, should mingw-get be treated, conceptually, as associated with a single root? If you want to install *separate* MSYS and MinGW, then you should run mingw-get with pkg-descriptor-mingw once (and it puts its metadata in {root1}/manifests/), and then run mingw-get with pkg-descriptor-msys a second time (so it puts its metadata this time in {root2}/manifests/), and thus manage all of the separate roots, well, separately? -- Chuck |
From: Keith M. <kei...@us...> - 2010-02-10 20:04:05
|
On Wednesday 10 February 2010 01:05:28 Charles Wilson wrote: > Keith Marshall wrote: > > On Monday 08 February 2010 20:15:40 Earnie Boyd wrote: > >> This is fine for alpha release. > > > > I've just tested that patch in Wine, (repeating the installation > > but without downloading again); seems to DTRT, so I may as well > > roll it into the first alpha. > > > >>> 3) I'm still considering the possibility of extending the > >>> tracking to not only record a per-package manifest of files as > >>> installed by each package, but also a system-wide files > >>> manifest with per-file records of providing packages; this > >>> would furnish a form of reference count for each file, so that > >>> we could accommodate multiple providers for common files, (at > >>> a cost of approximately doubling the size of the manifest, as > >>> it would be generated by item (2)). > >> > >> I think the benefit would be worth the increased size. > > I'm confused. Confused? Maybe. You definitely seem to be misunderstanding the distinction I would like to make, between a distribution manifest and an installation manifest; the former is what we would host on SourceForge, (an example being the mingw32-base.xml.lzma file I attached to my original post in this thread); the latter is a record of a particular installation, and is maintained exclusively on the local host for that installation. > Naturally, for uninstall, you need to keep a list of all files > associated with a specific package/tarball. Not necessarily. The original package tarball can serve as its own files manifest, so, as a bare minimum, only the list of installed packages needs to be recorded, since even the list of files can be reconstructed, simply by reading the tarball headers again. > Once you have that > data, then you can process it in multiple ways (e.g. derive the > reverse lookup: file->package fairly easily). You can't, if all you've recorded is a list of files; you also need to record the package association for each individual file. It is easy to reconstruct a files list from a package; it is not at all easy to get from a file name to the providing package, if the association isn't recorded. > So, I don't really see the functional need for including the > reverse lookup in the actual manifest, except as a speed > optimization. To some extent, I'm thinking aloud. I don't want to record anything which can be readily computed, but OTOH, I don't want to omit something which is needed, yet difficult to reverse engineer. > Furthermore, the idea of merging all of the file lists from all of > the packages into one giant manifest.xml just sounds...icky. One > file gets scrogged, and your entire installation is hosed (shades > of the rpmdb corruption issue). My present design has one installation manifest file per sysroot; running mingw-get with my sample profile.xml yields: $ ls var/lib/mingw-get/data/sys* var/lib/mingw-get/data/sysroot-0-008-91a1-ee654e.xml var/lib/mingw-get/data/sysroot-0-011-eb3d-1bf874.xml $ cat var/lib/mingw-get/data/sysroot-0-008-91a1-ee654e.xml <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <sysroot id="sysroot-0-008-91a1-ee654e" path="c:/MinGW"> <installed tarname="w32api-3.13-mingw32-dev.tar.gz"> <dir path="c:/MinGW/include" /> <file path="c:/MinGW/include/accctrl.h" /> ... </installed> ... </sysroot> $ cat var/lib/mingw-get/data/sysroot-0-011-eb3d-1bf874.xml <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <sysroot id="sysroot-0-011-eb3d-1bf874" path="c:/MSYS/1.0" /> (The latter is empty, because I haven't made any MSYS distribution manifest yet, for a trial installation to that sysroot). > (Plus, it's a maintainance > nightmare for US, to keep that file up-to-date on sourceforge) Nope. I'm discussing content of an installation manifest here; that is neither manually maintained, nor to be kept on SourceForge. > I could see instead storing the "primary" data in a bunch of > separate manifests (/mingw/manifests/gcc-4.3.4-3-mingw32-bin.xml > and the like), with a single manifest-of-manifests that refers to > each of those other ones. The current implementation will already permit multiple distribution manifests, subject to the constraints that all must be individually specified in var/lib/mingw-get/data/profile.xml,and that all must be uniquely named. Ultimately, I plan to support a more flexible setup based on a "manifest-of-manifests" concept, and with some form of file/repository name hashing to resolve possible name conflicts, if those manifests are distributed over multiple repositories. As noted above, I currently have one installation manifest per sysroot; conceivably, that could be decomposed to use a separate file per package, with the sysroot manifest tying them all together, but to do so will require more effort; we can move toward that arrangement later, if it's considered desirable. In the case of distribution manifests, I don't want them decomposed to any granularity finer than the scope of a single XML 'package' element, as it's conceptually outlined in the mingw32-base.xml.lzma manifest I attached previously; (that is, no more than one file per package, with all advertised releases of all related component packages listed together). > The reverse lookup info is less critical; if it's missing or > corrupted it can always be regenerated from the "primary" data. Provided that "primary" data encapsulates sufficient meta-information to permit such regeneration; my principal concern, for now, is not to omit anything which it may not be possible to regenerate later. > For speed, I could see this additional info being stored in a > mysqlite db or something, or a different .xml file separate from > all the ${PKG}-${VER}-${SUBSYS}-${TYPE}.xml ones. In any case, > parsing a giant xml with 10000 entries (my msys+mingw > installation, which I "manage" using a collection of custom > scripts, currently tracks 9399 installed filed) is going to be > quite slow, so for this "generated" database, I could see using a > different storage medium, like an actual db engine. I won't exclude that, as a possible design consideration, but it is definitely one for later. > > I'm leaning towards that opinion too, so are we great minds > > thinking alike, or are we the fools who seldom differ? :-) > > I think the *data* will be useful merely as a speed optimization, > since all of the information COULD be derived on-the-fly at > runtime from the primary manifest information And this is my main concern. Say I have installed foo-1.0-mingw32 at a time when that is present in the distribution manifest, but by the time I come to upgrade it it has gone through several evolutionary steps, becoming foo-2.0-mingw32, and foo-1.0-mingw32 is no longer advertised in the distribution manifest, (and I've synchronised my locally cached copy with the repository master). If I didn't capture sufficient of the original package meta-data, from the distribution manifest, into my installation manifest, at the time when I installed foo-1.0, then I may not have sufficient information available to regenerate it, to facilitate preparation for an upgrade. > (but then again, > xml-string parsing of that portion of a single-giant-manifest may > eat up all the time savings from precomputing the > reverse-file-provenance info!) Sure. It isn't the speed issue which is my primary concern; it is ensuring that sufficient meta-data is retained to permit the reverse look up, at any later time when it may be needed. > However, as I said above, I'm more concerned that the larger this > single giant manifest gets, the more susceptible it becomes to > corruption issues. If the manifest info is spread out into > multiple files, It is, to the extent described above. > then a single pkg manifest file becoming corrupted > is easily fixed simply by re-installing that single package. Not > so if it's a system-wide rpmdb kind of thing. > > >>> 4) I think that the installation manifest will need to include > >>> cross referenced dependency information, such that the > >>> dependants for each installed package are readily > >>> identifiable, (this is the reverse of the requirements > >>> specifications in current XML); this feature has not yet been > >>> developed. > > Again, as above, this reverse-lookup info can be derived > on-the-fly from the "primary" dependency info. Pre-computing the > reverse dependencies is just a speed optimization. Yes, but the reverse look up may be significantly more difficult than the forward identification; package foo-2.0-mingw32 may specify: <requires ge="bar-*-mingw32-dll-0" /> but package bar-1.5-mingw32 will not specify that it is required by foo, (or by any of the myriad other packages that may require it). It is relatively easy to add a note to the installation record for bar, that it is required by foo, at the time when foo is installed, (and to remove that note when foo is subsequently removed); it is achievable, but significantly more difficult to perform the reverse look up, to identify which packages will break if bar is removed prematurely. > >> Yes needed eventually but not for alpha. > > Agreed. > > > Okay. My only residual reticence stems from the need to be able > > retrospectively add information into to an existing installation > > manifest, as it becomes necessary, but without requiring a new > > installation. > > Easily obtained if you use the manifest-of-manifests approach. Umm. No, it isn't. The manifest-of-manifests approach makes it easy for the distribution manifest to evolve; it doesn't solve the problem of missing, but necessary, meta-data in the installation manifests, (regardless of how distributed they may be). > > However, I expect that an ability to validate and > > reconstruct the database on demand will be a useful feature, > > which should be provided eventually anyway. > > Again, if these two reverse-lookup entities are first computed > on-the-fly (during mingw-get's initial startup) for the alpha/beta > release, and only later (beta/gamma) you add the ability to > persist it in some form, then reconstructing THOSE pieces becomes > a simple matter of: rm file-provenance.db > rm pkg-required-by.db > ./mingw-get --persist-dbs > > >>> 5) Currently, only package lists which are explicitly > >>> identified in the user's local profile.xml will be processed; > >>> my idea is that we will eventually provide a master > >>> package-list.xml.lzma catalogue, as a single default download, > >>> and that this may recursively specify, download and merge any > >>> arbitrary number of secondary package list files, so > >>> permitting more granular package list maintenance. > > hmm... master package-lists-xml.tar.lzma? No. I really do mean a discrete collection of package-list.xml.lzma files at the point of hosting on SourceForge, because that's what I've coded mingw-get to expect; it decodes an lzma data stream directly through a wininet connection, to save a plain XML copy of the single package-list.xml file locally. (It could be just as easily an xz stream, or an uncompressed stream, but not so easily gz or bz2, because those formats assume that the input data comes from a file stream, and are less easily adapted to use wininet streams). > >>> I'm sure I'll think of more as mingw-get matures. I'm > >>> wondering if we can now consider it ready for a first alpha > >>> release (CLI only), before I tackle the above? Any thoughts? > >> > >> I think it is up to you mainly but I can see the benefit of > >> doing so. Perhaps unannounced alpha release would allow for > >> the benefit of asking trusted users to give it a whirl. > > > > I'll proceed on that basis, in the next few days. > > Looking forward to it. > > Aside: whether mingw-get uses a single manifest, or a collection > of them, where would it be stored? If you use mingw-get to > install both mingw packages (into c:\MinGW, for instance) and msys > packages (into c:\msys\1.0) -- or only one, or only the other -- > you'd like to have just one manifest location, not two. See my illustration, above. There will be a separate installation manifest for each sysroot. (The manifest name is hashed from the sysroot path name itself -- see the CVS source for details -- and both are stored in <mingw-get-root>/var/lib/mingw-get/data, where <mingw-get-root> is derived from the GetModuleFileName() for the <mingw-get-root>/bin/mingw-get.exe itself). > But...what if I want to maintain three different MinGW's and two > different MSYS's (which I do, on my personal machine)? It makes > sense for the manifest info to be stored "with" the stuff it > describes. But where IS that, in this scenario? You could handle that with distinct system-map specifications in your <mingw-get-root>/var/lib/mingw-get/data/profile.xml, (which is yours to configure as you wish). However, I haven't yet implemented the option to select an alternative system-map, at mingw-get start-up. Alternatively, you could put a separate mingw-get installation in each of your separate MinGW installations, and segregate based on which particular /mingw/bin/mingw-get.exe you run; for each case, the manifests will be found in /mingw/var/lib/mingw/data. > Or, should mingw-get be treated, conceptually, as associated with > a single root? No. The sample profile.xml already specifies support for multiple distinct roots, within a system-map association, (one root for each named subsystem); see the comments in profile.xml for more info. > If you want to install *separate* MSYS and MinGW, > then you should run mingw-get with pkg-descriptor-mingw once (and > it puts its metadata in {root1}/manifests/), and then run > mingw-get with pkg-descriptor-msys a second time (so it puts its > metadata this time in {root2}/manifests/), and thus manage all of > the separate roots, well, separately? No. It uses a central storage location, with hashed file names to segregate distinct content for installation manifests, as described above. Distribution manifests go in the same location, with the same base name as the repository file; (this may also need to be hashed, to accommodate possible name clashes from disparate repository URLs). -- Regards, Keith. |
From: Charles W. <cwi...@us...> - 2010-02-11 02:16:22
|
Keith Marshall wrote: > lots of stuff Thanks for your detailed answer. I think I understand a lot better now. -- Chuck |
From: Keith M. <kei...@us...> - 2010-02-11 21:24:02
|
On Thursday 11 February 2010 02:15:56 Charles Wilson wrote: > Thanks for your detailed answer. I think I understand a lot better > now. You're very welcome. Thank you for your insight. Providing such clarifications helps me to consolidate my ideas, and to question my design decisions, all of which helps to improve the product. As a further clarification: > > > Naturally, for uninstall, you need to keep a list of all files > > > associated with a specific package/tarball. > > > > Not necessarily. The original package tarball can serve as its > > own files manifest, so, as a bare minimum, only the list of > > installed packages needs to be recorded, since even the list of > > files can be reconstructed, simply by reading the tarball > > headers again. Actually, just a list of installed packages alone is insufficient; a more realistic expression of the minimum requirement would be: 1) The "canonical" package name, (as defined by our package naming convention); this provides the meta-data which mingw-get will use to definitively identify the package. 2) The "real" tarball name, if it differs from the canonical name; (it shouldn't for new packages, but may for older ones); this is the file to be consulted, for regeneration of the files list. 3) Any "requires" specs which may have been attached to the original distribution manifest for the package; this is required to permit reconstruction of the dependency graph, in the event of the original package specification having been withdrawn from the distribution manifest. Have I missed anything else? > > I currently have one installation manifest per > > sysroot; conceivably, that could be decomposed to use a separate > > file per package, with the sysroot manifest tying them all > > together, but to do so will require more effort; we can move > > toward that arrangement later, if it's considered desirable. The more I think about it, the more attractive such decomposition of the installation manifest seems to become. I won't delay "alpha-1" for the sake of implementing it, but I'll aim for including it in "alpha-2", to avoid having to re-engineer the uninstaller later, to accommodate the (fairly significant) change in manifest structure. -- Regards, Keith. |
From: Charles W. <cwi...@us...> - 2010-02-12 06:10:56
|
Keith Marshall wrote: > On Thursday 11 February 2010 02:15:56 Charles Wilson wrote: >>>> Naturally, for uninstall, you need to keep a list of all files >>>> associated with a specific package/tarball. >>> Not necessarily. The original package tarball can serve as its >>> own files manifest, so, as a bare minimum, only the list of >>> installed packages needs to be recorded, since even the list of >>> files can be reconstructed, simply by reading the tarball >>> headers again. They could be reconstructed in this way, but I'd prefer to record the filelist information, for each tarball, explicitly. If that data is missing, THEN go look for the .tarball and regenerate it. That way you don't need to keep your installation media around (I'm thinking CDROM snapshots of the tarballs; you wouldn't copy the tarballs onto your harddrive in that case, but you'd still want to record the file associations). Also, it's faster to read a list of files from a text file (or even .xml) than to untarlzma... > Actually, just a list of installed packages alone is insufficient; > a more realistic expression of the minimum requirement would be: 0) the files associated with a specific tarball. Probably "keyed" according to the canonical package name, rather than the tarball name. > 1) The "canonical" package name, (as defined by our package naming > convention); this provides the meta-data which mingw-get will use to > definitively identify the package. > > 2) The "real" tarball name, if it differs from the canonical name; > (it shouldn't for new packages, but may for older ones); this is the > file to be consulted, for regeneration of the files list. > > 3) Any "requires" specs which may have been attached to the original > distribution manifest for the package; this is required to permit > reconstruction of the dependency graph, in the event of the original > package specification having been withdrawn from the distribution > manifest. > > Have I missed anything else? A couple of levels of "meta group"? a) source code association: the 'gettext' metagroup includes all of the binary tarballs derived from the gettext src package, which are currently: gettext-0.17-1-mingw32-bin gettext-0.17-1-mingw32-dev gettext-0.17-1-mingw32-doc gettext-0.17-1-mingw32-ext gettext-0.17-1-mingw32-lic libasprintf-0.17-1-mingw32-dll-0 libgettextpo-0.17-1-mingw32-dll-0 libintl-0.17-1-mingw32-dll-8 plus gettext-0.17-1-mingw32-src Effectively, this is just an annotation for each non-src package, that says "There's my source code, over there:". Typically this will be "foo-*-src" but not always: libintl-dll-8 --> gettext-src. (Conversely, this could be represented instead as a metadata tag that all -src packages have, listing which non-src packages they are "responsible" for; the reverse lookup data could then be generated on the fly or cached). b) license package association: "My license info is in /that/ package, over there". Typically this will be "foo-*-lic", but not always. For instance, libasprintf-dll-0 --> gettext-lic. Similarly, this could instead be represented as a metadata tag that all -lic packages -- or any package that provides its own license text -- could have, and the reverse lookup data could be generated from that). c) metapackage: "msys basic installation" would include all the necessary -bin and -dll (and -lic) packages to reproduce, in effect, the old MSYS monolithic installation footprint. "msys DTK" would add the -bin, -dll, and related packages to reproduce the old DTK. And finally, "msys dvlpr" would include msys-gcc, msys-binutils, msysCORE-dev, w32api-msys-dev, all the other *-msys-dev packages, plus the msys-autotool packages needed to develop msys applications. So, if we have these metapackages, then perhaps each individual tarball also needs to specify its membership(s?). But none of this is necessary for alpha-1 or even alpha-2. And anything resembling (c) might be a few more revisions down the road. >>> I currently have one installation manifest per >>> sysroot; conceivably, that could be decomposed to use a separate >>> file per package, with the sysroot manifest tying them all >>> together, but to do so will require more effort; we can move >>> toward that arrangement later, if it's considered desirable. > > The more I think about it, the more attractive such decomposition of > the installation manifest seems to become. I won't delay "alpha-1" > for the sake of implementing it, but I'll aim for including it in > "alpha-2", to avoid having to re-engineer the uninstaller later, to > accommodate the (fairly significant) change in manifest structure. > Ack. -- Chuck |
From: Keith M. <kei...@us...> - 2010-02-17 20:39:13
|
On Friday 12 February 2010 06:10:44 Charles Wilson wrote: > Keith Marshall wrote: > > On Thursday 11 February 2010 02:15:56 Charles Wilson wrote: > >>>> Naturally, for uninstall, you need to keep a list of all > >>>> files associated with a specific package/tarball. > >>> > >>> Not necessarily. The original package tarball can serve as > >>> its own files manifest, so, as a bare minimum, only the list > >>> of installed packages needs to be recorded, since even the > >>> list of files can be reconstructed, simply by reading the > >>> tarball headers again. > > They could be reconstructed in this way, but I'd prefer to record > the filelist information, for each tarball, explicitly. If that > data is missing, THEN go look for the .tarball and regenerate it. > That way you don't need to keep your installation media around > (I'm thinking CDROM snapshots of the tarballs; you wouldn't copy > the tarballs onto your harddrive in that case, but you'd still > want to record the file associations). > > Also, it's faster to read a list of files from a text file (or > even .xml) than to untarlzma... Sure. I was just (pedantically) pointing out that it is possible to regenerate a files manifest, at any time, by reading the tarball... > > Actually, just a list of installed packages alone is > > insufficient; a more realistic expression of the minimum > > requirement would be: > > 0) the files associated with a specific tarball. Probably "keyed" > according to the canonical package name, rather than the tarball > name. ...so, yes, I omitted this because it isn't technically necessary to satisfy a "minimum requirement" criterion, but I do plan to do this, (and indeed, already have in a rudimentary fashion), anyway. > > 1) The "canonical" package name, (as defined by our package > > naming convention); this provides the meta-data which mingw-get > > will use to definitively identify the package. > > > > 2) The "real" tarball name, if it differs from the canonical > > name; (it shouldn't for new packages, but may for older ones); > > this is the file to be consulted, for regeneration of the files > > list. > > > > 3) Any "requires" specs which may have been attached to the > > original distribution manifest for the package; this is required > > to permit reconstruction of the dependency graph, in the event > > of the original package specification having been withdrawn from > > the distribution manifest. > > > > Have I missed anything else? > > A couple of levels of "meta group"? > > a) source code association: the 'gettext' metagroup includes all > of the binary tarballs derived from the gettext src package, which > are currently: gettext-0.17-1-mingw32-bin > gettext-0.17-1-mingw32-dev > gettext-0.17-1-mingw32-doc > gettext-0.17-1-mingw32-ext > gettext-0.17-1-mingw32-lic > libasprintf-0.17-1-mingw32-dll-0 > libgettextpo-0.17-1-mingw32-dll-0 > libintl-0.17-1-mingw32-dll-8 I was planning to accommodate this, (expanding on your example), with a <source tarname="gettext-&-mingw32-src.tar.gz" /> element, embedded within the <package name="gettext"> container element, at the same level as the individual <component> elements defining each of those component packages. Apart from "&" being a really bad choice for "identically matching version", (I'm thinking of using "%" as, perhaps, a more suitable alternative -- any other suggestions?), is there any reason why that will not suffice? > plus > > gettext-0.17-1-mingw32-src > > Effectively, this is just an annotation for each non-src package, > that says "There's my source code, over there:". Typically this > will be "foo-*-src" but not always: libintl-dll-8 --> gettext-src. > (Conversely, this could be represented instead as a metadata tag > that all -src packages have, listing which non-src packages they > are "responsible" for; the reverse lookup data could then be > generated on the fly or cached). I think my idea can work both ways: look for the <source> element embedded within the <component> element itself, and it becomes a "go there for my source" reference; if it isn't there, look for it in the containing <package> element, and it becomes a "here is the source for all my sibling <component> packages" reference. > b) license package association: "My license info is in /that/ > package, over there". Typically this will be "foo-*-lic", but not > always. For instance, libasprintf-dll-0 --> gettext-lic. > Similarly, this could instead be represented as a metadata tag > that all -lic packages -- or any package that provides its own > license text -- could have, and the reverse lookup data could be > generated from that). Easily handled with a <licence tarname="foo-&-mingw32-lic.tar.gz" /> element, analogous to the <source> reference. > c) metapackage: "msys basic installation" would include all the > necessary -bin and -dll (and -lic) packages to reproduce, in > effect, the old MSYS monolithic installation footprint. "msys > DTK" would add the -bin, -dll, and related packages to reproduce > the old DTK. And finally, "msys dvlpr" would include msys-gcc, > msys-binutils, msysCORE-dev, w32api-msys-dev, all the other > *-msys-dev packages, plus the msys-autotool packages needed to > develop msys applications. So, if we have these metapackages, > then perhaps each individual tarball also needs to specify its > membership(s?). Yes. This is already an identified requirement on my to-do list; a "virtual" package class, associated with no physical package file or download URI, which exists exclusively as a container for a meta-package collection, identified by <requires> specs. Naturally, the installation manifest would need to record this, in whatever format we eventually adopt for installed dependency tracking. > But none of this is necessary for alpha-1 or even alpha-2. And > anything resembling (c) might be a few more revisions down the > road. (c) is actually something I would like to get to fairly soon; it may not be too difficult to implement, at least in a rudimentary manner. -- Regards, Keith. |
From: Charles W. <cwi...@us...> - 2010-02-17 23:14:10
|
On Wed, 17 Feb 2010 15:25 +0000, "Keith Marshall" wrote: > On Friday 12 February 2010 06:10:44 Charles Wilson wrote: > > Also, it's faster to read a list of files from a text file (or > > even .xml) than to untarlzma... > > Sure. I was just (pedantically) pointing out that it is possible to > regenerate a files manifest, at any time, by reading the tarball... Ack. > > 0) the files associated with a specific tarball. Probably "keyed" > > according to the canonical package name, rather than the tarball > > name. > > ...so, yes, I omitted this because it isn't technically necessary to > satisfy a "minimum requirement" criterion, but I do plan to do this, > (and indeed, already have in a rudimentary fashion), anyway. Ack. > > A couple of levels of "meta group"? > > I was planning to accommodate this, (expanding on your example), with > a > <source tarname="gettext-&-mingw32-src.tar.gz" /> > > element, embedded within the <package name="gettext"> container > element, at the same level as the individual <component> elements > defining each of those component packages. Apart from "&" being a > really bad choice for "identically matching version", (I'm thinking > of using "%" as, perhaps, a more suitable alternative -- any other > suggestions?), is there any reason why that will not suffice? Yep, that looks good to me. And '%' is probably fine; at least it won't trip up most XML parsers (and since tinyxml has no support for encodings, we don't need to worry about wacky NLS versions of '%'; even so, many common encodings keep '%' at 0x25 anyway...) > I think my idea can work both ways: look for the <source> element > embedded within the <component> element itself, and it becomes a "go > there for my source" reference; if it isn't there, look for it in > the containing <package> element, and it becomes a "here is the > source for all my sibling <component> packages" reference. Yes, this is okay. That makes mingw-get's job somewhat harder (if this, then, else if that, ...) but computers are good at that sort of thing. > Easily handled with a <licence tarname="foo-&-mingw32-lic.tar.gz" /> > element, analogous to the <source> reference. Ack. > > c) metapackage: "msys basic installation" would include all the > Yes. This is already an identified requirement on my to-do list; > a "virtual" package class, associated with no physical package file > or download URI, which exists exclusively as a container for a > meta-package collection, identified by <requires> specs. Naturally, > the installation manifest would need to record this, in whatever > format we eventually adopt for installed dependency tracking. Really? I would think you'd simply want to track the packages exclusively individualluy, even if you installed a big set of packages all at once as part of a meta package. That's the way most linux package managers appear to work. Is it important to be able to 'uninstall' (all of the individual components of) virtual package all at once? > (c) is actually something I would like to get to fairly soon; it may > not be too difficult to implement, at least in a rudimentary manner. It's your decision, obviously. Easy is good... -- Chuck |
From: Charles W. <cwi...@us...> - 2010-02-17 23:20:04
|
On Wed, 17 Feb 2010 15:25 +0000, "Keith Marshall" wrote: > On Friday 12 February 2010 06:10:44 Charles Wilson wrote: > > Also, it's faster to read a list of files from a text file (or > > even .xml) than to untarlzma... > > Sure. I was just (pedantically) pointing out that it is possible to > regenerate a files manifest, at any time, by reading the tarball... Ack. > > 0) the files associated with a specific tarball. Probably "keyed" > > according to the canonical package name, rather than the tarball > > name. > > ...so, yes, I omitted this because it isn't technically necessary to > satisfy a "minimum requirement" criterion, but I do plan to do this, > (and indeed, already have in a rudimentary fashion), anyway. Ack. > > A couple of levels of "meta group"? > > I was planning to accommodate this, (expanding on your example), with > a > <source tarname="gettext-&-mingw32-src.tar.gz" /> > > element, embedded within the <package name="gettext"> container > element, at the same level as the individual <component> elements > defining each of those component packages. Apart from "&" being a > really bad choice for "identically matching version", (I'm thinking > of using "%" as, perhaps, a more suitable alternative -- any other > suggestions?), is there any reason why that will not suffice? Yep, that looks good to me. And '%' is probably fine; at least it won't trip up most XML parsers (and since tinyxml has no support for encodings, we don't need to worry about wacky NLS versions of '%'; even so, many common encodings keep '%' at 0x25 anyway...) > I think my idea can work both ways: look for the <source> element > embedded within the <component> element itself, and it becomes a "go > there for my source" reference; if it isn't there, look for it in > the containing <package> element, and it becomes a "here is the > source for all my sibling <component> packages" reference. Yes, this is okay. That makes mingw-get's job somewhat harder (if this, then, else if that, ...) but computers are good at that sort of thing. > Easily handled with a <licence tarname="foo-&-mingw32-lic.tar.gz" /> > element, analogous to the <source> reference. Ack. > > c) metapackage: "msys basic installation" would include all the > Yes. This is already an identified requirement on my to-do list; > a "virtual" package class, associated with no physical package file > or download URI, which exists exclusively as a container for a > meta-package collection, identified by <requires> specs. Naturally, > the installation manifest would need to record this, in whatever > format we eventually adopt for installed dependency tracking. Really? I would think you'd simply want to track the packages exclusively individualluy, even if you installed a big set of packages all at once as part of a meta package. That's the way most linux package managers appear to work. Is it important to be able to 'uninstall' (all of the individual components of) virtual package all at once? > (c) is actually something I would like to get to fairly soon; it may > not be too difficult to implement, at least in a rudimentary manner. It's your decision, obviously. Easy is good... -- Chuck |
From: Keith M. <kei...@us...> - 2010-02-18 18:00:13
|
On Wednesday 17 February 2010 23:19:56 Charles Wilson wrote: > > I think my idea can work both ways: look for the <source> > > element embedded within the <component> element itself, and it > > becomes a "go there for my source" reference; if it isn't there, > > look for it in the containing <package> element, and it becomes > > a "here is the source for all my sibling <component> packages" > > reference. > > Yes, this is okay. That makes mingw-get's job somewhat harder (if > this, then, else if that, ...) but computers are good at that sort > of thing. Exactly. The dependency resolver already does something similar, except in that case it is: look for <requires> elements within the container <package>, and if any are found, recursively resolve them, AND THEN do likewise within the container <component>, AND FINALLY do likewise within the <release> itself. > > > c) metapackage: "msys basic installation" would include all > > > the > > > > Yes. This is already an identified requirement on my to-do > > list; a "virtual" package class, associated with no physical > > package file or download URI, which exists exclusively as a > > container for a meta-package collection, identified by > > <requires> specs. Naturally, the installation manifest would > > need to record this, in whatever format we eventually adopt for > > installed dependency tracking. > > Really? I would think you'd simply want to track the packages > exclusively individualluy, even if you installed a big set of > packages all at once as part of a meta package. That's the way > most linux package managers appear to work. Well, on my Ubuntu box, apt *does* appear to keep track of installed meta-packages, (at least when viewed through synaptic). > Is it important to be able to 'uninstall' (all of the individual > components of) virtual package all at once? I wasn't thinking so much of making an uninstall of a meta-package also remove its prerequisites at the same time -- it's probably a bad idea to automatically uninstall prerequisites when a dependant is uninstalled. (However, IIRC, apt offers an option to force that; we could do likewise, but I don't think it should be the default). I was more thinking that the meta-package would be recorded as being installed, and on which prerequisites it depends. Then, removal of the meta-package could be a prerequisite for safe removal of those packages on which it depends. -- Regards, Keith. |
From: Charles W. <cwi...@us...> - 2010-02-19 00:09:08
|
Keith Marshall wrote: > On Wednesday 17 February 2010 23:19:56 Charles Wilson wrote: >> That makes mingw-get's job somewhat harder (if >> this, then, else if that, ...) but computers are good at that sort >> of thing. > > Exactly. The dependency resolver already does something similar, > except in that case it is: look for <requires> elements within the > container <package>, and if any are found, recursively resolve them, > AND THEN do likewise within the container <component>, AND FINALLY > do likewise within the <release> itself. This carries an implicit priority. If there is conflicting information (which is a bug in the .xml), then the first item found is assumed to be the "truth" and later ones are ignored. Right? >> Really? I would think you'd simply want to track the packages >> exclusively individualluy, even if you installed a big set of >> packages all at once as part of a meta package. That's the way >> most linux package managers appear to work. > > Well, on my Ubuntu box, apt *does* appear to keep track of installed > meta-packages, (at least when viewed through synaptic). Huh. It's been a long time since I used synaptic itself. > I wasn't thinking so much of making an uninstall of a meta-package > also remove its prerequisites at the same time -- it's probably a > bad idea to automatically uninstall prerequisites when a dependant > is uninstalled. Err...yeah. Pls don't do things behind my back <g>. I might have built an untracked package, installed into /mingw/local or something, that uses one of those "no longer needed" prereqs... > (However, IIRC, apt offers an option to force that; > we could do likewise, but I don't think it should be the default). Ack. > I was more thinking that the meta-package would be recorded as being > installed, and on which prerequisites it depends. Then, removal of > the meta-package could be a prerequisite for safe removal of those > packages on which it depends. So, installing or updating a meta-package would trigger installing or updating all of the indicated components, but uninstalling it would simply remove the reference in the installed.db? That would probably work. -- Chuck |
From: Keith M. <kei...@us...> - 2010-02-19 21:04:40
|
On Friday 19 February 2010 00:08:46 Charles Wilson wrote: > Keith Marshall wrote: > > On Wednesday 17 February 2010 23:19:56 Charles Wilson wrote: > >> That makes mingw-get's job somewhat harder (if > >> this, then, else if that, ...) but computers are good at that > >> sort of thing. > > > > Exactly. The dependency resolver already does something > > similar, except in that case it is: look for <requires> elements > > within the container <package>, and if any are found, > > recursively resolve them, AND THEN do likewise within the > > container <component>, AND FINALLY do likewise within the > > <release> itself. > > This carries an implicit priority. If there is conflicting > information (which is a bug in the .xml), then the first item > found is assumed to be the "truth" and later ones are ignored. > Right? Sort of. I start out with an empty dependency queue. Then, for each package/component to be installed, I read its <requires> specs [*], and for each, I check back through the queue looking for a prior entry with exactly matching specs for all of its name, subsystem, and component class, (note to self: it also needs to match component ABI version, if any). If no such matching entry exists, then I add one; otherwise, I check the minimum and maximum version requirements of the existing entry vs. the current request, and reset the prior entry to match the intersecting version sub-range of the two. (If there is no intersecting sub-range, then we have a conflict; I have not yet given much thought to what to do, in this event). In either case, if the queue was modified, I then look up the "best fit" match for the dependency, in the distribution manifest, and recursively evaluate *its* <requires> specs. [*] For each package/component, I also assume an unwritten <requires> spec -- one for the package/component itself! It is a prerequisite of the action we are seeking to perform. > [...] > > > I wasn't thinking so much of making an uninstall of a > > meta-package also remove its prerequisites at the same time -- > > it's probably a bad idea to automatically uninstall > > prerequisites when a dependant is uninstalled. > > Err...yeah. Pls don't do things behind my back <g>. I might have > built an untracked package, installed into /mingw/local or > something, that uses one of those "no longer needed" prereqs... My thinking, exactly! > [...] > > > I was more thinking that the meta-package would be recorded as > > being installed, and on which prerequisites it depends. Then, > > removal of the meta-package could be a prerequisite for safe > > removal of those packages on which it depends. > > So, installing or updating a meta-package would trigger installing > or updating all of the indicated components, but uninstalling it > would simply remove the reference in the installed.db? Yes. > That would probably work. I think so. -- Regards, Keith. |