From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-19 20:56:24
|
On Nov 19, 2012, at 2:54 PM, Roy Stogner <roy...@ic...> wrote: > > On Mon, 19 Nov 2012, Kirk, Benjamin (JSC-EG311) wrote: > >> Paul and I have been back and forth on installing the example >> binaries - if you think it is important I could do it… > > I'd vote "no" but wouldn't be bothered either way. My feeling is installing them with a couple line makefile would be better - $ cp -r /libmesh/example ./myexample make & play encourages more learning anyway. -Ben |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-19 21:42:11
|
On Nov 19, 2012, at 11:19 AM, Roy Stogner <roy...@ic...> wrote: > > On Mon, 19 Nov 2012, Kirk, Benjamin (JSC-EG311) wrote: > >> Yup, working on it now. for obvious reasons I don't want to add >> them to the repository, so I'm looking for a source forge/github >> file upload space we can use… > > Why not just add them to the same SourceForge file releases page as > the real releases? It can be automated with sftp from whatever > machine we trust to run the cron job. Do we have a tight disk quota > there? I would say we create a /home/pfs/project/libmesh/libmesh/libmesh-daily on frs.sourceforge.net and upload the tarballs created by 'make distcheck' iff distcheck successfully completes. Problem is I can only connect to that machine from ICES, as NASA blocks outbound ssh unless there is an exception in place. so, what we need then is the ICES buildbot (or whatever trusted machine) do a 'make distcheck && ./upload_daily_snapshot' or something. -Ben |
From: Derek G. <fri...@gm...> - 2012-11-19 21:54:24
|
Why is the snapshot useful? Do you really want to encourage people to use it? People will modify their snapshot and have no real way to keep up to date... leading to all sorts of problems down the road... Derek Sent from my iPhone On Nov 19, 2012, at 2:42 PM, "Kirk, Benjamin (JSC-EG311)" <ben...@na...> wrote: > On Nov 19, 2012, at 11:19 AM, Roy Stogner <roy...@ic...> wrote: > >> >> On Mon, 19 Nov 2012, Kirk, Benjamin (JSC-EG311) wrote: >> >>> Yup, working on it now. for obvious reasons I don't want to add >>> them to the repository, so I'm looking for a source forge/github >>> file upload space we can use… >> >> Why not just add them to the same SourceForge file releases page as >> the real releases? It can be automated with sftp from whatever >> machine we trust to run the cron job. Do we have a tight disk quota >> there? > > I would say we create a > > /home/pfs/project/libmesh/libmesh/libmesh-daily > > on frs.sourceforge.net > > and upload the tarballs created by 'make distcheck' iff distcheck successfully completes. > > Problem is I can only connect to that machine from ICES, as NASA blocks outbound ssh unless there is an exception in place. > > so, what we need then is the ICES buildbot (or whatever trusted machine) do a 'make distcheck && ./upload_daily_snapshot' or something. > > -Ben > > > > > > > > ------------------------------------------------------------------------------ > Monitor your physical, virtual and cloud infrastructure from a single > web console. Get in-depth insight into apps, servers, databases, vmware, > SAP, cloud infrastructure, etc. Download 30-day Free Trial. > Pricing starts from $795 for 25 servers or applications! > http://p.sf.net/sfu/zoho_dev2dev_nov > _______________________________________________ > Libmesh-devel mailing list > Lib...@li... > https://lists.sourceforge.net/lists/listinfo/libmesh-devel |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-19 22:01:34
|
On Nov 19, 2012, at 3:54 PM, Derek Gaston <fri...@gm...> wrote: > Why is the snapshot useful? Do you really want to encourage people to > use it? People will modify their snapshot and have no real way to > keep up to date... leading to all sorts of problems down the road… Among other things, it is a fallback for machines where svn is not available or inconvenient. A problem which will be exacerbated by a move to git. -Ben |
From: Roy S. <roy...@ic...> - 2012-11-19 22:12:09
|
On Mon, 19 Nov 2012, Kirk, Benjamin (JSC-EG311) wrote: > I would say we create a > > /home/pfs/project/libmesh/libmesh/libmesh-daily > > on frs.sourceforge.net > > and upload the tarballs created by 'make distcheck' iff distcheck successfully completes. Sounds good. > Problem is I can only connect to that machine from ICES, as NASA blocks outbound ssh unless there is an exception in place. > > so, what we need then is the ICES buildbot (or whatever trusted machine) do a 'make distcheck && ./upload_daily_snapshot' or something. I'm a little paranoid about storing unencrypted passwords on disk... Looks like we can store ssh keys on SourceForge, though, so I'd be happy doing daily connections from any system where I already had an active ssh-agent. "'screen' on Roy's home desktop" sounds less responsible than "buildbot", but it's got a decent uptime, and the current months-long-job running on it is nothing more than "using mencoder --ridiculously-expensive to recompress a bunch of BluRays". On a barely related topic: What's with the need for specific LIBMESH_ENABLE_FPARSER testing in example Makefiles? Shouldn't that be part of the general LIBMESH_INCLUDE construction? --- Roy |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-19 22:22:27
|
On Nov 19, 2012, at 4:12 PM, Roy Stogner <roy...@ic...> wrote: > On a barely related topic: > > What's with the need for specific LIBMESH_ENABLE_FPARSER testing in > example Makefiles? Shouldn't that be part of the general > LIBMESH_INCLUDE construction? Before doing make install, fparser's header files are buried in contrib/fparser. After install we put fparser.hh in $prefix/install/libmesh/fparser.hh In retrospect, now that we symlink everything into include/libmesh it would be better to put fparser.hh there as well. I'll fix that, probably tomorrow. -Ben |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-19 22:30:40
|
On Nov 19, 2012, at 4:22 PM, "Kirk, Benjamin (JSC-EG311)" <ben...@na...> wrote: > On Nov 19, 2012, at 4:12 PM, Roy Stogner <roy...@ic...> wrote: > >> On a barely related topic: >> >> What's with the need for specific LIBMESH_ENABLE_FPARSER testing in >> example Makefiles? Shouldn't that be part of the general >> LIBMESH_INCLUDE construction? > > Before doing make install, fparser's header files are buried in contrib/fparser. After install we put fparser.hh in $prefix/install/libmesh/fparser.hh > > In retrospect, now that we symlink everything into include/libmesh it would be better to put fparser.hh there as well. I'll fix that, probably tomorrow. But to expound, you'll notice not all the contrib/... headers get installed. Almost none of them are needed by a user application, but rather we need them to build the library. fparser is a special case, and I think I add exodus/nemesis too for the MOOSE guys, but installing the laspack header files under 'include/libmesh/...' seemed a step too far. Likewise with metis, parmetis, etc… -Ben |
From: John P. <jwp...@gm...> - 2012-11-20 21:06:45
|
On Mon, Nov 19, 2012 at 1:56 PM, Kirk, Benjamin (JSC-EG311) <ben...@na...> wrote: > My feeling is installing them with a couple line makefile would be better - > > $ cp -r /libmesh/example ./myexample > > make & play > > encourages more learning anyway. Yeah, I agree we don't really need the binaries but the installed examples are pretty useless if we don't also install a Makefile with them... -- John |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-20 21:23:20
|
On Nov 20, 2012, at 3:06 PM, John Peterson <jwp...@gm...> wrote: > On Mon, Nov 19, 2012 at 1:56 PM, Kirk, Benjamin (JSC-EG311) > <ben...@na...> wrote: >> My feeling is installing them with a couple line makefile would be better - >> >> $ cp -r /libmesh/example ./myexample >> >> make & play >> >> encourages more learning anyway. > > Yeah, I agree we don't really need the binaries but the installed > examples are pretty useless if we don't also install a Makefile with > them… Indeed. All the Makefile.old's are there, I will use these as the basis of an installed Makefile that uses libmesh-config to get the relevant configure bits. I don't want to encourage new users to include Make.common as a way to get at the configuration. -Ben |
From: John P. <jwp...@gm...> - 2012-11-20 23:23:34
|
Ben, Attached is a (possible) fix for your previous fix to enable legacy include paths (which still didn't work for me). I'm not sure this patch is totally safe because it refers to $(top_srcdir), i.e. where the SVN checkout of libmesh resides. Do we have to assume the user can delete their $(top_srcdir) at any time and continue using the installed libmesh just fine? If so, I'm not sure what the best way to support legacy include paths would be. -- John |
From: John P. <jwp...@gm...> - 2012-11-20 23:30:40
|
On Tue, Nov 20, 2012 at 4:23 PM, John Peterson <jwp...@gm...> wrote: > Attached is a (possible) fix for your previous fix to enable legacy > include paths (which still didn't work for me). I guess I could elaborate. The problem seems to be that ${INSTALL_DIR}/lib/x86_64-apple-darwin10.8.0_opt/pkgconfig/Make.common.opt does not know what $(libmesh_dir) is, so with your previous patch I was ending up with include paths like: -I/include/libmesh which is obviously wrong.... on the other hand it does know what $(top_srcdir) is. -- John |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-21 00:40:13
|
On Nov 20, 2012, at 5:23 PM, John Peterson <jwp...@gm...> wrote: > Ben, > > Attached is a (possible) fix for your previous fix to enable legacy > include paths (which still didn't work for me). > > I'm not sure this patch is totally safe because it refers to > $(top_srcdir), i.e. where the SVN checkout of libmesh resides. Well, at the top of Make.common we use top_srcdir if it is defined (i.e. within the automake environment), but then hackishly define it to @prefix@ when doing the install. So the Make.comon that exists in the install should be referring to the installed path, and then your patch is safe even when the original source tree is deleted. > Do we have to assume the user can delete their $(top_srcdir) at any > time and continue using the installed libmesh just fine? > > If so, I'm not sure what the best way to support legacy include paths would be. Yes, that is the expected behavior, and your patch seems to do just fine. Note it will not work for someone who includes Make.common inside an automake project (thus reactivating top_srcdir), but that is ill-advised. -Ben |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-21 23:39:39
|
On Nov 20, 2012, at 2:23 PM, Benjamin S. Kirk <ben...@na...> wrote: > > On Nov 20, 2012, at 3:06 PM, John Peterson <jwp...@gm...> wrote: > >> On Mon, Nov 19, 2012 at 1:56 PM, Kirk, Benjamin (JSC-EG311) >> <ben...@na...> wrote: >>> My feeling is installing them with a couple line makefile would be better - >>> >>> $ cp -r /libmesh/example ./myexample >>> >>> make & play >>> >>> encourages more learning anyway. >> >> Yeah, I agree we don't really need the binaries but the installed >> examples are pretty useless if we don't also install a Makefile with >> them… > > Indeed. All the Makefile.old's are there, I will use these as the basis of an installed Makefile that uses libmesh-config to get the relevant configure bits. I don't want to encourage new users to include Make.common as a way to get at the configuration. With r6433 we install a common makefile and the run script with each example, so the following works: assume I install into /tmp/foo: $ make && make install $ export LIBMESH_DIR=/tmp/foo $ cd $LIBMESH_DIR/examples $ for dir in */* ; do cd $dir ; make ; sh ./run.sh ; cd - ; done This installs $srcdir/contrib/utils/Makefile and the case-specific run.sh in the example directory. The idea is the user can copy a given example and start hacking. I'd like to rework the makefile not to include Make.common, and that means moving the compile rules into it directly. I'd also like to get some feedback on this approach since these self-contained examples should be the way people get introduced to the library, and I want to set good 'best practices' with the Makefile! -Ben |
From: Roy S. <roy...@ic...> - 2012-11-21 23:58:28
|
On Wed, 21 Nov 2012, Kirk, Benjamin (JSC-EG311) wrote: > I'd like to rework the makefile not to include Make.common, and that > means moving the compile rules into it directly. Why? In my experience, when you hand users ten copies of the same original Makefile (or when you so much as make it easier for them to copy than to symlink or svn:external a master copy...), you end up with 4 outdated copies, 4 copies with new (but mutually incompatible) design changes, and 2 copies outright broken. > I'd also like to get some feedback on this approach since these > self-contained examples should be the way people get introduced to > the library, and I want to set good 'best practices' with the > Makefile! Best practices are that you don't copy-and-paste code, no? Having "make install" do the copy-and-paste for you doesn't get out of that. --- Roy |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-22 00:06:49
|
On Nov 21, 2012, at 4:58 PM, Roy Stogner <roy...@ic...> wrote: >> >> I'd also like to get some feedback on this approach since these >> self-contained examples should be the way people get introduced to >> the library, and I want to set good 'best practices' with the >> Makefile! > > Best practices are that you don't copy-and-paste code, no? Having > "make install" do the copy-and-paste for you doesn't get out of that. Mostly because 'include our makefile' is frowned upon… We have time to iterate this though, not like I want to ship 0.9.0 tomorrow! -Ben |
From: Roy S. <roy...@ic...> - 2012-11-22 00:16:59
|
On Wed, 21 Nov 2012, Kirk, Benjamin (JSC-EG311) wrote: > Mostly because 'include our makefile' is frowned upon… Right - but it wouldn't be "our makefile", with a billion internally-used identifiers and things that might break other build systems; it would be "our provided make rules", specifically designed to be the core of a build system. > We have time to iterate this though, not like I want to ship 0.9.0 > tomorrow! Certainly. If we're brainstorming wishlist-y, I'd actually like to provide multiple examples of *build systems* too - one a two-part Makefile with a stripped down "Make.common" that does a better job of separating PETSc/SLEPc/libMesh/etc internals from deliberately provided (and LIBMESH_ prefixed) variables, one a barebones Makefile.am setup; even an scons or more exotic option if people wanted to contribute those. --- Roy |
From: Derek G. <fri...@gm...> - 2012-11-22 00:39:32
|
> We have time to iterate this though, not like I want to ship 0.9.0 > tomorrow! > Maybe we should do a poll to see how many of our users are using released versions of libMesh. I would say it is nearly zero. I know of over a hundred who are exclusively using the SVN version of libMesh: our users. With that in mind, we should take steps all along the way that allow the SVN version of libMesh to be used. Stating that "they can download nightlies" or "we have time until 0.9.0" doesn't do justice to the reality that many people are using the library right out of the repo. There seems to be a trend here lately where libMesh devs are continuing to push for users to not use the repo version of libMesh. This is exactly the wrong direction... and it goes against what the rest of the scientific library community is moving toward. With MOOSE we don't have releases _at all_. All of our users use the "trunk" version of our code. Our development process is called Continuous Integration (CI: http://en.wikipedia.org/wiki/Continuous_integration ). It has been working beautifully for over four years. In this development mode, every change to MOOSE is immediately integrated with all of our user's codes. No one is ever "out of sync" and there is no "integration" or "upgrading" ever to be done. This is all powered through a robust and thorough testing system that ensures that every application (of which there are now 20+) built using MOOSE continues to function through every change to MOOSE. We recently (like two weeks ago) had an external software quality audit where a team of people came in and examined every part of our software development cycle. Not only did we pass with flying colors and were granted NQA1 status (which means MOOSE can be used to create software that is used is the design of nuclear reactors) but the software quality team was blown away by the continuous development path we use. If you want to do some reading on the use of continuous integration (or nearly continuous) in scientific software development you can look at some of the papers by Roscoe Bartlett ( http://www.ornl.gov/~8vt/publications.html ). For instance: http://www.ornl.gov/~8vt/CSE_SoftwareIntegration_Strategies.pdf http://www.ornl.gov/~8vt/TribitsLifecycleModel_eScience_2012.pdf I'm not advocating everything he says... but the core ideas of "integrate early and often" are important in scientific software development... especially when that development entails intimately tying your code to a software framework. With these ideas in mind we should be striving to make the repo version of libMesh as accessible and simple to deal with as possible. We should encourage our users to use it. We should encourage our users to update _daily_ and we should work to make sure that what we do in the repo is not disruptive to that. I am straight up frustrated with the state of the repo at the moment. Derek |
From: Roy S. <roy...@ic...> - 2012-11-22 01:36:09
|
On Wed, 21 Nov 2012, Derek Gaston wrote: > Stating that "they can download nightlies" or "we have time until > 0.9.0" doesn't do justice to the reality that many people are using > the library right out of the repo. "They can download nightlies" isn't an attempt to make things harder for users, it's an attempt to assist users, motivated partly by your insistance that we support users without automake, and expanding that to add support for users without subversion (or git or whatever else we switch to) too. "We have time until 0.9.0" is the attitude we've *always* had; other than constantly branching the code, making specific "release" versions is the only way to avoid either paralyzing development (what would happen if every suggested change had to be well-planned enough to practically guarantee indefinite future backwards compatibility and support) or regularly yanking the rug out from under our users (what would happen if every implemented change was subject to reversion if and when a better alternative was found). The comments about constant integration are well-received, though. We've always had an "anyone can commit to trunk" sort of philosophy, but "anyone can commit to branches/testing; revisions which pass tests get synched to trunk" would be more responsible. --- Roy |
From: Kirk, B. (JSC-EG311) <ben...@na...> - 2012-11-22 01:26:08
|
On Nov 21, 2012, at 5:39 PM, "Derek Gaston" <fri...@gm...> wrote: > I am straight up frustrated with the state of the repo at the moment. > > Derek Understood, but CI only works when everyone can see results of tests. Libmesh.automake has been passing its tests for months, and Paul and I use it in production. If you want to point users to our trunk we need to see your test results. Period. Or quit bitching and svn switch them to something blessed - e.g. ^/tags/whatever -Ben |
From: Roy S. <roy...@ic...> - 2012-11-22 01:43:37
|
On Wed, 21 Nov 2012, Kirk, Benjamin (JSC-EG311) wrote: > If you want to point users to our trunk we need to see your test > results. Period. This is a good point. I'd feel much much happier about my "libmesh/testing gets synched to trunk when it passes tests" plan if those tests included all the various rigorous application tests we have rather than just the relatively rigor-free internal logic tests that get hit by our examples. Presuming that most of those applications aren't going to be available for public release in the immediate future, we'd need to have some way for the central buildbot (or trac, whatever) server to see external results, and moreover to get *synchronized* results rather than results testing whatever revision happens to be the latest available when the tests are run. Even if GRINS-buildbot reports success with r7001 and FINS-buildbot reports success with r7002 and MOOSE-trac reports success with r7004, that doesn't mean that any of those releases are guaranteed free of failures with all three test suites. > quit bitching This wasn't called for. --- Roy |
From: Paul T. B. <ptb...@gm...> - 2012-11-22 02:08:34
|
On Wed, Nov 21, 2012 at 5:43 PM, Roy Stogner <roy...@ic...>wrote: > > > quit bitching > > This wasn't called for. I'll support Ben here. This branch has been around for months. There's been lots of discussion of "hey, we're going to merge this into trunk soon". There's absolutely no way this was a surprise. There's been lots of time to test these things. There've been multiple offers from Ben and myself to help facilitate this. Ben has been extraordinarily responsive in helping people with the transition (which will have inevitable hiccups) and trying to accommodate requests for the behavior of the build system. And instead of trying to provide examples that are breaking that could be added into a regression suite, the responses that come back are "I am straight up frustrated with the state of the repo at the moment" which is not helpful for anyone. Paul |
From: Paul T. B. <ptb...@gm...> - 2012-11-22 02:13:27
|
On Wed, Nov 21, 2012 at 4:16 PM, Roy Stogner <roy...@ic...>wrote: > > Certainly. If we're brainstorming wishlist-y, I'd actually like to > provide multiple examples of *build systems* too - one a two-part > Makefile with a stripped down "Make.common" that does a better job of > separating PETSc/SLEPc/libMesh/etc internals from deliberately > provided (and LIBMESH_ prefixed) variables, one a barebones > Makefile.am setup; even an scons or more exotic option if people > wanted to contribute those. > I've been thinking about something like this, actually. Should we just select some subset of existing examples and add the build system, or do we want to setup/install separate special examples to only focus on the build system aspect? Either way, I can add/modify-existing-example-to an autotools system to serve as an example for building an application. I think it's prudent to do this ASAP given the recent merge. So, let me know your opinions and I'll drop an example in. Best, Paul |
From: Derek G. <fri...@gm...> - 2012-11-22 02:21:55
|
On Wed, Nov 21, 2012 at 7:25 PM, Kirk, Benjamin (JSC-EG311) < ben...@na...> wrote: > Understood, but CI only works when everyone can see results of tests. > Libmesh.automake has been passing its tests for months, and Paul and I use > it in production. > You and Paul are not average users. > If you want to point users to our trunk we need to see your test results. > Period. > Understood - this is the reason I worked quite a bit a while ago on getting a public test service up and running for libMesh. I didn't find a great solution at that time... and then got pulled off onto other things. But we need a centralized service that is coordinating builds and receiving back results. Once the decision is made about where libMesh is going to reside long term, we should look into this again. > Or quit bitching and svn switch them to something blessed - e.g. > ^/tags/whatever > Firstly, I do apologize for "bitching". I am really not trying to come off that way. I just truly do not think the current changes were for the better... and I don't know what else to do about than to point out reasons why. If I could, I would just go in there and revert the automake merge instead of "bitching". As it is, I'm trying to provide information about the rest of the way the world interacts and uses libMesh and other scientific software so that we can constructively come to a system that works better than the automake system. Let me explain a little more: I love libMesh. I think it's an awesome triumph of scientific framework software. I am constantly in awe at all the thought put into the way the system works. All of that thought went into creating a library that presents an incredibly useful interface to it's users and enables scientists and engineers to create complex parallel applications. Because of this, I think the build system should be just as well thought out... and just as easy to interact with. In the same way that you don't have to be a parallel programming specialist or a computer scientist to use libMesh... I don't think you should need to be a specialist to _build_ libMesh. I don't want this great piece of software to be overlooked simply because it is difficult to interact with the build system. As for our users. They do use a vetted version of _SVN_ libMesh. They are not pulling from your repo, so they are insulated (for the moment) from these changes. However, they _will_ use a version from SVN very soon (as soon as either we make a change in libMesh that requires updating... or you guys add some cool features that warrants an update). They will never use a release version of libMesh. Sorry for stirring up shit the night before Thanksgiving! Let me just say: Happy Thanksgiving Everyone! I really appreciate and am thankful for the opportunity to work (and debate!) with all of you! ;-) Derek |
From: Paul T. B. <ptb...@gm...> - 2012-11-22 02:44:23
|
On Wed, Nov 21, 2012 at 6:21 PM, Derek Gaston <fri...@gm...> wrote: > Let me just say: Happy Thanksgiving Everyone! I really appreciate and am > thankful for the opportunity to work (and debate!) with all of you! ;-) > Couldn't have put it better myself. +1. Happy turkey day all. Best, Paul |
From: Roy S. <roy...@ic...> - 2012-11-22 02:26:21
|
On Wed, 21 Nov 2012, Paul T. Bauman wrote: > Should we just select some subset of existing examples and add the > build system My initial thought would be to create a new examples/builds/ directory; in each subdirectory use the same plain, dumb code (maybe my heatsystem.C code with an even-more-slashed-down version of the physics-independent bits), but use a different build system in each. Leave the rest of the examples alone for now, then switch them all to whichever builds/builds_ex? framework looks best next year. (my guess would be automake with some factored-out bits, but I'm prepared to be surprised) But even then, leave the other builds/ entries unchanged, so that people using cmake/scons/plain-make/whatever have easy examples to work from and/or places to add new examples. Automake sucks less than everything else I've tried, but not so much less that we don't want to make life as easy as possible for people who disagree. --- Roy |