From: Patrick S. <sch...@in...> - 2008-04-28 12:41:52
|
Hi, I know that we (me as the Debian maintainer of mantis and you as the upstream author) have different preferences and opinions on how to manage upgrades. Well, on the other hand, I understand your statement about this and so I could accept some manual intervention when upgrading the database, even by doing it through the webinterface. Its worth discussing if there would be a way to seperate not so logic-intensive upgrade scripts out of the application into seperate SQL files but the benefit is low so the best option in the long run is probably a php-based upgrade script, which can run unattended. (I don't really like that, but its possible and would be the best compromise between your needs and my needs as the Debian maintainer.) Ok, now I think that this part is clear, so I can move on to the real topic of this mail. While their might be an interest conflict while upgrading I really don't see a problem for the database creation. It should be possible with pure SQL files. Possibly this could be handled by the mentioned script, but SQL is a lot more transparent, so I would prefer this. So my question is: Is there a way for you to provide database creation scripts with every mantis release? I could understand that the maintainance efforts are to high to care for the php-part AND an sql part so I suggest to write something as a part of the PHP install scripts that can be run from the console and would output the sql statements that would normally be run to install the db. So as a package maintainer all that I would do to create a create_database script is just run this script (probably with the db type as a parameter if you make differences there). Would this be possible? Thanks in advance, best Regards, Patrick |
From: John R. <jr...@le...> - 2008-04-28 14:18:20
|
Patrick Schoenfeld wrote: > Hi, > > I know that we (me as the Debian maintainer of mantis and you as the > upstream author) have different preferences and opinions on how to > manage upgrades. Well, on the other hand, I understand your statement > about this and so I could accept some manual intervention when upgrading > the database, even by doing it through the webinterface. Its worth > discussing if there would be a way to seperate not so logic-intensive > upgrade scripts out of the application into seperate SQL files but the > benefit is low so the best option in the long run is probably a > php-based upgrade script, which can run unattended. (I don't really like > that, but its possible and would be the best compromise between your > needs and my needs as the Debian maintainer.) > Ok, now I think that this part is clear, so I can move on to the real > topic of this mail. While their might be an interest conflict while upgrading > I really don't see a problem for the database creation. > It should be possible with pure SQL files. Possibly this could be > handled by the mentioned script, but SQL is a lot more transparent, so I > would prefer this. So my question is: Is there a way for you to provide > database creation scripts with every mantis release? I could understand > that the maintainance efforts are to high to care for the php-part AND > an sql part so I suggest to write something as a part of the PHP install > scripts that can be run from the console and would output the sql > statements that would normally be run to install the db. So as a package > maintainer all that I would do to create a create_database script is > just run this script (probably with the db type as a parameter if you > make differences there). > > Would this be possible? > > Thanks in advance, > best Regards, > > Patrick Please don't interpret this as an attack of any sort. As a major proponent of the Debian/Ubuntu distros, I would love to see Mantis reach those users more easily. This is just me pointing out all the hurdles I see with what you are trying to accomplish. Anyways, the real problem with pure SQL based setup is that We support too many database types to make this feasible. As you may have noticed, we use a shorthand version of SQL to define the incremental database schema, which is interpretted at run-time by AdoDB based upon which database type the user has specified. Even for installing Mantis on the two most popular "free" databases available in Debian (mysql vs postgresql), the generated SQL is likely very different because of hov each dbms defines column types, etc. The end result is that Mantis *can't* easily be simplified like you want it to, because there are too many possibilities. The only thing I can think is to create a special system that uses AdoDB to interpret the shorthand and transform it into raw SQL for each database type, and then save those queries. However, as has been mentioned previously in the mailing list, because Mantis strives for such far-reaching backward compatibility, while still making large changes to the infrastructure of the schema, we need upgrade functions that are written in PHP as well, so that we can smartly process and upgrade old information into the new system of storage. Case in point: category -> category_id upgrade path. So this sort of thing would definitely need to be taken into account when handling upgrades versus new installs via any fedora/debian packages. -- John Reese LeetCode.net |
From: <pa...@qu...> - 2008-04-28 15:40:58
|
This is getting top quoted as it's in some silly web interface atm (at 1280x1024, i'm pretty sure you can make a web based email client that has a box for the body with >10 lines and 30 chars wide).... To me, this all seems to come down to ~2-3 basic problems with the current approach: Over time, we've done or tried doing the following: i) Mantis moved from a native .sql file to a more automated approach. ii) Mantis tried adding support for multiple DB's. iii) Mantis started using adodb. I dont think mantis is alone in bundling a webbased installer - The installers i'm thinking of currently are those that come with two applications: a) moodle (fairly large, I suspect it's in debian?) b) fudforum (maybe not?). I'd be interested what approach debian take with these as they a) support multiple DB backends, b) provide some level of automated routine for updates. It might be we can learn from this approach... One thing I think we lack or could improve on is the way we currently allow people to 'print sql queries' -> We have functionality to allow sql queries to be printed out, atm, only accessible via a web interface after logging in. This functionality should probably be available by some command line means. (i.e. php cmd line script not some wget install.php hack) The second thing is adodb. Adodb seems to have a number of cases where the datadict support (which is what the update stuff uses) is lacking some functionality or is inconsistent. I know i've recently reported some postgres stuff on their forum, and from our wiki page ( http://wiki.mantisbt.org/doku.php/mantisbt:adodb_fixes?s=adodb) someone working with zend/IBM emailed the adodb guy(s)? about DB2. I think the current case is none of the issues reported have been fixed in an adodb release. In my case, the postgres stuff was submitted recently, and i've yet to receive any feedback (<1 week ago). >From a mantis perspective, it's not "a problem" as we can patch the bundled adodb with any bug fixes. However, i'm not sure where that leaves say debian + mantis running on DB2 if they dont want to use the bundled adodb but instead a global package, and adodb haven't responded to bug reports. Paul > Hi John, > > On Mon, Apr 28, 2008 at 10:17:26AM -0400, John Reese wrote: >> Please don't interpret this as an attack of any sort. As a major >> proponent of the Debian/Ubuntu distros, I would love to see Mantis reach >> those users more easily. This is just me pointing out all the hurdles I >> see with what you are trying to accomplish. > > I don't interpret it as a attack. But I interpret the behaviour as > throwing stones in my way. Because I'm doing what I can to near myself > into your direction. I try to understand your problems, but you seem to > offend the simplest solution from your developers point of view, while I > try to > make it simple for the users AND keep things sane for Debian. > >> Anyways, the real problem with pure SQL based setup is that We support >> too many database types to make this feasible. As you may have noticed, > > You don't seem to have read properly what I've written. I did not talk > about a pure SQL based setup. I didn't say: Give me an SQL script that > works on every database you support (well, btw. it seems that the only > properly supported database is mysql if I look at the bugtracker and the > second mail I wrote, so the "we support so many databases" argument is > limited in its extend). What I said is: Give me an upgrade script that I > can use non-interactive from within scripts (e.g. as a PHP script) and > give me a tool to create SQL dumps _for the *initial* *creation*_ of the > database. I understand all the arguments you have for upgrade problems. > So I say: Okay, I'm able to come one step closer to you, but you have to > come a step closer to me, too. Thats what I lack to see, currently. > But the creation is totally different from that. Yes, you are true. The > file formats are different, thats why I said give me a tool that dumps > sql scripts given a specific database. I would be able to commit to even > use a php script for the creation of the database, but I don't see where > the benefit is in that, except that the process of creating the mantis > database becomes a lot more non-transparent then it is now. Thats why > I'm "voting" for "pure" SQL in this step of story. > >> The end result is that Mantis *can't* easily be simplified like you want >> it to, because there are too many possibilities. The only thing I can >> think is to create a special system that uses AdoDB to interpret the >> shorthand and transform it into raw SQL for each database type, and then >> save those queries. > > Thats more or less what I asked for. Well, that does not help me in the > short run, because I'm in pressure to create a current package, but it > would help me in the long run. For now I would just need a > create_database.sql for PostgreSQL and MySQL (because on Debian I will > not support more DBMS) and a fix for the issue with old Debian > installations, > but that last thing is a thing that I have to do. > >> However, as has been mentioned previously in the mailing list, because >> Mantis strives for such far-reaching backward compatibility, while still >> making large changes to the infrastructure of the schema, we need >> upgrade functions that are written in PHP as well, so that we can >> smartly process and upgrade old information into the new system of >> storage. Case in point: category -> category_id upgrade path. So this >> sort of thing would definitely need to be taken into account when >> handling upgrades versus new installs via any fedora/debian packages. > > See above. A transition is _not_ part of a new install, so this argument > is invalid. I agree with you for upgrades. See my above comments. > > Regards, > Patrick > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > mantisbt-dev mailing list > man...@li... > https://lists.sourceforge.net/lists/listinfo/mantisbt-dev > > |
From: Patrick S. <sch...@in...> - 2008-04-28 20:00:46
|
Hi, On Mon, Apr 28, 2008 at 04:40:26PM +0100, pa...@qu... wrote: > I dont think mantis is alone in bundling a webbased installer - The well, there is nothing bad about it. The webbased installer is most likeley the best approach if you are not installing through the package management of your favorite distribution. But if you do then I'd say that you'd want to make use of its advantages. To me one of the main advantages in Debian was, is and should be the good working upgrade path. We have a good package management for this and some complementing tools (like dbconfig-common) that can help to ensure that this stays this way. Thus a webbased approach is clearly suboptimal in this case. > installers i'm thinking of currently are those that come with two > applications: a) moodle (fairly large, I suspect it's in debian?) b) moodle is Debian, yes. Its using wwwconfig-common which is a framework for webbased applications for doing common tasks like creating apache configuration etc. It also does create the database with the tools which are part of wwwconfig-common and it seems it uses a moodle script to populate it. But I'm not exactly sure. Hm, well, wwwconfig-common is not the best tool for handling databases, dbconfig-common is said to be more advanced in that. And really it is able to handle the task its supposed to, but it needs sql files or at least some sort of shell-based script to call. > fudforum (maybe not?). I'd be interested what approach debian take with Not in Debian. > these as they a) support multiple DB backends, b) provide some level of > automated routine for updates. It might be we can learn from this > approach... No, it does not seem to do automated updates. Personally I don't like that approach and its clearly not the way I want to go with mantis, because the current state of the mantis packaging is far more advanced and I don't even think about stepping back. > One thing I think we lack or could improve on is the way we currently > allow people to 'print sql queries' -> We have functionality to allow sql > queries to be printed out, atm, only accessible via a web interface after > logging in. This functionality should probably be available by some > command line means. (i.e. php cmd line script not some wget install.php > hack) Yes, I already said that this would be helpful. Well, lets precise that: It would be good if the webbased tool would _work_ _at_ _all_(because it does produce non-sense as we saw with the recent issues with config_table not properly populated) but a cmd line script would sureley be the creme de la creme. > The second thing is adodb. Adodb seems to have a number of cases where the > datadict support (which is what the update stuff uses) is lacking some > functionality or is inconsistent. I know i've recently reported some Now it comes to me asking how the other people do. Obviously as a distribution we avoid by all means to use a code-copy that comes with mantis because from a security-support pov its unmaintainable. So from this pov the approach to patch adodb and ship it with mantis is clearly unacceptable. Well, AFAICT adodb is used by a lot of other applications as well. Wouldn't these projects suffer from the same problems that you have? Multi-DB support is IMHO nothing very special and as far as I remembered most applications I had to do with managed at least PostgreSQL and MySQL well. mantis does not seem to. And even if the problem is with adodb, if it is really that bad then I ask myself: Why stick with it? Where is the benefit with it, if it is that bad? Is Pear DBI more worse? Aren't their any other solutions? > >From a mantis perspective, it's not "a problem" as we can patch the > bundled adodb with any bug fixes. However, i'm not sure where that leaves > say debian + mantis running on DB2 if they dont want to use the bundled > adodb but instead a global package, and adodb haven't responded to bug > reports. As I said from my perspective as a Debian maintainer this is the worst thing you as upstream can do. Effectiveley it means that the security team has no chance to keep up with fixing _one_ security bug in a suite like libphp-adodb which is multiplied over several packages if we'd follow your approach. It would mean that I (as the maintainer) would need to track security teams work and see how it applies to my packages. I can't do that. Till now I've driven a fairly naive way about this. Because I do use libphp-adodb and mostly it works well. Well, except for PostgreSQL which seems to suffer from your missing fixes that obvious are missing in the Debian package. That is a sort of problem that really needs to be cleared. But I won't change back to approach to ship a adodb copy with the mantis package, as it would result in the problems we had for the last Debian release. E.g. the security team refusing to support mantis in a stable release. Regards, Patrick |
From: Patrick S. <sch...@in...> - 2008-04-28 14:35:24
|
Hi John, On Mon, Apr 28, 2008 at 10:17:26AM -0400, John Reese wrote: > Please don't interpret this as an attack of any sort. As a major > proponent of the Debian/Ubuntu distros, I would love to see Mantis reach > those users more easily. This is just me pointing out all the hurdles I > see with what you are trying to accomplish. I don't interpret it as a attack. But I interpret the behaviour as throwing stones in my way. Because I'm doing what I can to near myself into your direction. I try to understand your problems, but you seem to offend the simplest solution from your developers point of view, while I try to make it simple for the users AND keep things sane for Debian. > Anyways, the real problem with pure SQL based setup is that We support > too many database types to make this feasible. As you may have noticed, You don't seem to have read properly what I've written. I did not talk about a pure SQL based setup. I didn't say: Give me an SQL script that works on every database you support (well, btw. it seems that the only properly supported database is mysql if I look at the bugtracker and the second mail I wrote, so the "we support so many databases" argument is limited in its extend). What I said is: Give me an upgrade script that I can use non-interactive from within scripts (e.g. as a PHP script) and give me a tool to create SQL dumps _for the *initial* *creation*_ of the database. I understand all the arguments you have for upgrade problems. So I say: Okay, I'm able to come one step closer to you, but you have to come a step closer to me, too. Thats what I lack to see, currently. But the creation is totally different from that. Yes, you are true. The file formats are different, thats why I said give me a tool that dumps sql scripts given a specific database. I would be able to commit to even use a php script for the creation of the database, but I don't see where the benefit is in that, except that the process of creating the mantis database becomes a lot more non-transparent then it is now. Thats why I'm "voting" for "pure" SQL in this step of story. > The end result is that Mantis *can't* easily be simplified like you want > it to, because there are too many possibilities. The only thing I can > think is to create a special system that uses AdoDB to interpret the > shorthand and transform it into raw SQL for each database type, and then > save those queries. Thats more or less what I asked for. Well, that does not help me in the short run, because I'm in pressure to create a current package, but it would help me in the long run. For now I would just need a create_database.sql for PostgreSQL and MySQL (because on Debian I will not support more DBMS) and a fix for the issue with old Debian installations, but that last thing is a thing that I have to do. > However, as has been mentioned previously in the mailing list, because > Mantis strives for such far-reaching backward compatibility, while still > making large changes to the infrastructure of the schema, we need > upgrade functions that are written in PHP as well, so that we can > smartly process and upgrade old information into the new system of > storage. Case in point: category -> category_id upgrade path. So this > sort of thing would definitely need to be taken into account when > handling upgrades versus new installs via any fedora/debian packages. See above. A transition is _not_ part of a new install, so this argument is invalid. I agree with you for upgrades. See my above comments. Regards, Patrick |
From: Glenn H. <thr...@lo...> - 2008-04-29 00:00:09
|
On 28-Apr-08, at 8:42 AM, Patrick Schoenfeld wrote: > Hi, > > I know that we (me as the Debian maintainer of mantis and you as the > upstream author) have different preferences and opinions on how to > manage upgrades. Well, on the other hand, I understand your statement > about this and so I could accept some manual intervention when > upgrading > the database, even by doing it through the webinterface. Its worth > discussing if there would be a way to seperate not so logic-intensive > upgrade scripts out of the application into seperate SQL files but the > benefit is low so the best option in the long run is probably a > php-based upgrade script, which can run unattended. (I don't really > like > that, but its possible and would be the best compromise between your > needs and my needs as the Debian maintainer.) > Ok, now I think that this part is clear, so I can move on to the real > topic of this mail. While their might be an interest conflict while > upgrading > I really don't see a problem for the database creation. > It should be possible with pure SQL files. Possibly this could be > handled by the mentioned script, but SQL is a lot more transparent, > so I > would prefer this. So my question is: Is there a way for you to > provide > database creation scripts with every mantis release? I could > understand > that the maintainance efforts are to high to care for the php-part AND > an sql part so I suggest to write something as a part of the PHP > install > scripts that can be run from the console and would output the sql > statements that would normally be run to install the db. So as a > package > maintainer all that I would do to create a create_database script is > just run this script (probably with the db type as a parameter if you > make differences there). > > Would this be possible? Hi Patrick, This is in my work queue for the 1.2.0 release, assuming my house holds up :). My plan is to create a version of the web based installer that will run from the command line using php. The refactoring is about half done. ,,, Glenn -- Glenn Henshaw Logical Outcome Ltd. e: thr...@lo... w: www.logicaloutcome.ca |
From: Patrick S. <sch...@in...> - 2008-04-29 10:33:59
|
On Mon, Apr 28, 2008 at 07:59:55PM -0400, Glenn Henshaw wrote: > This is in my work queue for the 1.2.0 release, assuming my house > holds up :). My plan is to create a version of the web based installer > that will run from the command line using php. The refactoring is > about half done. Sounds good, even though I'd still prefer if those tool would be able to dump SQL commands for the database creation (only). So from my point of view it would be ideal if this tool would understand these modes of operation: - Installation - Dump SQL Commands for manual database creation for a given db-type - Upgrade an existing installation (to a new database scheme) It would work for me without the dump function if the installation has a well-defined interface which I can use to supply it with informations like the database server hostname, database user, database password and the database admin credentials. However I see a danger in that exchange for possible new security issues which leads to the conclusion that this needs to be handled with care. Next to other arguments (transparence for example) this is one of the reasons for the dump function, which would enable me as a maintainer to completely do without any interface between debconf on the Debian side and a script you provide on the other side. All I would do is run the dump sql command when preparing a new release and package sql scripts and only run an upgrade script in upgrade situations (where the data comes from the mantis configuration and is therefore not security relevant). Besides the technical details: A database change as you repeatedly talked about which will change data instead of only structure will happen in 1.2.0 for the first time, right? So for now it would be enough to have an upgrade sql script. Is that right? Or would I need to go with the technical more worse situation for 1.1.1 and let people upgrade themselves? Best Regards, Patrick |