SourceForge has been redesigned. Learn more.

data representation (text files or database)

1 2 3 > >> (Page 1 of 3)
  • Martin Zibricky

    Martin Zibricky - 2008-03-19

    Should we use for internal representation just flat(xml text) files or a type of local database?
    Or use a combination of both? Text files for saving data and maintaining database for faster searching?
    When using database what type of the database to use?

    Personally I like text files but I'm aware database representation being faster. Especially for searching.

    • Michael Bishop

      Michael Bishop - 2008-03-24

      I would think that the speed of database searches makes it a must have (for both songs and bibles). But adding something like MySQL just for searches is probably overkill - maybe SQLite would be worth a look.

      Are there any reasons why the text files need to be retained? If it is for ease of copying between installations, I imagaine it shouldn't be too difficult to add an export / database dump and import facility to handle this.

    • Martin Zibricky

      Martin Zibricky - 2008-03-24

      MySQL might be used for remote database storage if we implement this feature.
      For discussion about remote database storage see this forum:

      Text files might be retained because of historical reasons (are already in use, people got familiar with text files) or someone like creating new songs only with text editor (only using command prompt) without the need to start the application (I like this for example).

      This decision (text files or local database) could be the hardest.

    • David Chivers

      David Chivers - 2008-03-26

      The same reasons for using Python or Ruby over Java would hold true for using SQLite over MySQL or Postgres. SQLite does not use a database engine, has no dependencies, and is highly portable. That's probably why uses it. Some products use SQLite "out of the box" but provide a choice of database back-ends. We can be similarly "database agnostic" by using something like wxWidgets' DatabaseLayer module.
      I like the idea of exporting and importing songs from compressed XML, perhaps as songname.xml.gz (using the zlib module in either Python or Ruby). Local database storage combined with text-based import/export provides the best of both worlds in my humble opinion.

    • celsowm

      celsowm - 2008-03-26

      Is a very important decision...

      Look this:


      why not XML Databases? Apache XIndice, MonetDB, Sedna ...

    • Martin Zibricky

      Martin Zibricky - 2008-03-26

      We should vote about this topic.

      Now I'm undecided what type of storage is better. Maybe database might be really more suitable.

      I'm against using xml database. In my opinion XML databases are not well known and established or am I wrong?

      when we'll decide using database for local storage it should be easier to implement remote database storage functionality.

    • celsowm

      celsowm - 2008-03-26

      I vote in the sqlite + function of export XML :D

    • Martin Zibricky

      Martin Zibricky - 2008-03-26

      OK. So we restrict voting only to DATABASE or XML. Sqlite or XML export is for another topic :)

      People, please vote for DATABASE or XML like celsowm did!!!

    • Jonathan Stafford

      I vote for DATABASE.

      I think this will allow more flexibility in regards to speed of operation as well as easier searching.

    • SvA

      SvA - 2008-03-26

      sqlite is a single user database, so a shared resource pool cannot be implemented using it. All other portable database systems I know off add a huge overhead (when it comes to distribution size, ease of instalation, maintenance, support, runtime resources ...)

      XML files can be handled much more flexible than data in a database (especialy when the most central part of each entry is a BLOB anyway) I do things like version control, global search and replace, syncing several repositories, all kind of reports, all with some commandline tools and very basic scripting and some common standard tools. With data in a database you loose much flexibility. For almost anything you will neeed to do real programming (or export - process - import).

      BTW, I don't think there's any real advantage of storing images, videos, sounds and the like within a database. It just make exchanging data more difficult.

      So, my vote is very clearly XML.

      And then, we need an indexed search engine, which is capable of combining fixed fields (like song title, author ...) with full text (i.e lyrics or bible text) Does anybody know of anything like this which we could use?

      By the way, what about the database mangement systems that had been sugested. Do they support accelerated searches of bulk text? Or do they just exchange file system overhead for database overhead when it comes to a bible text or lyrics search?

      • Martin Zibricky

        Martin Zibricky - 2008-03-27

        Please don't mix LOCAL and REMOTE storage method. The voting is about LOCAL storage method!

        LOCAL storage:
        - of course it is not suitable saving all to database(DB), especially media files (images, videos, ...)
        - the voting is mainly focused on lyrics
        - here isn't any advanced DB (mysql, postgresql, ..) suitable

        REMOTE storage:
        - here might be saving media files to DB useful
        - here any advanced DB comes to play
        - this is topic for another thread

        sqlite: All I know about this is that sqlite is suitable for local storage and is integraded with Qt so no other dependency is needed.

        • SvA

          SvA - 2008-03-31

          I have still not understood what you refere to by local and remote storage. Do you want to implement both? if yes, what's the purpose of the local, what the purpose of the remote storage. Which data goes where, is there any synchronisation needed.

          Is remote storage synonymous with centralised storage (and thus local storage storage of copies of tha data privat to each client)? If not what's the difference?

          If there's either local or remote storage? or ist it local as well as remote? Should it be a user choice or a design decision? Shouldn't we decide _first_ on which storage model we want before we decide on implementation details of that storage model?

          • Martin Zibricky

            Martin Zibricky - 2008-04-02

            I would like implement both storage methods.
            local storage - purpose: not much different than in opensong

            remote storage - purpose: I'm not sure with that yet. It depends on user choice. Some ideas:
            - synchronizing(adding, deleting) data with other users (e.g. I can write new lyrics on Saturday evening and the operator could download new songs on Sunday morning)
            - or changingsong could use remote storage to load data for presentation

            I think using both storage method simultaneously is not desirable.

            To remote storage should go same data as are on local host. (I think)

            Remote storage in your meaning is not synonymous for centralized storage (all clients share the same data).

            I hope I have answered all of your questions.

      • Martin Zibricky

        Martin Zibricky - 2008-03-27

        As a search engine we could try clucene. But I'm not sure if there is done any glue between python and clucene.

    • ALok

      ALok - 2008-03-27

      I say, SQLite, for searchables: songs and bibles mostly. And text/xml for everything else.
      BUT there would need to be a mass export and mass import tool that puts each song in its own file.

    • David Leigh

      David Leigh - 2008-03-27

      SQLite with XML import/export.  The mass of SQLite tools out there and the ability to query with SQL will be a major advantage over XML.

    • Gert Vrieling

      Gert Vrieling - 2008-03-27

      I vote for XML, according to the arguments of SVA-de.
      XML is not that difficult to understand so I guess it is the most simple way to make use of songs etc.
      The database version is more difficult when you are talking about filling tha database.
      This is a major reason why we have choosen OpenSong instead of the other (good) programs wich uses databases.
      When there is a discission made to use databases, it is a must to add good import and export tools to the program.
      Otherwise you make it to difficult to others to use and fill the program.
      There are no databases for the several foreigns (other languages) songs etc. So that might be a point of considering the discission.

      • Martin Zibricky

        Martin Zibricky - 2008-03-27

        What do you mean by this?
        > There are no databases for the several foreigns (other languages) songs etc.
        Does it mean that some database systems are not able to handle texts of several foreign languages? Just for example that I cannot save "English" text to database or something like that?

        Could you be more accurate with examples of DB and language incompatibilities? Localization is one of our top priorities. So this is important point!

    • Jochen Weiss

      Jochen Weiss - 2008-03-28

      I vote also for XML (text files) because ...
      ...I don't see any performance problems on a local installtion with XML. It is not sooo much data to handle and we have usally not multiple access, so it won't be time-critical. Are there any known problems with OpenSong ?
      ... XML is easy to parse.
      ... you can read and edit it very easy with thirt party tools. 
      ... we can use the old OpenSong files which is very important for the acceptance of ChangingSong as a successor of OpenSong.
      ... it's to exchange songs with others
      ... ChangingSong can run out of the box.
      ... less dependence to other projects.
      ... easier to install

      Maybe we could make an abtraction level for all data access so it won't be a big problem if we ever realize a database is a must to change.

    • Michael Bishop

      Michael Bishop - 2008-03-29

      I've put a copy of the PHP scripts I use on my local machine up on the web. Not sure if this is the right thread, but it does demonstrate a compromise between using database and XML. The scripts read in the XML files (usually directly from the Opensong folder) for editing etc, but use a MySQL database for searching.

      I wrote this code because I was frustrated with the limited search ability in Opensong and because I wanted to be able to include chord diagrams on lead sheets. In addition, I find the ability to create sets directly from search results quite a help.

      There may be a few glitches in the code because I've had to change a couple of things to get it working on  our webserver, but you should get the idea.

      The link is:

    • Martin Zibricky

      Martin Zibricky - 2008-03-29

      Hi Mike,
      So do you vote for XML?
      Indexing and searching is another topic.
      Where did you put a copy of PHP scripts? And what license do you apply for them? GPL? Your scripts might be great inspiration for remote database storage(database schema) or for including chord diagrams.

    • Michael Bishop

      Michael Bishop - 2008-03-29

      I have no objection to using XML on the presentation side and if it makes for a smooth transition for Opensong users, I would vote XML.

      [You obviously want to deal with indexing and searching elsewhere, but for what it's worth, if searching using a database is quicker, I would like to see one used].

      The scripts can now be downloaded from the link I posted earlier. I hadn't thought much about licensing and don't know all the subtleties about the various options, but they're freely available for anyone to do with as they wish - so GPL, I suppose.

      • Martin Zibricky

        Martin Zibricky - 2008-03-29

        Thanks for your scripts. If they are freely available for anyone to do with as they wish then you could use the term "public domain" for them. So every one can apply to them any license he want's.

    • Wayne

      Wayne - 2008-03-30

      My vote for songs = DATABASE

      I really like the XML files in OpenSong as it allows me to easily syncronise song folders using existing applications, however the speed of listing the songs in the folders causes me much frustration.  A cache / database index is a nice addition to speed things up, but then it requires time for refreshing.  I vote for database on the condition that there must be good import / export support and database synchronisation tools.

      For the bible modules I vote for XML.

      • Martin Zibricky

        Martin Zibricky - 2008-03-30

        We don't need to worry about Bible format if we use for Bible modules "The Sword Project" framework.

1 2 3 > >> (Page 1 of 3)

Log in to post a comment.