Re: [Asaph-developers] xml format

[David F. <da...@sj...>]

Daniel Azuma wrote:

>Hi David,
>
>Okay, time for a view into the (largely undocumented) guts of the XML
>format...
>  
>
Hi Daniel,

Wow, thanks for all the details!

>On Tue, 11 May 2004, David Fraser wrote:
>  
>
>>1) Is there a way to mark blocks as chorus / verse etc. The reason this
>>would be useful is for style changes e.g. displaying the chorus in
>>italics (I know it can currently be indented).
>>    
>>
>In the model, SongBlocks are FieldContainers. That means, in the XML, you
>can have a <fields> element inside <block>. (It needs to be the first
>element if it is present.) See the <song> elements for examples of the
>structure of <fields>.
>
>The canonical way to mark the block type is to use a stringfield with name
>"type". Set the value to "verse" for verse, "chorus" for chorus, "bridge"
>for bridge. I don't have any other constants defined at the moment.
>
>(Also note that the constants for these in model/SongBlock.java are
>currently wrong and will be fixed in 0.5.4. The field name should be
>"type", not "name".)
>
>This is undocumented and not currently actually used by Asaph's GUI for
>anything, but block fields are part of Asaph's model, and so Asaph will
>preserve them when it edits the file.
>  
>
That's fantastic, thanks for the info... I will probably only get to 
doing this after some of the other tasks, but its helpful to know.

>>2) Likewise, it would be nice to be able to mark lines as translations
>>(we sing a number of non-English songs that often have translations
>>below each line).
>>    
>>
>Lines can't be marked. That was a design decision I went back and forth
>on, but finally decided it wasn't necessary. However, runs of text CAN be
>marked with a type. In fact, that is how "comments" are implemented.
>
>Here's how it works:
>
>The <line> element is a container that is supposed to contain <text> and
><c> elements. So the "clean" way to represent a line in XML would look
>something like this:
>
><line><text>I </text><c main="D"/><text>love You Lord</text></line>
>
>The <text> element can include an optional "type" attribute. Comments are
>actually text elements with type="comment". For example:
>
><line><text type="comment">(chorus)</text></line>
>
>However, Asaph's XML format includes several "shorthand" modifications of
>this notation. First, if raw text appears in the <line> element, it is
>treated as though it were in a <text> subelement. So the first example
>above could be written as:
>
><line>I <c main="D"/>love You Lord</line>
>
>Second, the <comment> element is equivalent to <text type="comment">. So
>the second example above could be written as:
>
><line><comment>(chorus)</comment></line>
>
>Asaph outputs its files using the above two shorthand rules because they
>result in smaller files, and because I found them easier to read in a text
>editor.
>  
>
I agree, having to put "text" markers on every section would be 
annoying, it is clean and nice to read.

>Anyway, the point of the above discussion is that you can mark any text
>run with a type by explicitly including the <text> element, and supplying
>a type. The only canonical type I have defined right now is "comment". But
>you can define your own for now, and Asaph will preserve it when it writes
>out a file. You can't mark a line, but you can mark all the text runs
>within a line. If you're including translation lines in a song, the
>translation line probably won't have chord markings in it, so there will
>probably only be one text run in each line. It would look something like:
>
><line><text type="translation">translation of the above</text></line>
>  
>
Yes, this would work great for us, I'll have a look at implementing it.
We actually take a openoffice document that has all the songs and 
convert it to the song database, so what I am doing is improving the 
conversion tool to automatically pick up comments, translations etc... 
Once it is all finished we can just dump the document and use the song 
database. Also I haven't yet done chord conversion, I just ignore the 
chords lines, so that needs to be done, like it was for the experimental 
text import. I would like to see if I can use some OpenOffice 
intelligence to work out the chord placement within the line...)

>>I was also wondering about the use of ids to mark lines, blocks, songs,
>>etc. They seem a bit redundant to me in that they are predictable and
>>don't neccessarily add much information to the song database.
>>    
>>
>Okay, IDs. I figured this question would come up eventually. Here's the
>rationale behind how Asaph uses IDs.
>
>IDs are meant to be a unique, permanent, serializable reference to a song,
>chordset, variation, or block. Song IDs are unique within a database.
>Chordset, variation, and block IDs are unique within a song. They are
>intended to provide a permanent way for external entities to identify
>songs, or certain elements within songs, even if the song changes in minor
>ways.
>  
>
OK, that makes sense, I can see why you have them now...

>Thus, IDs should not change if changes are made around them. For instance,
>the ID of a song within its database will not change even if you change
>the song title. Or the ID of a block within a song will not change even if
>you reorder blocks, delete blocks, or otherwise cause the block's
>numerical index within the song to change. Therefore, it is "safe", for
>example, for an external entity like a URI to refer to the chorus of a
>song by its block ID, while it is not safe to refer to it as "The second
>block", because the former will not change if a block is inserted in the
>beginning, whereas the latter will.
>
>IDs should never be visible to the user, except if they show up in a URI
>or something like that. Unfortunately, they do show up in filenames in the
>indexed database, but the intent for indexed databases is that users
>shouldn't mess with the individual files anyway.
>  
>
OK that makes sense. I think because I had never done extensive editing 
I had only seen sequential ids, but if blocks/songs maintain ids after 
changes, thats cool. Is there a standard syntax for URIs referring to 
songs / blocks?

>>In particular, I was thinking that it would be quite useful to be able
>>to have a song file specified for the song (in an indexed database) so
>>instead of being song_b_a.xml it would be nothing_but_the_blood.xml.
>>The advantage of this would be that you could have multiple indexes on
>>the same set of songs and you could use the indexes to save lists of
>>songs e.g. the list of songs for this Sunday morning etc.
>>    
>>
>The way something like this should work-- at least, the way it is going to
>work in Asaph when it starts doing presentations and more advanced output
>capabilities-- is that you should have a data structure that points to
>songs by their song ID. One of the key APIs I'm looking at designing for
>1.1 is a "song list" API. So a presentation would run off a song list that
>is effectively just an array of pairs (database, songID). For example:
>
>(database 1, song D)
>(database 1, song A)
>(database 2, song D)
>(database 1, song E)
>
>and so forth.
>
>Similarly, within each song, you'd generally not want to show the blocks
>in the order they appear in the database. For a song with three verses and
>a chorus, the song is usually structured like this:
>
>block A: verse 1
>block B: chorus
>block C: verse 2
>block D: verse 3
>
>because that is how it usually is printed in a lead sheet. But when you do
>your presentation, you probably want something like:
>
>verse 1
>chorus
>verse 2
>chorus
>verse 3
>chorus
>
>or even going back and repeating verses, and so forth. So a song list
>with block reordering would actually look something like this:
>
>(database1, song D)
>  block A
>  block B
>  block C
>  block B
>  block D
>  block B
>(database1, song A)
>etc...
>
>That way, you don't have to modify the song in the database. You're just
>building a list of "pointers" to songs and parts of songs in the database,
>and leaving the database itself alone. That's kind of important: the way I
>see Asaph being used is that users should be able to do all manner of
>manipulating songs for presentations and building songbooks and the like,
>without having to modify the database itself. A church can have a central
>database of songs that is read-only to everyone except one person who is
>responsible for managing songs and dealing with all the CCLI stuff. But a
>music team member who needs to just put together a presentation can come
>in and build one of the above lists, and will thus have the freedom to do
>whatever reordering is necessary without having to make database changes.
>  
>
This sounds great, and it all makes lots of sense.

>So the key here is the use of IDs, which are like pointers (to use
>developer-speak) to those elements. Song D is still song D even if someone
>goes in later and changes its title; the song list you're using for your
>presentation won't break. Block B is still the chorus even if someone goes
>in later and inserts an introduction before the first verse. The
>introduction might be located before the first verse, but it will be
>assigned the "next" ID, which would be "E". Block B is still the chorus,
>and your song list doesn't have to change, and your presentation doesn't
>break.
>
>The naming scheme of song xml files (.asml) files within an indexed
>database is song_<id>_<revision>.asml. This is because songs should always
>be specified by their unique ID, not by name or anything else, because
>there is no guarantee that anything else will be unique, nor is there any
>guarantee that other things won't change. In my personal database, there
>are three songs entitled "I Want To Know You". So I can't use the title as
>the filename.
>
>Anyway, that's the vision. And if you have comments on that, please feel
>free to speak up.
>  
>
I think its great. Maybe it would be a nice idea to specify an xml 
format for the song list idea.
That way lists could be saved and loaded relative to a (simple or 
indexed) database.

>Now, that said, I'm guessing your problem is that you're just copying
>songs into a new database, and feeding the entire database into pyasaph
>for a presentation, (which is fine for now), and you're having trouble
>coming up with an easy way to allow a user to specify the order of songs,
>because in Asaph's database, there really isn't the concept of the order
>of the songs, and so Asaph doesn't provide any user interface to reorder
>songs. I don't really have a good solution for this, because the intent
>for Asaph itself is eventually to hide all of the ID junk behind an API
>and user interface for creating song lists. You could throw together and
>document a simple mechanism for specifying the order of song IDs in an
>input text file or something. (Sorry, I still haven't had a chance to
>download and try out pyasaph myself, so I don't really have a good
>understanding of what it's currently doing in terms of ordering songs.)
>  
>
Yes I think I will have a look at this, though I think an xml file for 
song lists would be the best solution.
Actually what I've focussed on more is having a simple way to search 
song titles.
So there is a text box above the list of songs that filters as you type 
(matching substrings).
This makes it very quick to find songs if for example the worship leader 
chooses a song that wasn't on the list.
So we generally have the whole database loaded, and then open all the 
songs that are on the list.
There is a multiple document interface so you can see a few songs at 
once or select one from a window drop down.
It seems to be working OK for now. I'll post some screenshots when I get 
a chance.

The main improvement I would like to make is that at the moment I just 
use an HTML window to display the songs (on the second display if there 
is one). But this doesn't have any concept of blocks, so you have to 
scroll up and down in it using the mouse.
What I want to do is allow the controller to double-click a block on the 
main screen and for that block to move to the top of the display screen. 
So I have to define a view class that will do this...

Thanks for the help and info

David