Re: [Asaph-developers] xml format

[Daniel A. <da...@ka...>]

Hi David,

Okay, time for a view into the (largely undocumented) guts of the XML
format...

On Tue, 11 May 2004, David Fraser wrote:

> 1) Is there a way to mark blocks as chorus / verse etc. The reason this
> would be useful is for style changes e.g. displaying the chorus in
> italics (I know it can currently be indented).

In the model, SongBlocks are FieldContainers. That means, in the XML, you
can have a <fields> element inside <block>. (It needs to be the first
element if it is present.) See the <song> elements for examples of the
structure of <fields>.

The canonical way to mark the block type is to use a stringfield with name
"type". Set the value to "verse" for verse, "chorus" for chorus, "bridge"
for bridge. I don't have any other constants defined at the moment.

(Also note that the constants for these in model/SongBlock.java are
currently wrong and will be fixed in 0.5.4. The field name should be
"type", not "name".)

This is undocumented and not currently actually used by Asaph's GUI for
anything, but block fields are part of Asaph's model, and so Asaph will
preserve them when it edits the file.

> 2) Likewise, it would be nice to be able to mark lines as translations
> (we sing a number of non-English songs that often have translations
> below each line).

Lines can't be marked. That was a design decision I went back and forth
on, but finally decided it wasn't necessary. However, runs of text CAN be
marked with a type. In fact, that is how "comments" are implemented.

Here's how it works:

The <line> element is a container that is supposed to contain <text> and
<c> elements. So the "clean" way to represent a line in XML would look
something like this:

<line><text>I </text><c main="D"/><text>love You Lord</text></line>

The <text> element can include an optional "type" attribute. Comments are
actually text elements with type="comment". For example:

<line><text type="comment">(chorus)</text></line>

However, Asaph's XML format includes several "shorthand" modifications of
this notation. First, if raw text appears in the <line> element, it is
treated as though it were in a <text> subelement. So the first example
above could be written as:

<line>I <c main="D"/>love You Lord</line>

Second, the <comment> element is equivalent to <text type="comment">. So
the second example above could be written as:

<line><comment>(chorus)</comment></line>

Asaph outputs its files using the above two shorthand rules because they
result in smaller files, and because I found them easier to read in a text
editor.

Anyway, the point of the above discussion is that you can mark any text
run with a type by explicitly including the <text> element, and supplying
a type. The only canonical type I have defined right now is "comment". But
you can define your own for now, and Asaph will preserve it when it writes
out a file. You can't mark a line, but you can mark all the text runs
within a line. If you're including translation lines in a song, the
translation line probably won't have chord markings in it, so there will
probably only be one text run in each line. It would look something like:

<line><text type="translation">translation of the above</text></line>

> I was also wondering about the use of ids to mark lines, blocks, songs,
> etc. They seem a bit redundant to me in that they are predictable and
> don't neccessarily add much information to the song database.

Okay, IDs. I figured this question would come up eventually. Here's the
rationale behind how Asaph uses IDs.

IDs are meant to be a unique, permanent, serializable reference to a song,
chordset, variation, or block. Song IDs are unique within a database.
Chordset, variation, and block IDs are unique within a song. They are
intended to provide a permanent way for external entities to identify
songs, or certain elements within songs, even if the song changes in minor
ways.

Thus, IDs should not change if changes are made around them. For instance,
the ID of a song within its database will not change even if you change
the song title. Or the ID of a block within a song will not change even if
you reorder blocks, delete blocks, or otherwise cause the block's
numerical index within the song to change. Therefore, it is "safe", for
example, for an external entity like a URI to refer to the chorus of a
song by its block ID, while it is not safe to refer to it as "The second
block", because the former will not change if a block is inserted in the
beginning, whereas the latter will.

IDs should never be visible to the user, except if they show up in a URI
or something like that. Unfortunately, they do show up in filenames in the
indexed database, but the intent for indexed databases is that users
shouldn't mess with the individual files anyway.

> In particular, I was thinking that it would be quite useful to be able
> to have a song file specified for the song (in an indexed database) so
> instead of being song_b_a.xml it would be nothing_but_the_blood.xml.
> The advantage of this would be that you could have multiple indexes on
> the same set of songs and you could use the indexes to save lists of
> songs e.g. the list of songs for this Sunday morning etc.

The way something like this should work-- at least, the way it is going to
work in Asaph when it starts doing presentations and more advanced output
capabilities-- is that you should have a data structure that points to
songs by their song ID. One of the key APIs I'm looking at designing for
1.1 is a "song list" API. So a presentation would run off a song list that
is effectively just an array of pairs (database, songID). For example:

(database 1, song D)
(database 1, song A)
(database 2, song D)
(database 1, song E)

and so forth.

Similarly, within each song, you'd generally not want to show the blocks
in the order they appear in the database. For a song with three verses and
a chorus, the song is usually structured like this:

block A: verse 1
block B: chorus
block C: verse 2
block D: verse 3

because that is how it usually is printed in a lead sheet. But when you do
your presentation, you probably want something like:

verse 1
chorus
verse 2
chorus
verse 3
chorus

or even going back and repeating verses, and so forth. So a song list
with block reordering would actually look something like this:

(database1, song D)
  block A
  block B
  block C
  block B
  block D
  block B
(database1, song A)
etc...

That way, you don't have to modify the song in the database. You're just
building a list of "pointers" to songs and parts of songs in the database,
and leaving the database itself alone. That's kind of important: the way I
see Asaph being used is that users should be able to do all manner of
manipulating songs for presentations and building songbooks and the like,
without having to modify the database itself. A church can have a central
database of songs that is read-only to everyone except one person who is
responsible for managing songs and dealing with all the CCLI stuff. But a
music team member who needs to just put together a presentation can come
in and build one of the above lists, and will thus have the freedom to do
whatever reordering is necessary without having to make database changes.

So the key here is the use of IDs, which are like pointers (to use
developer-speak) to those elements. Song D is still song D even if someone
goes in later and changes its title; the song list you're using for your
presentation won't break. Block B is still the chorus even if someone goes
in later and inserts an introduction before the first verse. The
introduction might be located before the first verse, but it will be
assigned the "next" ID, which would be "E". Block B is still the chorus,
and your song list doesn't have to change, and your presentation doesn't
break.

The naming scheme of song xml files (.asml) files within an indexed
database is song_<id>_<revision>.asml. This is because songs should always
be specified by their unique ID, not by name or anything else, because
there is no guarantee that anything else will be unique, nor is there any
guarantee that other things won't change. In my personal database, there
are three songs entitled "I Want To Know You". So I can't use the title as
the filename.

Anyway, that's the vision. And if you have comments on that, please feel
free to speak up.

Now, that said, I'm guessing your problem is that you're just copying
songs into a new database, and feeding the entire database into pyasaph
for a presentation, (which is fine for now), and you're having trouble
coming up with an easy way to allow a user to specify the order of songs,
because in Asaph's database, there really isn't the concept of the order
of the songs, and so Asaph doesn't provide any user interface to reorder
songs. I don't really have a good solution for this, because the intent
for Asaph itself is eventually to hide all of the ID junk behind an API
and user interface for creating song lists. You could throw together and
document a simple mechanism for specifying the order of song IDs in an
input text file or something. (Sorry, I still haven't had a chance to
download and try out pyasaph myself, so I don't really have a good
understanding of what it's currently doing in terms of ordering songs.)

Daniel