#1785 Merge MusicXML File(s) causes MIDI playack sound loss for entire session

[musewhirl]

I'm certain this is just some strange edge case, but under some circumstances importing a MusicXML file not only kills the audio for already existing tracks, but the new tracks imported are also silent. I've not found a way to restore sound at this point, apart from reverting to a backup.

I reverse engineered an affected session to simplify the scenario. Minor changes to it results in the problem not being observed any more. So, I've trapped it in a test.rg file and corresponding test.xml file so the bug can't sneak off. That doesn't mean it is reproducible for others though.

Attachments:

• test.rg file: It has two tracks, both associated with instrument #1. Track 1 has a single repeated segment at measure 1 containing three sequential quarter notes. Track 2 is empty. The session time signature is 3/4, but that probably does not matter.
• test.xml: 16 tracks, each with one measure containing one whole note.

Steps to reproduce:

1. Open the attached test.rg session.
2. Hit play and you should hear the repeated quarter note
3. Return the playhead to the start of the composition
4. Use File/Merge/Merge XML File(s)... to navigate to the test.xml
5. Now hit play and no audio whatsoever is heard...at least in my case

It's pretty bizarre! None of the tracks are muted, and muting and unmuting them has no effect. I am also unaware of RG having any overall mute option(?), which could have got stuck. It's always been per track as far as I know. Deleting the imported tracks has no effect. In fact, the session appears permanently damaged. I even checked the rg-file after a save, but failed to find anything unusual there either.

Clues:
I've determined that it's related to one, perhaps two, factors:

• The test.rg has two tracks, and both are associated with the same instrument #1
• The attached XML imports 16 tracks and triggers the problem
• However, if the number of imported tracks is reduced to 14 tracks, the problem disappears, and audio is heard like normal

On the surface, it seems that the magic MIDI number (16) may be involved, whether it be number of instruments/channels or tracks. But it's obviously a processing bug of some kind, which occurs during import, but rarely surfaces.

[Philip Leishman]

The problem appears to be the track id.
Normally track ids start at 0 and count up from there,
The test.rg file (which is not gzipped) has track ids > 1000.
I manually changed the ids to be 0-20 and I believe the problem is fixed.
How did you create the test.rg file ?
Can you reproduce the problem with a file which has been saved by rosegarden ?

[musewhirl]

The file was saved by rosegarden! All I did was ungzip it to take a look and see if there was some misplaced mute in there. I found nothing strange myself, and just didn't bother to gzip it back since RG doesn't care one way of the other. I didn't touch the track ids.

The original session has been around for years, and saved many times without issues...even after importing MusicXML files. It just seemed to stop on this one occasion out of the blue. I don't recall anything odd beforehand. When I retrieved the backup and loaded a different MusicXML, that worked fine, but loading the test.xml would fail every time.

More specifically, that test.rg file was originally created normally using RG, but like I said years ago and some very early version of RG. All I've done since is load XML files with new tracks into it, and remove the old tracks when not needed...all using RG's GUI...and on this occasion, no sound for that particular test.xml.

As for track ids, I didn't change them. I agree that the track ids normally start at zero. However, if you try first importing XMLs a number of times, even deleting the ones which previously existed, the new track ids probably do not reset to zero. I don't see how they can?

1. Can you confirm that?
2. Also, if the track id is the problem, then start with your own test.rg starting with 0-20, which you now say is fixed, and all you have to do is load that XML repeatedly to see if there is some kind of maximum
3. Make sure that you first import the tracks, so they are assigned new track ids, then delete the previous ones (obviously the remaining ones will not re-number themselves...I assume :-))

Note for (2), you could just increase the number of tracks to load too, but to replicate my usage, repeated imports of the same file will suffice. If the MIDI sound does disappear, then that is indeed the problem...some kind of limit on imports. The only remaining problem is that in my case, the same original RG file which stops for the test.xml, worked for a different MusicXML file...presumably it still contained (past tense now), whatever track ids it had at that time.

[musewhirl]

I forgot to mention that even the broken test.rg file starts working, regardless of the track id numbering, if you remove the last imported track, or it might of been the last two, even without saving. The audio does return. I didn't check, but I doubt the track ids of the remaining tracks changed, but you can check that too.

I went back to verify this before posting just to see if it was one or two tracks that needed to be delete, and the impossible happened. The audio worked!!! I couldn't believe my ears. I thought I got files mixed up, but I double checked both the test.rg and the test.xml and both have the original date. I carried out my own steps twice and I heard the audio! I've not even used RG since I opened the bug and uploaded the files. The bug had escaped. But it couldn't have! So, I downloaded the test.rg and test.xml, followed the steps and I still heard audio...and just possibly giggling from the bug.

Yet, from your reply, you reproduced the bug? And even went further to alter the RG file to make it go away. So, I assume you got no audio when you followed the first steps?

I can't figure out what's going on yet, or what I've missed, but I need to get back to no sound even to have chance to check, and so far every thing I do involving importing those tracks works.

1. Is it still broken at your end? (please say yes)
2. Can you follow the steps, but remove the last imported track and play from the start again
3. And repeat that one more time if the silence remains i.e. remove one more track

[musewhirl]

Ignore the deletion test in the previous post because I found a way to prove that it does not return the audio.

Finally, I recreated the scenario again with a new test-2.rg file. I had no choice but to go back to the original session and again remove all the extra noise (using RG's GUI) i.e. tracks and segments of music which have nothing to do with the issue. I'm uploading it because it's closer to the original scenario. Here, the first track is the same as the original i.e. just a click track, which reminded me of something I had neglected to mention for the history of the original session.

You were right about noticing the track numbers. In test-2.rg you will see that the first track id is indeed zero. In the end, I don't believe that will really matter. But the second track id is 1023. That's because over time, I never deleted the first click track. I only imported a number of XML tracks and removed any previously imported tracks i.e. track zero would normally always be there because it never needed to change...just click, while the new tracks would be created with higher and higher numbers by RG during import. Perhaps there is some limit involved?

So, if you repeat the steps to reproduce for this bug using test-2.rg, you will still get no sound as before, assuming you got no sound for the previous test.rg. Whatever I do now...test.rg works defiantly.

But to prove that the track id alone is not the problem, and making sure you backup test-2.rg so you can repeat your testing, here is another scenario, but do not save the session during it:

1. Open test-2.rg
2. Add a new track, so you now have three tracks
3. Remove track 2 i.e the one which raised your suspicions (track id 1023), leaving whatever RG comes up with for the new track
4. You should now have two tracks again
5. Import test.xml and you should still get no sound

The point is that whatever the history of the file, RG created the new track just now and the import problem is still there. You could try swapping out the click track too I suppose, and then you have two tracks created by the current version of RG and presumably the same problem. Beware intervening saves during testing though...I'm not sure, but I think that might interfere with the test in some way.

If there is some kind of track id limit, then the only way to determine it for an RG file created from scratch with the current version, is to first create a session with one track e.g. the click track although what it contains is immaterial. Then create a large XML file to quickly build up the track ids, deleting imports as you go, but leaving the first track alone, until the sound stops. I may be able to put some time aside to try it, but you might get there first or by other means.

[Philip Leishman]

OK Rosegarden save and load preserves track ID so I guess adding and deleting tracks over time may push the track id up to these high values.
I use asan which means I actually get a crash when importing the xml (heap-buffer-overflow ). If that happens anything can happen - this is the source of the bug.
I thought of remapping the track Ids when loading the file - that would fix this issue.
Having said that - why shouldn't the Ids be large !
There is a limit of 1024 tracks in ControlBlock however this is actually used as an Id limit.
I will look into this further.

[Philip Leishman]

I have removed the track (and track id) limit from ControlBlock.
The original test.rg test.xml merge works now.
I did some basic testing but further testing would be helpful.
See codeberg merge request

[musewhirl]

Ahhh, but the test.rg/test.xml combo works for me now, without any fixes. In fact, I cannot get it to fail any more, which is strange.

However, the test-2.rg.xml/test.xml does fail (last time I checked). So, if that originally failed for you prior to your fix and worked after it, that would be more certain.

Incidentally, after posting, I realized that 1023 was so close to 1024 that it felt like one of those computer-related favorites. I thought some variable might have flipped back to 0, or worse, -1.

[musewhirl]

This bug is pretty bad. We'll have to be careful with testing.

I returned to the original session, and now it fails again...no sound. I copied the file to a backup directory.

Without the fix, I had to change the RG session file track ids manually, as you had done. Only problem is that I have lots of segments in that session and I had to match the track ids in those too. Luckily, there still existed a find and replace sequence which didn't happen to hit on other unrelated attribute values. However, over time it would not be possible to do this.

In addition, I proved its randomness:

• I went to look at the backup of the original session mentioned above...and it worked. All I had done was copy the file to a different directory...Oh..and I did shutdown and restart RG. I think it was the latter which did the trick.
• My comment about "the last time I looked" at test-2.rg panned out. I went back today and it half worked. The click track did have sound where there was no sound at all, but the imported tracks were still silent. I closed RG and restarted, and that time all silence again.

[Philip Leishman]

I have update ControlBlock with mutex locking and restructured a bit.
My tests work OK.
See merge request.