From: Florian J. <fl...@wi...> - 2014-01-22 01:45:16
Attachments:
signature.asc
|
Hi all, Lots of things in MusE are pretty complicated, because they involve communication with the audio thread, or because they require us to pre-allocate things outside of the audio thread and so on. I want to question this realtime capability of MusE, please bear with me: My first point to legitimate all this is: MusE aims to be realtime capable, but *it is not*. What makes me say that? Well, even though realtime patterns are implemented in MusE (such as message passing with the audio thread, because we can't use mutexes, and pre-allocating buffers outside), they are not consequently used. Some examples: when undo()ing, this can take an undetermined amount of time inside the audio thread; it basically scales in O( number_of_operations_to_undo ), which might be large (e.g. if i have deleted 1000 events at once.). MusE might fail to meet the realtime constraints -> glitch. in many places, MusE does tempomap lookups *in the audio thread*. C++'s map structure, however, does not make any guarantees about it's worst-case-runtime. If we're unlucky, we might fail to meet our realtime constraints. (I must admit, a large part of this is my fault.) Every frame, MusE collects the events to play in the next buffer period by finding begin and end iterator in the EventList, and loops through lots of data structures. Most of these are from the STL and do not make any timing guarantees. Additionally, it's plain unnecessary. One could just increment an iterator (which, of course, still needs to be an iterator in a realtime-safe structure), and a search-in-binary-tree would only be necessary upon a seek(). The Audio thread may not allocate any memory, therefore, lots of uglyness is done to do this allocation outside of the audio thread (e.g. by the gui thread, and then only passing (smart) pointers). However, then, newly created events are inserted into a std::multimap. Which internally needs to malloc() a new node of the binary tree (or whatever internal data representation is used). Malloc is known to not fulfill any realtime constraints -> muse might glitch. Certain things are done by switching the audio thread into Idle state, which stops processing any data, and instead delivers zeros. While MusE will not fail at realtime here, this still introduces glitches. We may not do this, msgIdle must be eliminated. This puts me in a dilemma: a) we need to *massively* redesign MusE to make it realtime capable. This would involve hard separation between audio and GUI, no shared data! They should be put into different processes, even. It would probably affect GUI performance. We must restrict editing operations while playing back. Forbid tempo changes, forbid "large" event manipulations while playing. b) We drop this hard realtime thing. MusE will become a studio software which should not, but *might* glitch. We would greatly reduce code complexity by not having to respect all the corner cases. We will keep the message passing, but we will use certain realtime-breaking functions, like std::map's functions. I can't really see another way out of this. Actually, i like a), but i fear that this will be too much work; i think that would almost equal a full rewrite from scratch. Could we probably even use mutexes for synchronisation? I mean, our critical blocks when editing are pretty short, so maybe priority inversion could help us? With priority inversion, when a low-prio thread holds a mutex that is required by a high-prio thread, then it temporarily also gains high priority until the mutex is resolved; then control is given to the high-prio thread. Please discuss, I'm totally insecure about this :/ Florian |
From: Tim E. R. <ter...@ro...> - 2014-01-23 03:38:12
|
On January 22, 2014 02:45:08 AM Florian Jung wrote: > Hi all, > > Lots of things in MusE are pretty complicated, because they involve > communication with the audio thread, or because they require us to > pre-allocate things outside of the audio thread and so on. > > I want to question this realtime capability of MusE, please bear with me: > > > My first point to legitimate all this is: > > MusE aims to be realtime capable, but *it is not*. > > What makes me say that? Well, even though realtime patterns are > implemented in MusE (such as message passing with the audio thread, > because we can't use mutexes, and pre-allocating buffers outside), they > are not consequently used. Some examples: > > when undo()ing, this can take an undetermined amount of time inside the > audio thread; it basically scales in O( number_of_operations_to_undo ), > which might be large (e.g. if i have deleted 1000 events at once.). MusE > might fail to meet the realtime constraints -> glitch. > > in many places, MusE does tempomap lookups *in the audio thread*. C++'s > map structure, however, does not make any guarantees about it's > worst-case-runtime. If we're unlucky, we might fail to meet our realtime > constraints. (I must admit, a large part of this is my fault.) > > Every frame, MusE collects the events to play in the next buffer period > by finding begin and end iterator in the EventList, and loops through > lots of data structures. Most of these are from the STL and do not make > any timing guarantees. Additionally, it's plain unnecessary. One could > just increment an iterator (which, of course, still needs to be an > iterator in a realtime-safe structure), and a search-in-binary-tree > would only be necessary upon a seek(). > > The Audio thread may not allocate any memory, therefore, lots of > uglyness is done to do this allocation outside of the audio thread (e.g. > by the gui thread, and then only passing (smart) pointers). > However, then, newly created events are inserted into a std::multimap. > Which internally needs to malloc() a new node of the binary tree (or > whatever internal data representation is used). Malloc is known to not > fulfill any realtime constraints -> muse might glitch. > > Certain things are done by switching the audio thread into Idle state, > which stops processing any data, and instead delivers zeros. > While MusE will not fail at realtime here, this still introduces > glitches. We may not do this, msgIdle must be eliminated. > > > > This puts me in a dilemma: > > a) we need to *massively* redesign MusE to make it realtime capable. > This would involve hard separation between audio and GUI, no shared > data! They should be put into different processes, even. > It would probably affect GUI performance. > We must restrict editing operations while playing back. Forbid tempo > changes, forbid "large" event manipulations while playing. > b) We drop this hard realtime thing. MusE will become a studio software > which should not, but *might* glitch. We would greatly reduce code > complexity by not having to respect all the corner cases. We will > keep the message passing, but we will use certain realtime-breaking > functions, like std::map's functions. > > I can't really see another way out of this. > > Actually, i like a), but i fear that this will be too much work; i think > that would almost equal a full rewrite from scratch. > > Could we probably even use mutexes for synchronisation? I mean, our > critical blocks when editing are pretty short, so maybe priority > inversion could help us? > > With priority inversion, when a low-prio thread holds a mutex that is > required by a high-prio thread, then it temporarily also gains high > priority until the mutex is resolved; then control is given to the > high-prio thread. > > Please discuss, I'm totally insecure about this :/ > Florian Relax, you are right. It has taken me a long time but I realize now that the way MusE handles processing is wrong. I mean I've known for some time that the heavy processing in the process callback was a bit weird. I watch LAD and know that others use threads to pump data in and out of the Jack process callback (I recall at least one app uses multiple threads, impulse response app I think). But I was not quite sure what to do, or why ours was so wrong. The last week or so, examining the code I'm thinking more about it now. To make a long story (which follows) short, we'll do it the right way, something like this: https://github.com/jackaudio/jack2/blob/master/example-clients/capture_client.c A bit of back-story may help to understand the current MusE situation: --------------- I coded Windows audio apps since early '90s. Both MMSystem and DirectSound. And I used mutexes (Critical Sections). I dug up my very old code yesterday. I started using Linux in 1997. I didn't code much of my own except fixing small bugs locally, until mid 2000s. I'm proud to have some fixes in the kernel - the SBLive driver and supporting DSP loader and GUI. Was very challenging. By around 2005 I had tried MusE, and the others. I said before probably one of the best reasons for choosing MusE was simply that input 'Transpose' box. Nobody else has it, even today I think. I could tell it was a good app, most resembling what I was used to - Cakewalk. I set about (not by choice!) fixing some painfully obvious crashes/bugs. But I was still a Linux audio app novice. Werner *already* had MusE-2 going (now muse-evolution in attic), but it was in heavy development but not really usable. Whereas MusE-1 was at least usable. And I needed something stable *NOW*, if ya know what I mean. So I chose MusE-1, as a sort of 'proving ground' - if I could fix bugs in MusE-1, learn how it works, maybe help Werner later, not step on his toes. But work on MusE-2 stalled, I think because he moved on to MusE-Score. My fixes made MusE-1 stable which is what we all needed - as musicians. If you think we do bad threading things now (you're right but) you should have seen all the many baaad crashes of threading things going on before. Like, things all over the place were manipulating STL containers while other threads were erasing items from them. Ugly. Do I care if I simply manipulate a boolean variable across threads? (Which I have done). I researched it recently and found, well, it's relatively safe but bad practice. We've never had a bug/crash because of it. IIUC some OSs like Linux and even CPUs atomize such fundamental operations. I tend to expect or hope that they would, not wanting to deal with it myself. What's your understanding of this, and is it a good thing? Shortly before you joined, Qt3 was dying very quickly, like within weeks. We talked about what to do. MusE-1 was Qt3 at the time. As MusE-2 was already a Qt4 app, I said "Well, I might as well abandon all work now and move over to Werner's MusE-2." But we all agreed that it was gonna be a /long/ time to make MusE-2 stable. In fact when I asked, the consensus was "Sick with the MusE-1 code base". Then something funny happened. Robert used Qt4's 'automatic' porting tool, just for a laugh, on MusE-1-Qt3. He said "Well I dunno if this is good, many things still broken...", shaking his head with sinking dismay, metaphorically speaking. Well, I took that hard work and /ran/ with it. And just then, Orcan joined and we all ported the thing over to Qt4. I guess what I'm saying is don't be too critical of the path chosen... So here we are. Stable. Usable. Correct? Expandable? Cutting edge? Mm... ----------------- Moving on... I believe we can make our process engine proper in... a few weeks, after discussing to make sure we know what we're doing and it'll work. Rough blueprint: ============== The Jack process callback should do one job only: Move data into and out of the Jack ports. You know all that crap we do in our process callback? That's gotta be moved outta there into OUR audio thread. As the Jack capture_client example shows above, one should really only use ring buffers (FIFO queues), and such, to PUMP data from/to a 'worker' thread (our audio thread) to/from the Jack ports, during the Jack process callback. And that's about all. Not all the stuff we do. I'm now seriously wondering why we have an audio thread at all, if all our heavy processing is done in our Jack process callback. I think, at one time, MusE was headed in the right direction but took a wrong turn somewhere. Someone long ago had actually linked the audio thread with Jack's thread using a pthread join or something like that. Sletz from the Jack team saw that one day, when I asked him a question. He gave us such *shit*. I never heard him *swear* loudly and publicly before or since. With these new changes, now the audio thread really MATTERS: Basically we keep our Audio::process() method and modify it, but now call it only from within our audio thread, gathering data and so on, putting/pulling data from the ring buffers. One thing I thought of: Once you put data to a ring buffer you can't take it out again from the same thread. So during a transport seek, our Jack process callback would need an indication of this seek condition to clear and ignore any 'stale' older invalid ring buffer items. Jack has such functions to check transport state. Now, there's the question of synchronization. Some operations in the audio thread may need to wait until our Jack callback has finished or has data, such as 'live' input effects operations that need input data first. Yet some operations need only just gingerly pre-pump items into the ring buffers without waiting - perhaps playing a wave file, as a rough example. We should consider what exactly we need the audio thread to do for us... Here I will interject that if we were to keep the current system, I was going to install a ring buffer for all 'operations' (in the undo/redo system) which only require Song::executeOperationGroup2 and Song::revertOperationGroup2. That is, operations like UndoOp::AddEvent or UndoOp::ModifyEvent do not require stage 1 or stage 3 non-realtime operations. It is really stupid that many simple operations wait for the audio thread. Rule of thumb, as you say Flo: The GUI should be de-coupled from audio so that almost NOTHING in the GUI needs to wait for ANYTHING. Careful timing, queuing, signaling, and stream inhibiting upon mouse presses, will help. For example: When somebody turns a midi controller knob in the controller panels, Audio::msgPlayMidiEvent() is called which waits at /each/ movement. Stupid baaad slooow, with large Jack periods. Using a ring buffer, the knob is free to turn at full speed and pump changes into the ring buffer and whatever reads the ring buffer deals with the changes, possibly even at a much slower rate than the knob is turning. See how I did this with all the /audio/ controls such as volume and pan. I mean crap, we even have an audio-thread-to-GUI-thread signaling mechanism (see Song::seqSignal()) which theoretically could be used to trigger the Song::update() calls that would be sooo needed, after a ring buffer 'operations' item handling would be finished in the process callback. (But it's a poor example that uses a pipe with single character messages.) Anyways we won't try to pursue that route now... If you want to start a new branch I'll try to work out some details and let's talk about how to do this. I think it would be relatively straight forward... Here's to "Doin' it right on the wrong side of town" (old Canadian hit song). --- I leave you with an idea to chew on: While looking at the undo/redo code I realized it can be destructive - changes can be lost if you undo, then change something, redo turns off. I thought "What if we could keep everything, discard nothing?" Thus I give you... Undo "Trees". Turns out emacs and vim have had "Undo Trees" for years. But apparently not many other apps. Look it up! As you can imagine, it needs a viewer of some kind, which emacs/vim have. Imagine being able to go to and from ANY modified state from ANY time in the project's entire history ! --- Thanks. Tim. |
From: Tim E. R. <ter...@ro...> - 2014-01-23 04:04:27
|
On January 22, 2014 10:37:55 PM Tim E. Real wrote: > You know all that crap we do in our process callback? > That's gotta be moved outta there into OUR audio thread. > ...snip... > I'm now seriously wondering why we have an audio thread at all, D'oh! Stupid me, our Audio class is not based on our Thread class. We currently don't really have our own 'audio thread' apart from Jack's process callback, do we... Er, well you know what I ultimately meant: We create an audio thread which does all the work. Carry on... Tim. |
From: Tim E. R. <ter...@ro...> - 2014-01-23 05:47:15
|
On January 22, 2014 10:37:55 PM Tim E. Real wrote: > While looking at the undo/redo code I realized it can be destructive - > changes can be lost if you undo, then change something, redo turns off. > I thought "What if we could keep everything, discard nothing?" > > Thus I give you... Undo "Trees". > > Turns out emacs and vim have had "Undo Trees" for years. > But apparently not many other apps. Look it up! > As you can imagine, it needs a viewer of some kind, which emacs/vim have. > > Imagine being able to go to and from ANY modified state from ANY time > in the project's entire history ! Forgot to mention, I was reminded of... trees... like GIT trees. How about using a real CVS system for songs, changes, and project files? Concurrent music project users? And across the internet? GIT API? I mean... if someone was to simply post a MusE .med file to a CVS repo, it would automatically have CVS tree tracking eh? But is it really possible to go from ANY node in a (GIT) CVS tree to ANY other node and back again? I would hope so... T. |
From: Florian J. <fl...@wi...> - 2014-01-23 17:46:33
Attachments:
signature.asc
|
Hi Tim :) i've read through all of your mails, and can mainly agree with you. Just a few things for now, more will follow until next week, I think. Am 23.01.2014 schrieb Tim E. Real: > We currently don't really have our own 'audio thread' > apart from Jack's process callback, do we... that's true, the audio thread IS the jack-callback-thread. > Rule of thumb, as you say Flo: The GUI should be de-coupled from audio > so that almost NOTHING in the GUI needs to wait for ANYTHING. > Careful timing, queuing, signaling, and stream inhibiting upon mouse presses, > will help. Can you please elaborate more on 'timing' and 'stream inhibiting'? > I guess what I'm saying is don't be too critical of the path chosen... Well, the path itself is to be criticized. What I'm not too critical about is the people who made this path happen ;). It wasn't too obvious that all this will implode at some point, so nobody is to blame here. But it did implode, so we need to analyse our mistakes made. > I believe we can make our process engine proper in... a few weeks, > after discussing to make sure we know what we're doing and it'll > work. > If you want to start a new branch I'll try to work out some details and let's > talk about how to do this. I think it would be relatively straight forward... I'm not sure whether we should try to fix MusE 2.0. I think, we should really start from scratch, with an empty project, and write it from bottom up again. If it turns out that we can copy'n'paste most of it, then that's cool. If it turns out that we cannot, then MusE is really in need of a rewrite at these places. There would be no point in "trying to fix" it then. I want to get away from this "greedy" development attitude (not meant pejorative, but like in "greedy algorithm"): We never give away a feature that we already have, and therefore try to get the best out of *what we have*. We should, however, get the best out of "what's possible". I think that we are caught in a local maximum, and we need to let go first, and can then approach the global maximum. Also, most importantly: I would not start right now. We should -- probably the first time in MusE's life -- make a draft. A design paper, you know, something like "If the GUI wants to do $things, it MUST do $stuff. During this, it SHOULD display a progress indicator. This MAY be..." After we have this design document, we proof-read it. And then we let jack developers and/or LAD proof-read it. And then we proof-read it again and THEN, we start coding. Benefits from this: The design paper almost doubles as documentation. Interested coders can quickly get a grip on MusE and support us. Please, let's not make the same mistake again. Let's think about what to do, make a complete specification. That way we notice issues like "clone parts interfere with audio streams" early, and not when it's already too late. I'd say, few weeks might be too optimistic, but I think we can do this in less than 8 weeks. A fair tradeoff for a shiny, new, well-designed MusE with an intuitive user interface :). > Thus I give you... Undo "Trees". That would be great :) But implementing this is postponed, we have more important and work-intensive stuff to do, okay? Of course, we should already draft its design, so there are no bad surprises later on. > Git, CVS, version control Another great idea, but same thing ;) BTW, What programming languages do all of you speak fluently? There might be languages that are more appropriate for writing GUI code than C++ (?), just a thought. Greetings, and looking forward to a overwhelming MusE 3, Florian |
From: Tim E. R. <ter...@ro...> - 2014-01-24 05:20:52
|
On January 22, 2014 10:37:55 PM Tim E. Real wrote: > > Relax, you are right. > > It has taken me a long time but I realize now that the way MusE handles > processing is wrong. > > To make a long story (which follows) short, we'll do it the right way, > something like this: > > https://github.com/jackaudio/jack2/blob/master/example-clients/capture_clien > t.c > > Rough blueprint: > ============== > > The Jack process callback should do one job only: > Move data into and out of the Jack ports. > > You know all that crap we do in our process callback? > That's gotta be moved outta there into OUR audio thread. No! Man, I hate to flip-flop... but I'm /reversing/ my position regarding our Jack process callback. We CAN'T use another thread to transfer data to the callback. It is *right* the way it is. Really : The example capture_client.c given is naive and too simple: It is half-duplex. So it can gingerly pump as many ring buffer items as it wants and it doesn't care about how long it takes to process them. Also, this example is a disk writer - it NEEDS another thread to do that as doing such things in the RT thread is bad. MusE however is a full-duplex. Waaay different ballgame. It NEEDS to be synchronous: Read some input data from Jack if available, process it, add any playback material for this period, and write this data to Jack outputs - ALL /during/ a single process callback, not /after/ it's done. Problem: If you use your OWN thread to transfer data to the Jack process callback, your thread must wait until the process callback is done before it can have access to the Jack buffers. That introduces one EXTRA period of latency. Yes you /can/ do it, but there's no reason except in special cases. Today I dug deep for answers: There are two Jack APIs for processing: The one we use which is the 'callback' API, and the non-callback API. http://jackaudio.org/files/docs/html/index.html The non-callback API looked intriguing because it gives you full access to the special RT thread which the Jack backend creates. Exactly what I was thinking. See the simple example which uses it: https://github.com/jackaudio/jack2/blob/master/example-clients/tw.c Ardour has been using non-callback API for years. But... in the end the two APIs do exactly the /same/ thing and one is really no better off using one over the other. See this: http://permalink.gmane.org/gmane.comp.audio.jackit/23984 Stéphane Letz 21 Apr 07:40 2011: "... But anyway I don't really see the point of your "clean" version. The " set process calllback" mode was the"historical" mode (classic, easy to understand...), now we also have the "give you thread and use jack_cycle_wait/jack_cycle_signal pair" that give all flexibility needed... " which was in response to a question about the API: http://comments.gmane.org/gmane.comp.audio.jackit/23953 So either API, it still needs to be a real-time ballgame. What do I mean? I found *this* absolute *gem* of a response from Paul in response to a question from somebody trying to do the exact SAME thing I suggested, using your OWN thread as a data pump worker: http://jack-audio.10948.n7.nabble.com/Jack-Devel-Using-the-callback-free-API-with-self-created-RT-threads-td17046.html Paul Davis Sep 11, 2013: "JACK provides a thread primarily in the interest of a uniform cross-platform model. if you try this on OS X or windows, you will find that the APIs there also provide a thread in which your callback will run. what you're doing is adding another layer of buffering between your own RT thread(s) and JACK - this is contrary to the entire purpose and intent of a pull-mode audio API. it adds to your latency and increases the complexity of your code design. realtime audio is always implemented, ultimately, around a pull model (the hardware defines when transfer takes place). if you want to ignore that and somehow imagine that you can do your own scheduling, then you just need to add a ringbuffer between your other threads and the one that runs the callback. the only cases i know where this is important is when running FFT's or other similar algorithms which have significant internal latency due to their window- based model of accumulating a certain amount of data before being able to process. you can always wrap a pull model as a push model simply by adding buffering, but the normal case for doing this is where the audio generation code is running in a non-RT context and needs to be "isolated" from the demands of RT processing. it doesn't appear that your program fits into this category. " See how Paul both *mirrored* and *confirmed* everything I said ? Welcome to real time. You either have enough process time or not. If not, you increase the Jack period size OR buy a faster computer. As I said before, I agree there are some things we can do to speed it up. Another is maybe splitting the work up between different process calls. We can use certain Jack functions to determine how much time is left in the process call before we gotta get outta there. Another is using mutex or trylocks as you say. I noticed even Ardour and some of the Jack examples use it WITHIN the process. I'm not sure. Wouldn't we need to lock our data OUTSIDE of the process? I mentioned this great discussion about locks and real-time: http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing Kinda scared me off mutexes and such. Don't worry too much, we'll find ways to help it. But no getting away from that darn process cycle... Cheers. Tim. |
From: Florian J. <fl...@wi...> - 2014-01-24 12:42:53
Attachments:
signature.asc
|
Am 24.01.2014 06:20, schrieb Tim E. Real: > On January 22, 2014 10:37:55 PM Tim E. Real wrote: >> >> Relax, you are right. >> >> It has taken me a long time but I realize now that the way MusE handles >> processing is wrong. >> >> To make a long story (which follows) short, we'll do it the right way, >> something like this: >> >> https://github.com/jackaudio/jack2/blob/master/example-clients/capture_clien >> t.c >> >> Rough blueprint: >> ============== >> >> The Jack process callback should do one job only: >> Move data into and out of the Jack ports. >> >> You know all that crap we do in our process callback? >> That's gotta be moved outta there into OUR audio thread. > > No! > > Man, I hate to flip-flop... but I'm /reversing/ my position regarding > our Jack process callback. We CAN'T use another thread to > transfer data to the callback. > > It is *right* the way it is. Really : okay, that was the only point i questioned and wanted to come back later ;) Yup, doing stuff in the process callback is actually how things are done. However, I think we do too much in there. Seeking the correct MIDI position (multimap::upper_bound) is not the task of the audio thread, that's the task of some seeking/butler/prefetch thread. It's not like it isn't broken. It's only not broken the way you described it ;) How about multiprocessing? We should really support this. Actually, processing all the plugins could -- if done right! -- be split between the prefetcher and the audio thread. The prefetcher prefetches and precalculates all tracks where no input is routed to, while the audio thread does exactly the same thing for all tracks where input is actually routed to (because it must pipe this input through the plugins as well). > Today I dug deep for answers: > ... > As I said before, I agree there are some things we can do to speed it up. > Another is maybe splitting the work up between different process calls. > We can use certain Jack functions to determine how much time is left > in the process call before we gotta get outta there. Multithreading. > Another is using mutex or trylocks as you say. > I noticed even Ardour and some of the Jack examples use it WITHIN the process. > I'm not sure. Wouldn't we need to lock our data OUTSIDE of the process? We must lock in both threads: In the audio thread (i.e. everything which is called from process() or descendants), and in the GUI thread. Yeah, locking. I'm not sure whether we should not just abandon this message passing thing and just use mutexes. YES, we will *lose* realtime capability *while editing*. Glitches may occur, priority inversion might happen. > I mentioned this great discussion about locks and real-time: > http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing > Kinda scared me off mutexes and such. Yes, mutexes are bad for realtime. But, do we actually need realtime always? I mean, when I'm editing while I'm playing back, then i do not need a 100%-guarantee of glitch-free-ness. Of course, they *should* not happen to often, but *might*, which is not a problem in those situations usually. Hmmm. Tim: How about creating a design document step by step before starting to do anything? Yes? No? Cheers, flo |
From: Florian J. <fl...@wi...> - 2014-01-27 00:06:04
Attachments:
signature.asc
|
i have an idea, stolen from ardour. there are three threads: GUI audio prefetch there is 'the model', containing every information about our song. This is shared among GUI and prefetch thread. audio never touches it. GUI and prefetch do mutex locking when operationg on the model. All data, (also MIDI data), is read from the prefetch thread and written into ringbuffers, to be read and played by the audio thread. the audio thread does effects etc. There are three kinds of operations: fast operations: are executed immediately, with low latency. Implemented by the GUI sending a message over a ringbuffer directly to the audio thread, like we do everything currently. Examples: sliding mixer controls muting/unmuting tracks slow oeprations: might take up to some seconds to apply. Implemented by mutex-locking and editing the model. The prefetcher might wait for a moment, but no problem because our prefetch buffer is large enough. (*) Examples: moving, creating, deleting notes or new wave parts. turning on/off tracks offline operations: cannot be executed while playing back require the audio thread to be msgIdled; used for "large", seldomly Examples: adding/removing (but not bypassing) effects, adding/removing tracks *) in case of a audio buffer underflow, we de-click the thing: The ringbuffer always holds some hundred frames more than it actually needs. If the number of available frames in the ringbuf drops under a few hundred, then a soft underrun occurs and MusE will play back the remaining samples, but fading out (thus, de-clicking everything). If the buffer comes back again, MusE will fade-in for ~100 samples. Also, time-stretching is done inside the audio thread, and not inside the prefetching thread, to be able to quickly adapt new tempi (-> external synchronisation). The prefetcher must prefetch an appropriate number of samples, obviously. (We may assume that the external tempo does not deviate by more than 10% from the tempo map). Work in progress, share your thoughts! A Happy MusE 3.0 Florian |
From: Tim E. R. <ter...@ro...> - 2014-01-31 00:34:21
|
On January 27, 2014 01:05:55 AM Florian Jung wrote: > All data, (also MIDI data), is read from the prefetch thread and written > into ringbuffers, to be read and played by the audio thread. the audio > thread does effects etc. snip > Also, time-stretching is done inside the audio thread, and not inside > the prefetching thread, to be able to quickly adapt new tempi (-> > external synchronisation). The prefetcher must prefetch an appropriate > number of samples, obviously. (We may assume that the external tempo > does not deviate by more than 10% from the tempo map). Yes I figure the ring buffers should store individual samples rather than blocks, so that we can pull as many or little samples as the stretcher needs, from the process thread. Reviewing our prefetch system: In WaveTrack::getData(), which is called in the process thread, it calls _prefetchFifo.get(). Not quite sure yet but I think it already allows us to pull an arbitrary number samples rather than blocks. Which is good. ** Now here's a really IRONIC thing I just found out: Take a look at our Fifo class (node.h, cpp) which is the class used for _prefetchFifo. You see a lot of pthread_mutex_lock etc. These locks were only supposed to be compiled if *NOT* running an i386 arch, otherwise it is supposed to use NO locks at all on an i386 arch. (I guess that answers my question about built-in atomic operations on i386 Linux ? Good !) However, the macro i386 is old I believe, and is NOT defined here. I could swear at one time long ago it was. I believe it was replaced with __i386__ which IS defined here, and __i386__ IS used elsewhere in MusE. So... Here we are worrying about using locks... and ironically locks *are* already being used here, with no noticeable problems, inside one of the most critical sections of our process callback... What are you guys seeing on your machines? Are those sections compiled in or not? Thanks. Tim. |
From: Florian J. <fl...@wi...> - 2014-02-02 13:39:48
Attachments:
signature.asc
|
ooooh, why didn't we think about that one?! Lock-free atomic data structures. Most of the locking/synchronisation fun we have is because we need to insert things into lists. Inserting tracks into the track list, inserting parts into the part list, inserting events into the eventlist. In most situations, this is the only critical action that. Also, we might claim one advantage for us, if neccessary: Usually, threads which *write* to data structures are not time critical. So we do not need to care about two concurrent writes to be safe, we may just mutex-lock here. Only concurrent write and read must be safe and lock-free Lock-free data structures exist! There's not only ringbuffers, but also linked lists that can have atomic operations, I'll read more about this soon. But for our situation, such a list could be written trivially easy: Qt offers us QAtomicPointers which, well, are atomical ;) So our list nodes look like this: struct list_node_t { data_t* pointer_to_data; QAtomicPointer<list_node_t> next; QAtomicInteger refcount; }; Reading from this list works just like linked lists usually work. Inserting will prepare the new node, and only at the end it will (atomically) set the precedessor's "next"-pointer to itself. If concurrent read access happens in the meantime, then it will either see the new node and use it correctly, or it will see the old version of the list. No data structure corruption :) Deleting will also be just one atomic pointer change. The deleted list node may *NOT* be free()d yet, since other threads might still hold that one. But that can be solved, too. That way, as opposed to what i said before, MIDI data does *not* need to be prefetched. Instead, GUI and Audio thread just concurrently access the same data structure. Since all operations on MIDI can be atomically, this will be not a problem at all! (Except for recording. Need to read more about the topic.) I also talked to a developer of the ModPlug Tracker, which is a music tracker software for windows. Trackers are a bit easier, because all data is arranged in a fixed-size "grid". They aren't locking anything. They're just writing and reading data concurrently; everything that would lead to data corruption is done with atomic pointers or integers, and the really tiny rest is actually done by putting audio in IDLE state. I hope that lock-free, atomic data structures could greatly help us :) What do you think? Tim, Robert? Cheers, flo |
From: Florian J. <fl...@wi...> - 2014-02-05 23:08:16
Attachments:
signature.asc
|
without many words: https://github.com/liblfds this plus some code to physically delete all the logically deleted things (thus free()ing memory), which is executed "at appropriate times", i.e. when song is stopped and audio may go into idle mode shortly. this is gonna work out :) say hello to a lock-free MusE 3 Cheers flo |
From: Dennis S. <mus...@wi...> - 2014-01-27 01:02:35
|
Hi Florian, On Mon, 27 Jan 2014 01:05:55 +0100 Florian Jung <fl...@wi...> wrote: > i have an idea, stolen from ardour. > > there are three threads: > GUI > audio > prefetch > > there is 'the model', containing every information about our song. This > is shared among GUI and prefetch thread. audio never touches it. > > GUI and prefetch do mutex locking when operationg on the model. > > All data, (also MIDI data), is read from the prefetch thread and written > into ringbuffers, to be read and played by the audio thread. the audio > thread does effects etc. That sounds like a very reasonable approach. Though I'd try to avoid mutex locks between GUI and prefetch thread, if possible. For me the prefetch thread is not as time critical as "audio" but it has hard deadlines, too in order to avoid buffer underruns. So here is a slightly modified idea in pseudo-code as it's too late for me to describe it in words. :-) I hope you get the idea despite the only vaguely C++ like syntax. public class Model { public: boolean changes_allowed() { return this._changes_allowed }; void start_editing() { this._changes_allowed = False; } void finish_editing() { this._changes_allowed = True; } private: boolean _changes_allowed = True; } Model* model_active = new Model(); // Used by prefetch thread Model* model_edited = new Model(); // Used by GUI thread Mutex* model_swap = new Mutex(); Basically there are two copies of the Model. The prefetch thread uses the object pointed to by "model_active". Changes can only be made by the GUI but it works on another copy pointed to by "model_edited", akin to this: if (!model_edited.changes_allowed()) { return; } Model* _model = model_edited; _model.start_editing(); // Make changes model_swap.acquire_or_wait(); model_edited = model_active; model_active = _model; model_swap.release(); // Send signal to prefetch thread that the pointer changed. // The prefetch thread might need to invalidate its current operation // and start over again. Buffer underrun may also occur but hopefully // less likely because the Model object is not locked while it's // being edited. // Wait until the prefetch thread has picked up the new pointer delete model_edited; model_edited = new Model(model_active); model_edited.finish_editing(); Dennis |
From: Florian J. <fl...@wi...> - 2014-01-27 05:55:02
Attachments:
signature.asc
|
Am 27.01.2014 02:02, schrieb Dennis Schulmeister: > Hi Florian, > > That sounds like a very reasonable approach. Though I'd try to avoid > mutex locks between GUI and prefetch thread, if possible. For me the > prefetch thread is not as time critical as "audio" but it has hard > deadlines, too in order to avoid buffer underruns. > > So here is a slightly modified idea in pseudo-code as it's too late for > me to describe it in words. :-) [you meant having one model in usage and one copy for editing, and swapping their pointers after editing since swapping is a 'fast' operation] Thanks for your feedback :)! Hm, yeah, that would be an option. Though duplicating the whole model is a bit much, I think. Note that we would need to copy all sorts of audio structures as well, which can be quite some amount of data. I've thought about (and ardour does it :)) using a Diff system: When the GUI wants to edit stuff, it creates a Diff (which is not applied yet), which is ready to apply, with everything prepared. At the end: mutex.lock(); mydiff.apply(); // should work reasonably fast mutex.unlock(); Since the audio prefetch prefetches at least 0.5 seconds (doesn't it?), and human UI latency perception is definitely too high at 0.5 seconds, I guess that we will be fine with this: Because the operating system will give the GUI thread enough CPU time to complete and unlock the mutex within, say, 0.4 seconds, which is enough to not buffer-underrun. Cheers, Florian |
From: Dennis S. <mus...@wi...> - 2014-01-27 22:04:53
|
On Mon, 27 Jan 2014 06:54:49 +0100 Florian Jung <fl...@wi...> wrote: > Am 27.01.2014 02:02, schrieb Dennis Schulmeister: > > Hi Florian, > > So here is a slightly modified idea in pseudo-code as it's too late for > > me to describe it in words. :-) > > [you meant having one model in usage and one copy for editing, and > swapping their pointers after editing since swapping is a 'fast' operation] Uhm yes. I knew it was easy ... > I've thought about (and ardour does it :)) using a Diff system: So the model contains a list of changes filled by the GUI. During the lock phase all changes are applied at once. If most time is spent waiting on the user to complete his actions, this should work. But it will complicate things due to the indirect nature of making changes to the model. The interface needs to be well thought about, before-hand. Dennis |
From: Florian J. <fl...@wi...> - 2014-01-28 00:23:09
Attachments:
signature.asc
|
Am 27.01.2014 23:04, schrieb Dennis Schulmeister: > On Mon, 27 Jan 2014 06:54:49 +0100 > Florian Jung <fl...@wi...> wrote: > >> Am 27.01.2014 02:02, schrieb Dennis Schulmeister: >>> Hi Florian, > >>> So here is a slightly modified idea in pseudo-code as it's too late for >>> me to describe it in words. :-) >> >> [you meant having one model in usage and one copy for editing, and >> swapping their pointers after editing since swapping is a 'fast' operation] > > Uhm yes. I knew it was easy ... > >> I've thought about (and ardour does it :)) using a Diff system: > > So the model contains a list of changes filled by the GUI. During the > lock phase all changes are applied at once. If most time is spent > waiting on the user to complete his actions, this should work. But it > will complicate things due to the indirect nature of making changes to > the model. The interface needs to be well thought about, before-hand. it will not complicate things more than things are complicated anyway: we already must implement diffs or a similar concept for Undo/Redo, so we can use it for this as well. Just keeping copies of the whole model for every single undo operation is not a good idea ;) Or have you an alternative which allows us to undo more cheaply? Cheers flo |
From: Florian J. <fl...@wi...> - 2014-01-28 09:48:48
Attachments:
signature.asc
|
Am 28.01.2014 04:32, schrieb Tim E. Real: > On January 27, 2014 11:04:44 PM Dennis Schulmeister wrote: >> On Mon, 27 Jan 2014 06:54:49 +0100 >> >> Florian Jung <fl...@wi...> wrote: > >> i have an idea, stolen from ardour. >> >> there are three threads: >> GUI >> audio >> prefetch >> >> there is 'the model', containing every information about our song. This >> is shared among GUI and prefetch thread. audio never touches it. >> >> GUI and prefetch do mutex locking when operationg on the model. >> >> All data, (also MIDI data), is read from the prefetch thread and written >> into ringbuffers, to be read and played by the audio thread. the audio >> thread does effects etc. > > What do you mean all data? > A prefetch and ring buffer are only for playing/recording disk material, > such as wave files. You can't 'prefetch' audio from Jack inputs. I meant "all disk data, *including* MIDI data", sorry. > During our Jack process callback, the thing reaches waaay down > into the routing system, which *may* include Audio Input tracks, > starting from Audio Output tracks and working its way back from there, > and gathers up all the data to be written into the Jack outputs - > immediately. Doing that in another thread is useless and adds latency. > I just explained that. > Unless ... > While in the Jack process callback, once you have read the input data > from Jack input ports, you could *immediately* pass control to some > other thread, and make the Jack process callback wait for that thread, > and if it does not respond in enough /time/, finish the Jack process callback > by simply writing zeros to all Jack output ports. But how? Naah. I considered this for multi threading, but I don't think it'll work. > >> There are three kinds of operations: >> >> fast operations: are executed immediately, with low latency. >> Implemented by the GUI sending a message over a ringbuffer directly >> to the audio thread, like we do everything currently. > > Not quite. We currently wait for the audio thread. > >> Examples: sliding mixer controls >> muting/unmuting tracks > > This is exactly what I said I wanted to do. > > Install a ring buffer so that operations like these do not wait for the audio > messaging. These only use operations 'stage 2' realtime and not the > non-realtime stage 1 or 3, which would be harder to work with. > > What I meant about careful timing, mouse press inhibiting etc: > > Currently moving a midi slider or knob sends a message and waits, > guaranteeing synchronization - that the GUI continues only after all of the > work has been done. The next time the heartbeat routines update the > position of that control, its value is /already/ current. > So there is no jitter moving the control. (Just jerky with big Jack periods.) > However, if you use a ring buffer to decouple the GUI movements from audio, > the GUI control is now /free/ to move at any speed it wants but the actual > underlying controller values may not be updated until some time later > when the audio thread has processed the messages. > Thus the heartbeat routine comes along and attempts to update the control > with a /stale/ previous value which has not been updated yet by audio. > Thus you get horrible control jitter as it flips back and forth. I'd actually really put GUI movements directly into the ringbuffer, without waiting. All data will apply in the next audio period, with the same amount of latency a mapped MIDI-controls-automation would have. About updating with stale values: Just don't update controls that are currently held down, and we're fine. > > So what I did with all audio controls was: > a) Installed a ring buffer, decoupling the GUI from audio. > b) While such controls are being pressed down with the mouse, I inhibit > /any/ updating of the control from the heartbeat routines (that includes > any audio automation streams).[...] Cool! Seems like we already have the "Mixer controls ringbuffer" thing I wanted :) > > There is another detail: Things like midi controls use the heartbeat function > to periodically update their values, but things like controller graphs rely > on songChanged signals to redraw themselves. > Currently we signal songChanged from 'operations' routines in GUI thread. > But when using a ring buffer, we can't signal a songChanged until the audio > thread has finished its work processing the ring buffer. So we need to > signal the GUI thread right from inside the audio thread. We have such > an 'audio-to-GUI' message pipe mechanism as I mentioned, but it sends > only single characters. We would need to modify it, if that's what we use. I would consider controller graphs (as opposed to mixer controls) as a "slow operation": i.e. there is no need for "fast" synchro via the ringbuffers we just mentioned. > >> >> slow oeprations: might take up to some seconds to apply. >> Implemented by mutex-locking and editing the model. The prefetcher >> might wait for a moment, but no problem because our prefetch buffer >> is large enough. (*) >> >> Examples: moving, creating, deleting notes or new wave parts. >> turning on/off tracks >> >> offline operations: cannot be executed while playing back >> require the audio thread to be msgIdled; used for "large", seldomly >> Examples: adding/removing (but not bypassing) effects, >> adding/removing tracks >> >> >> *) in case of a audio buffer underflow, we de-click the thing: >> The ringbuffer always holds some hundred frames more than it actually >> needs. If the number of available frames in the ringbuf drops under a >> few hundred, then a soft underrun occurs and MusE will play back the >> remaining samples, but fading out (thus, de-clicking everything). >> If the buffer comes back again, MusE will fade-in for ~100 samples. >> >> >> >> Also, time-stretching is done inside the audio thread, and not inside >> the prefetching thread, to be able to quickly adapt new tempi (-> >> external synchronisation). The prefetcher must prefetch an appropriate >> number of samples, obviously. (We may assume that the external tempo >> does not deviate by more than 10% from the tempo map). > > >>> Am 27.01.2014 02:02, schrieb Dennis Schulmeister: >>>> Hi Florian, >>>> >>>> So here is a slightly modified idea in pseudo-code as it's too late for >>>> me to describe it in words. :-) >>> >>> [you meant having one model in usage and one copy for editing, and >>> swapping their pointers after editing since swapping is a 'fast' >>> operation] >> >> Uhm yes. I knew it was easy ... >> >>> I've thought about (and ardour does it :)) using a Diff system: > > Again that's like what I said, albeit by way of a VCS system, also for song > files as well, so we can use complete undo trees. > But what do you mean diff system? Not an actual text based diff file I hope. > This system needs to operate on data structures. > Which... we already have - the undo/redo stacks. > Maybe not how they are currently used, but the actual stacks themselves. > I'd like to turn them into trees. yes. I actually repeated you, trying to get everything we want in one text, not cluttered about a whole discussion ;) > >> So the model contains a list of changes filled by the GUI. During the >> lock phase all changes are applied at once. If most time is spent >> waiting on the user to complete his actions, this should work. But it >> will complicate things due to the indirect nature of making changes to >> the model. The interface needs to be well thought about, before-hand. >> >> Dennis > > Yes of course, if we use locks, then gather up all changes that we possibly > can and then apply them in one quick burst inside a lock. > Just... I don't know if that's as quickly done as said. Depends on your view of "quick". It will not be as quick as we would need for realtime operations. It would be too slow if the *audio* thread would wait for this lock. But the audio thread doesn't. The *prefetch* thread waits for this lock, and thus, we may take up to half as long as our prefetch queue is. Locking this mutex and "quickly" applying the diff will be WAY more "quick" than reading audio data from disk, so we're safe. > Actually this 'gathering' phase is pretty much what we already have with the > operations/undo/redo stack system. But now it is a question of /when/ to > apply them. That's a bit too detailed for what I tried to describe. But I'd apply them roughly the way operation groups are applied currently. Only difference to the Undo system will be: Each UndoOp is a class, inheriting from UndoOpBase, and will implement the virtual functions apply() and undo(). That's no conceptual change, but will lead to nicer code. BTW, i'm trying to write a document about how I imagine MusE3. I'll put it on github later, i'd be glad if you would read it. Seems like we have basically the same thoughts, right? Cheers, flo > > Thanks. > Tim. > > ------------------------------------------------------------------------------ > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you real-time visual feedback on key > security issues and trends. Skip the complicated setup - simply import > a virtual appliance and go from zero to informed in seconds. > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Lmuse-developer mailing list > Lmu...@li... > https://lists.sourceforge.net/lists/listinfo/lmuse-developer > |
From: Florian J. <fl...@wi...> - 2014-01-28 21:39:11
Attachments:
signature.asc
|
Am 28.01.2014 22:27, schrieb Dennis Schulmeister: > On Tue, 28 Jan 2014 10:48:38 +0100 > Florian Jung <fl...@wi...> wrote: > >>> Yes of course, if we use locks, then gather up all changes that we possibly >>> can and then apply them in one quick burst inside a lock. >>> Just... I don't know if that's as quickly done as said. >> >> Depends on your view of "quick". >> >> It will not be as quick as we would need for realtime operations. It >> would be too slow if the *audio* thread would wait for this lock. >> >> But the audio thread doesn't. The *prefetch* thread waits for this lock, >> and thus, we may take up to half as long as our prefetch queue is. > > Are you sure about this? As long as the model is in inconsistent state > that audio thread can't reliably access it. But then, acquiring a mutex > in the audio thread will be a sure receipt for dropouts. The audio thread is never accessing the model. Never. The audio thread does never lock any mutexes. Instead, the prefetch thread accesses the model, and locks on mutexes. The prefetch thread only fills the ringbuffers that are consumed by the audio thread. But these ringbuffers always hold more data than the audio thread will actually consume in the next frame. That's why it's not a problem when the prefetch thread has to wait on a mutex. The number of remaining frames in the ringbuffer will fall down, but because the ringbuffer is chosen large enough, there will be no dropouts. After the prefetch thread has acquired the lock, it can re-fill the ring buffer with enough data. This concept actually works, ardour is doing it like that. And consider that locking on mutexes is *way* less problematic than waiting for disk access. And have you had any disk-access-related dropout in muse yet? > > I see the point with undo/redo. And as Tim points out, that it already > provides the infrastructure for a "list of changes to be applied" on > the model. But even with this list I think there should be a "working > copy" of the model objects. If a pointer swap can't be atomic one could > pass the new model pointer to the audio thread through the ring buffer. Keeping a working copy around is a waste of resources. And because the audio thread never touches the model, as noted above, we have absolutely no problem to simply do mutex locking and editing the one working copy we have. Consider that with multiple working copies, we would need to synchronise all audio streams (you cannot random-access a .OGG file), which would massively complicate things (i even think that it's impossible.) Cheers, flo > > Dennis > > ------------------------------------------------------------------------------ > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you real-time visual feedback on key > security issues and trends. Skip the complicated setup - simply import > a virtual appliance and go from zero to informed in seconds. > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Lmuse-developer mailing list > Lmu...@li... > https://lists.sourceforge.net/lists/listinfo/lmuse-developer > |
From: Dennis S. <mus...@wi...> - 2014-01-28 22:13:39
|
On Tue, 28 Jan 2014 22:39:02 +0100 Florian Jung <fl...@wi...> wrote: > The audio thread is never accessing the model. Never. > The audio thread does never lock any mutexes. Ok, that makes sense, of course. :) Dennis |
From: Tim E. R. <ter...@ro...> - 2014-01-28 03:32:53
|
On January 27, 2014 11:04:44 PM Dennis Schulmeister wrote: > On Mon, 27 Jan 2014 06:54:49 +0100 > > Florian Jung <fl...@wi...> wrote: > i have an idea, stolen from ardour. > > there are three threads: > GUI > audio > prefetch > > there is 'the model', containing every information about our song. This > is shared among GUI and prefetch thread. audio never touches it. > > GUI and prefetch do mutex locking when operationg on the model. > > All data, (also MIDI data), is read from the prefetch thread and written > into ringbuffers, to be read and played by the audio thread. the audio > thread does effects etc. What do you mean all data? A prefetch and ring buffer are only for playing/recording disk material, such as wave files. You can't 'prefetch' audio from Jack inputs. During our Jack process callback, the thing reaches waaay down into the routing system, which *may* include Audio Input tracks, starting from Audio Output tracks and working its way back from there, and gathers up all the data to be written into the Jack outputs - immediately. Doing that in another thread is useless and adds latency. I just explained that. Unless ... While in the Jack process callback, once you have read the input data from Jack input ports, you could *immediately* pass control to some other thread, and make the Jack process callback wait for that thread, and if it does not respond in enough /time/, finish the Jack process callback by simply writing zeros to all Jack output ports. But how? The purpose of a system like this as I mentioned, outside of process callback, would be to avoid taking too long in the process callback itself. If we take too long there, we get a glitch /and/ an xrun. Conversely, with the system described above, if we take too long in some other thread but are able to return soon enough from it so that the Jack process callback keeps going and writes muting zeros to all outputs, we get a glitch but /no/ xruns. That's the only difference. > There are three kinds of operations: > > fast operations: are executed immediately, with low latency. > Implemented by the GUI sending a message over a ringbuffer directly > to the audio thread, like we do everything currently. Not quite. We currently wait for the audio thread. > Examples: sliding mixer controls > muting/unmuting tracks This is exactly what I said I wanted to do. Install a ring buffer so that operations like these do not wait for the audio messaging. These only use operations 'stage 2' realtime and not the non-realtime stage 1 or 3, which would be harder to work with. What I meant about careful timing, mouse press inhibiting etc: Currently moving a midi slider or knob sends a message and waits, guaranteeing synchronization - that the GUI continues only after all of the work has been done. The next time the heartbeat routines update the position of that control, its value is /already/ current. So there is no jitter moving the control. (Just jerky with big Jack periods.) However, if you use a ring buffer to decouple the GUI movements from audio, the GUI control is now /free/ to move at any speed it wants but the actual underlying controller values may not be updated until some time later when the audio thread has processed the messages. Thus the heartbeat routine comes along and attempts to update the control with a /stale/ previous value which has not been updated yet by audio. Thus you get horrible control jitter as it flips back and forth. So what I did with all audio controls was: a) Installed a ring buffer, decoupling the GUI from audio. b) While such controls are being pressed down with the mouse, I inhibit /any/ updating of the control from the heartbeat routines (that includes any audio automation streams). Eliminating movement jitter. When the mouse is released, theoretically the shown value should match the next update, well soon anyway, unless automation streams then take over. To fix more elaborate situations, with the midi controller graphs for example (probably the best worst-case example to use - they also wait), the graphs currently hold their own data as 'drawing items'. By pushing to a ring buffer the mouse would be free to move as fast as it wants and the graph is free to be drawn at the same speed, but the underlying midi controller data actually changes later when the audio thread does it. There is another detail: Things like midi controls use the heartbeat function to periodically update their values, but things like controller graphs rely on songChanged signals to redraw themselves. Currently we signal songChanged from 'operations' routines in GUI thread. But when using a ring buffer, we can't signal a songChanged until the audio thread has finished its work processing the ring buffer. So we need to signal the GUI thread right from inside the audio thread. We have such an 'audio-to-GUI' message pipe mechanism as I mentioned, but it sends only single characters. We would need to modify it, if that's what we use. > > slow oeprations: might take up to some seconds to apply. > Implemented by mutex-locking and editing the model. The prefetcher > might wait for a moment, but no problem because our prefetch buffer > is large enough. (*) > > Examples: moving, creating, deleting notes or new wave parts. > turning on/off tracks > > offline operations: cannot be executed while playing back > require the audio thread to be msgIdled; used for "large", seldomly > Examples: adding/removing (but not bypassing) effects, > adding/removing tracks > > > *) in case of a audio buffer underflow, we de-click the thing: > The ringbuffer always holds some hundred frames more than it actually > needs. If the number of available frames in the ringbuf drops under a > few hundred, then a soft underrun occurs and MusE will play back the > remaining samples, but fading out (thus, de-clicking everything). > If the buffer comes back again, MusE will fade-in for ~100 samples. > > > > Also, time-stretching is done inside the audio thread, and not inside > the prefetching thread, to be able to quickly adapt new tempi (-> > external synchronisation). The prefetcher must prefetch an appropriate > number of samples, obviously. (We may assume that the external tempo > does not deviate by more than 10% from the tempo map). > > Am 27.01.2014 02:02, schrieb Dennis Schulmeister: > > > Hi Florian, > > > > > > So here is a slightly modified idea in pseudo-code as it's too late for > > > me to describe it in words. :-) > > > > [you meant having one model in usage and one copy for editing, and > > swapping their pointers after editing since swapping is a 'fast' > > operation] > > Uhm yes. I knew it was easy ... > > > I've thought about (and ardour does it :)) using a Diff system: Again that's like what I said, albeit by way of a VCS system, also for song files as well, so we can use complete undo trees. But what do you mean diff system? Not an actual text based diff file I hope. This system needs to operate on data structures. Which... we already have - the undo/redo stacks. Maybe not how they are currently used, but the actual stacks themselves. I'd like to turn them into trees. > So the model contains a list of changes filled by the GUI. During the > lock phase all changes are applied at once. If most time is spent > waiting on the user to complete his actions, this should work. But it > will complicate things due to the indirect nature of making changes to > the model. The interface needs to be well thought about, before-hand. > > Dennis Yes of course, if we use locks, then gather up all changes that we possibly can and then apply them in one quick burst inside a lock. Just... I don't know if that's as quickly done as said. Actually this 'gathering' phase is pretty much what we already have with the operations/undo/redo stack system. But now it is a question of /when/ to apply them. Thanks. Tim. |
From: Dennis S. <mus...@wi...> - 2014-01-28 21:27:27
|
On Tue, 28 Jan 2014 10:48:38 +0100 Florian Jung <fl...@wi...> wrote: > > Yes of course, if we use locks, then gather up all changes that we possibly > > can and then apply them in one quick burst inside a lock. > > Just... I don't know if that's as quickly done as said. > > Depends on your view of "quick". > > It will not be as quick as we would need for realtime operations. It > would be too slow if the *audio* thread would wait for this lock. > > But the audio thread doesn't. The *prefetch* thread waits for this lock, > and thus, we may take up to half as long as our prefetch queue is. Are you sure about this? As long as the model is in inconsistent state that audio thread can't reliably access it. But then, acquiring a mutex in the audio thread will be a sure receipt for dropouts. I see the point with undo/redo. And as Tim points out, that it already provides the infrastructure for a "list of changes to be applied" on the model. But even with this list I think there should be a "working copy" of the model objects. If a pointer swap can't be atomic one could pass the new model pointer to the audio thread through the ring buffer. Dennis |
From: Florian J. <fl...@wi...> - 2014-01-31 12:28:59
Attachments:
signature.asc
|
Hm. Haven't i already replied to this? I thought so, but the mail is not in my Sent folder, so nevermind, if you already have a reply ;) Am 31.01.2014 01:34, schrieb Tim E. Real: > On January 27, 2014 01:05:55 AM Florian Jung wrote: >> All data, (also MIDI data), is read from the prefetch thread and written >> into ringbuffers, to be read and played by the audio thread. the audio >> thread does effects etc. > > snip > >> Also, time-stretching is done inside the audio thread, and not inside >> the prefetching thread, to be able to quickly adapt new tempi (-> >> external synchronisation). The prefetcher must prefetch an appropriate >> number of samples, obviously. (We may assume that the external tempo >> does not deviate by more than 10% from the tempo map). > > Yes I figure the ring buffers should store individual samples rather > than blocks, so that we can pull as many or little samples as the > stretcher needs, from the process thread. > Reviewing our prefetch system: > > In WaveTrack::getData(), which is called in the process thread, it calls > _prefetchFifo.get(). > > Not quite sure yet but I think it already allows us to pull an arbitrary > number samples rather than blocks. > Which is good. indeed :) > > ** Now here's a really IRONIC thing I just found out: > > Take a look at our Fifo class (node.h, cpp) which is the class used for > _prefetchFifo. You see a lot of pthread_mutex_lock etc. > > These locks were only supposed to be compiled if *NOT* running an > i386 arch, otherwise it is supposed to use NO locks at all on an i386 arch. > > (I guess that answers my question about built-in atomic operations > on i386 Linux ? Good !) > > However, the macro i386 is old I believe, and is NOT defined here. > I could swear at one time long ago it was. > I believe it was replaced with __i386__ which IS defined here, and > __i386__ IS used elsewhere in MusE. > > So... Here we are worrying about using locks... and ironically locks > *are* already being used here, with no noticeable problems, > inside one of the most critical sections of our process callback... Please don't take this as an excuse for sloppiness. If we want to create a live-capable application, we have to do it properly. If we want to create a studio-only application which might trash the one or other recording, then fine. (I'd probably be okay with that one, too) > > What are you guys seeing on your machines? Are those sections > compiled in or not? Not compiled in, but irrelevant because i have insanely high buffer sizes usually. Cheers, flo > > Thanks. > Tim. > > ------------------------------------------------------------------------------ > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you real-time visual feedback on key > security issues and trends. Skip the complicated setup - simply import > a virtual appliance and go from zero to informed in seconds. > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk > _______________________________________________ > Lmuse-developer mailing list > Lmu...@li... > https://lists.sourceforge.net/lists/listinfo/lmuse-developer > |