From: Tito L. <tit...@gm...> - 2013-07-01 10:42:11
|
Hi, I'm writing a music/dsp programming environment in Common Lisp called Incudine (http://incudine.sourceforge.net). Currently it works only with SBCL and the performance is really good. Thank you for the magic! Incudine uses different memory pools and the format for the sample is DOUBLE-FLOAT by default, unboxed type on the C heap (an example of the expansion of DEFSYNTH is here: http://incudine.sf.net/tutorial_04.html). Therefore there is not garbage during the synthesis in realtime of a well definited synth. However there is an inevitable noise if a synth is compiled while something is playing in realtime. I'm thinking about the possibility to use two instances of SBCL, one dedicated to the compilation: - SBCL-A sends to SBCL-B all the stuff to compile (with a possible cons-pool) The priority of SBCL-B is lower than SBCL-A. - SBCL-A loads the fasl (without to use the disk if possible) compiled by SBCL-B (is it considerable for gc?). The latency for non-synthdef things is probably not big. - All the standard-output is redirected to SBCL-B Besides, two instances of SBCL require a thorough distribution of the resources. To complicate the code is not my preferred sport but the end justifies the means. However, the possibility to stop-the-world is lower but yet possible. The alternative is to use another implementation of CL when the performance is not so important. To use more cores for the synthesis could help in this direction. I would like to know your advice about this problem. Tito Latini PS: about 1.1.9, tested 1656e54 without problems on x86_64 |
From: Faré <fa...@gm...> - 2013-07-01 14:16:23
|
You can use tmpfs not to hit the disk. But yes, being able to load a fasl from stream would be nice. As for avoiding GC, unwanted compile-time side-effects, etc., you might want to (pre)fork your compilation server after loading your common core. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. On Mon, Jul 1, 2013 at 6:41 AM, Tito Latini <tit...@gm...> wrote: > Hi, > > I'm writing a music/dsp programming environment in Common Lisp called > Incudine (http://incudine.sourceforge.net). Currently it works only > with SBCL and the performance is really good. Thank you for the magic! > > Incudine uses different memory pools and the format for the sample is > DOUBLE-FLOAT by default, unboxed type on the C heap (an example of the > expansion of DEFSYNTH is here: http://incudine.sf.net/tutorial_04.html). > Therefore there is not garbage during the synthesis in realtime of a > well definited synth. However there is an inevitable noise if a synth > is compiled while something is playing in realtime. > > I'm thinking about the possibility to use two instances of SBCL, one > dedicated to the compilation: > > - SBCL-A sends to SBCL-B all the stuff to compile > (with a possible cons-pool) > The priority of SBCL-B is lower than SBCL-A. > > - SBCL-A loads the fasl (without to use the disk if possible) > compiled by SBCL-B (is it considerable for gc?). The latency > for non-synthdef things is probably not big. > > - All the standard-output is redirected to SBCL-B > > Besides, two instances of SBCL require a thorough distribution of the > resources. To complicate the code is not my preferred sport but the > end justifies the means. However, the possibility to stop-the-world is > lower but yet possible. > > The alternative is to use another implementation of CL when the > performance is not so important. To use more cores for the synthesis > could help in this direction. > > I would like to know your advice about this problem. > > Tito Latini > > PS: about 1.1.9, tested 1656e54 without problems on x86_64 > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > _______________________________________________ > Sbcl-devel mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-devel > |
From: Tito L. <tit...@gm...> - 2013-07-01 15:01:21
|
thank you for the answer On Mon, Jul 01, 2013 at 10:15:46AM -0400, Faré wrote: > You can use tmpfs not to hit the disk. but the portability is gone > But yes, being able to load a fasl from stream would be nice. > > As for avoiding GC, unwanted compile-time side-effects, etc., > you might want to (pre)fork your compilation server > after loading your common core. in this case the value of `--dynamic-space-size' is the same for both the instances |
From: Martin C. <cra...@co...> - 2013-07-01 15:59:00
|
Tito Latini wrote on Mon, Jul 01, 2013 at 12:41:10PM +0200: > Hi, > > I'm writing a music/dsp programming environment in Common Lisp called > Incudine (http://incudine.sourceforge.net). Currently it works only > with SBCL and the performance is really good. Thank you for the magic! > > Incudine uses different memory pools and the format for the sample is > DOUBLE-FLOAT by default, unboxed type on the C heap (an example of the > expansion of DEFSYNTH is here: http://incudine.sf.net/tutorial_04.html). > Therefore there is not garbage during the synthesis in realtime of a > well definited synth. However there is an inevitable noise if a synth > is compiled while something is playing in realtime. What kind of noise? Do you mean pauses? What kind of compilation, anyway? Martin > I'm thinking about the possibility to use two instances of SBCL, one > dedicated to the compilation: > > - SBCL-A sends to SBCL-B all the stuff to compile > (with a possible cons-pool) > The priority of SBCL-B is lower than SBCL-A. > > - SBCL-A loads the fasl (without to use the disk if possible) > compiled by SBCL-B (is it considerable for gc?). The latency > for non-synthdef things is probably not big. > > - All the standard-output is redirected to SBCL-B > > Besides, two instances of SBCL require a thorough distribution of the > resources. To complicate the code is not my preferred sport but the > end justifies the means. However, the possibility to stop-the-world is > lower but yet possible. > > The alternative is to use another implementation of CL when the > performance is not so important. To use more cores for the synthesis > could help in this direction. > > I would like to know your advice about this problem. > > Tito Latini > > PS: about 1.1.9, tested 1656e54 without problems on x86_64 > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > _______________________________________________ > Sbcl-devel mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-devel -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer <cra...@co...> http://www.cons.org/cracauer/ |
From: Tito L. <tit...@gm...> - 2013-07-01 17:02:24
|
Martin Cracauer wrote: > What kind of noise? Do you mean pauses? There is a C thread for the audio driver without a lisp callback called from C. The C thread and the realtime lisp thread are synchronized (futex works fine on linux). Lisp fills the output buffer, sample by sample, but if the gc stops the rt thread, the C side continues to write the content of the buffer on the soundcard. The result is a fast repetition of the same chunk of samples. Generally it is not a problem during the exploration of the sounds, but it is not good for livecoding in a concert or to change on-the-fly the rules of a musical installation. > What kind of compilation, anyway? All or quasi. tito |
From: Martin C. <cra...@co...> - 2013-07-01 17:35:39
|
Tito Latini wrote on Mon, Jul 01, 2013 at 07:01:21PM +0200: > Martin Cracauer wrote: > > What kind of noise? Do you mean pauses? > > There is a C thread for the audio driver without a lisp callback > called from C. The C thread and the realtime lisp thread are > synchronized (futex works fine on linux). Lisp fills the output > buffer, sample by sample, but if the gc stops the rt thread, the C > side continues to write the content of the buffer on the soundcard. > The result is a fast repetition of the same chunk of samples. > Generally it is not a problem during the exploration of the sounds, > but it is not good for livecoding in a concert or to change on-the-fly > the rules of a musical installation. So your problem is the garbage produced by the compiler, which stops the world, that world being the one that should have continued to shove data into the audio buffer? > > What kind of compilation, anyway? > > All or quasi. Why are you compiling in the same process? Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer <cra...@co...> http://www.cons.org/cracauer/ |
From: Tito L. <tit...@gm...> - 2013-07-01 19:24:02
|
Martin Cracauer wrote: > Tito Latini wrote on Mon, Jul 01, 2013 at 07:01:21PM +0200: > > Martin Cracauer wrote: > > > What kind of noise? Do you mean pauses? > > > > There is a C thread for the audio driver without a lisp callback > > called from C. The C thread and the realtime lisp thread are > > synchronized (futex works fine on linux). Lisp fills the output > > buffer, sample by sample, but if the gc stops the rt thread, the C > > side continues to write the content of the buffer on the soundcard. > > The result is a fast repetition of the same chunk of samples. > > Generally it is not a problem during the exploration of the sounds, > > but it is not good for livecoding in a concert or to change on-the-fly > > the rules of a musical installation. > > So your problem is the garbage produced by the compiler, which stops > the world, that world being the one that should have continued to > shove data into the audio buffer? yes > > > What kind of compilation, anyway? > > > > All or quasi. > > Why are you compiling in the same process? The generation of the sound from scratch, with a realtime feedback, defining virtual ugens (VUG; yes, the name is inspired by VOP) and compiling on-the-fly the combination of multiple VUGs is an invaluable feature. I'm just curious to know if this problem has been resolved in other contexts or if there are specific strategies in SBCL to facilitate the isolation of the compilation. Tito Latini |
From: James M. L. <llm...@gm...> - 2013-07-01 19:34:16
|
There is a tool in quicklisp which _might_ be useful, https://github.com/lmj/lfarm (warning: self-promotion). ;;; in the dedicated compiler process (ql:quickload :lfarm-server) (lfarm-server:start-server "127.0.0.1" 11111 :background t) ;;; in the main lisp process (ql:quickload :lfarm-client) (setf lfarm:*kernel* (lfarm:make-kernel '(("127.0.0.1" 11111)))) (defun compile-remotely (code) (let ((channel (lfarm:make-channel))) (lfarm:submit-task channel (lambda () ;; TODO: generate pathname and delete later (alexandria:with-output-to-file (out "tmp.lisp" :if-does-not-exist :create) (with-standard-io-syntax (prin1 code out))) (compile-file "tmp.lisp"))) ;; TODO: delete fasl (load (lfarm:receive-result channel)))) (compile-remotely '(defun foo () 99)) ;=> T (foo) ;=> 99 There are some tricks to avoid consing (reusing buffers, sending a task name instead of a lambda, defining a custom protocol), but these may not be significant in comparison to compilation, which is now gone. If not consing is an absolute imperative then you probably expected to write everything from the ground up anyway. As Fare mentioned it would be nice if we could compile/load to/from arbitrary streams instead of going through files. Perhaps it could be offered as an extension, say, COMPILE-SEQUENCE and LOAD-SEQUENCE. On Mon, Jul 1, 2013 at 6:41 AM, Tito Latini <tit...@gm...> wrote: > Hi, > > I'm writing a music/dsp programming environment in Common Lisp called > Incudine (http://incudine.sourceforge.net). Currently it works only > with SBCL and the performance is really good. Thank you for the magic! > > Incudine uses different memory pools and the format for the sample is > DOUBLE-FLOAT by default, unboxed type on the C heap (an example of the > expansion of DEFSYNTH is here: http://incudine.sf.net/tutorial_04.html). > Therefore there is not garbage during the synthesis in realtime of a > well definited synth. However there is an inevitable noise if a synth > is compiled while something is playing in realtime. > > I'm thinking about the possibility to use two instances of SBCL, one > dedicated to the compilation: > > - SBCL-A sends to SBCL-B all the stuff to compile > (with a possible cons-pool) > The priority of SBCL-B is lower than SBCL-A. > > - SBCL-A loads the fasl (without to use the disk if possible) > compiled by SBCL-B (is it considerable for gc?). The latency > for non-synthdef things is probably not big. > > - All the standard-output is redirected to SBCL-B > > Besides, two instances of SBCL require a thorough distribution of the > resources. To complicate the code is not my preferred sport but the > end justifies the means. However, the possibility to stop-the-world is > lower but yet possible. > > The alternative is to use another implementation of CL when the > performance is not so important. To use more cores for the synthesis > could help in this direction. > > I would like to know your advice about this problem. > > Tito Latini > > PS: about 1.1.9, tested 1656e54 without problems on x86_64 > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > _______________________________________________ > Sbcl-devel mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-devel |
From: Tito L. <tit...@gm...> - 2013-07-01 20:00:19
|
On Mon, Jul 01, 2013 at 03:34:07PM -0400, James M. Lawrence wrote: > There is a tool in quicklisp which _might_ be useful, > https://github.com/lmj/lfarm (warning: self-promotion). Thank you, looks interesting, I'll give it a glance. |
From: Luís O. <lu...@gm...> - 2013-07-01 23:21:20
|
On Mon, Jul 1, 2013 at 8:34 PM, James M. Lawrence <llm...@gm...> wrote: > As Fare mentioned it would be nice if we could compile/load to/from > arbitrary streams instead of going through files. Perhaps it could be > offered as an extension, say, COMPILE-SEQUENCE and LOAD-SEQUENCE. As a side note, this would also be useful for SLIME which is also forced to go through files for things like slime-compile-defun and similar. -- Luís Oliveira http://kerno.org/~luis/ |
From: Martin C. <cra...@co...> - 2013-07-01 20:20:57
|
Tito Latini wrote on Mon, Jul 01, 2013 at 09:23:00PM +0200: [...] > > So your problem is the garbage produced by the compiler, which stops > > the world, that world being the one that should have continued to > > shove data into the audio buffer? > > yes > > > > > What kind of compilation, anyway? > > > > > > All or quasi. > > > > Why are you compiling in the same process? > > The generation of the sound from scratch, with a realtime feedback, > defining virtual ugens (VUG; yes, the name is inspired by VOP) and > compiling on-the-fly the combination of multiple VUGs is an invaluable > feature. > > I'm just curious to know if this problem has been resolved in other > contexts or if there are specific strategies in SBCL to facilitate the > isolation of the compilation. As said, forking, compiling in the child and then loading the fasl in the parent, is an option. Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer <cra...@co...> http://www.cons.org/cracauer/ |
From: Tito L. <tit...@gm...> - 2013-07-02 06:42:52
|
Martin Cracauer wrote: > As said, forking, compiling in the child and then loading the fasl in > the parent, is an option. ok, useful to know there aren't esoteric techniques to resolve this problem. I'll weigh the sacrifice of the portability and the size of the dynamic space in the choice of the strategy. To avoid the use of the disk will be a priority. Many thanks to all for the feedback. Tito Latini |
From: James M. L. <llm...@gm...> - 2013-07-02 14:25:33
|
On Tue, Jul 2, 2013 at 2:41 AM, Tito Latini <tit...@gm...> wrote: > Martin Cracauer wrote: >> As said, forking, compiling in the child and then loading the fasl in >> the parent, is an option. > > ok, useful to know there aren't esoteric techniques to resolve this > problem. I'll weigh the sacrifice of the portability and the size of > the dynamic space in the choice of the strategy. To avoid the use of > the disk will be a priority. Forking isn't the only option; there is sb-ext:run-program as well as the external-program portability layer in quicklisp. Tmpfs can be used with any lisp, of course, and there are RAM disks for other OSes. |
From: Nikodemus S. <nik...@ra...> - 2013-07-02 10:12:27
|
Re. loading fasls from a stream. While LOAD doesn't support it currently, it could. All that needs to be done is for the fasl-header check to construct a concatenated stream to pass to load-as-source if there was no fasl header. While last working on the area I didn't have enough steam for that, and so just punted and allowed fasl loading only from seekable streams. Meta-dotting into LOAD and FASL-HEADER-P from there should zero anyone interested into the right area pretty quickly. Cheers, -- nikodemus |
From: Tito L. <tit...@gm...> - 2013-07-02 12:19:27
|
clean, thank you On Tue, Jul 02, 2013 at 01:12:20PM +0300, Nikodemus Siivola wrote: > Re. loading fasls from a stream. > > While LOAD doesn't support it currently, it could. All that needs to > be done is for the fasl-header check to construct a concatenated > stream to pass to load-as-source if there was no fasl header. > > While last working on the area I didn't have enough steam for that, > and so just punted and allowed fasl loading only from seekable > streams. > > Meta-dotting into LOAD and FASL-HEADER-P from there should zero anyone > interested into the right area pretty quickly. > > Cheers, > > -- nikodemus |
From: Martin C. <cra...@co...> - 2013-07-02 17:25:47
|
Tito Latini wrote on Tue, Jul 02, 2013 at 08:41:47AM +0200: > Martin Cracauer wrote: > > As said, forking, compiling in the child and then loading the fasl in > > the parent, is an option. > > ok, useful to know there aren't esoteric techniques to resolve this > problem. I'll weigh the sacrifice of the portability and the size of > the dynamic space in the choice of the strategy. To avoid the use of > the disk will be a priority. Where does the disk come in as a problem? The garbage from compilation will stop your world via the global GC. Are you running the compiler in a separate thread? Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer <cra...@co...> http://www.cons.org/cracauer/ |
From: Tito L. <tit...@gm...> - 2013-07-02 20:49:22
|
On Tue, Jul 02, 2013 at 01:25:38PM -0400, Martin Cracauer wrote: > Tito Latini wrote on Tue, Jul 02, 2013 at 08:41:47AM +0200: > > Martin Cracauer wrote: > > > As said, forking, compiling in the child and then loading the fasl in > > > the parent, is an option. > > > > ok, useful to know there aren't esoteric techniques to resolve this > > problem. I'll weigh the sacrifice of the portability and the size of > > the dynamic space in the choice of the strategy. To avoid the use of > > the disk will be a priority. > > Where does the disk come in as a problem? The garbage from compilation > will stop your world via the global GC. Are you running the compiler > in a separate thread? There is a rt-thread, a nrt-thread, a fast-nrt-thread and a thread for the REPL. The communication with the rt-thread is lock-free. The expansion of DEFSYNTH is a big block and the compiler triggers the gc that stops the world. A possibility to reduce the effect of the gc is to use a separate instance of SBCL to compile the monster, avoiding the use of the disk to save/load the fasl. I deduced there is not a common recipe for this task, therefore I can use what I want. I appreciated all the replies, now it is the time to think. Tito Latini |
From: Martin C. <cra...@co...> - 2013-07-02 21:33:54
|
Tito Latini wrote on Tue, Jul 02, 2013 at 10:48:17PM +0200: > On Tue, Jul 02, 2013 at 01:25:38PM -0400, Martin Cracauer wrote: > > Tito Latini wrote on Tue, Jul 02, 2013 at 08:41:47AM +0200: > > > Martin Cracauer wrote: > > > > As said, forking, compiling in the child and then loading the fasl in > > > > the parent, is an option. > > > > > > ok, useful to know there aren't esoteric techniques to resolve this > > > problem. I'll weigh the sacrifice of the portability and the size of > > > the dynamic space in the choice of the strategy. To avoid the use of > > > the disk will be a priority. > > > > Where does the disk come in as a problem? The garbage from compilation > > will stop your world via the global GC. Are you running the compiler > > in a separate thread? > > There is a rt-thread, a nrt-thread, a fast-nrt-thread and a thread for > the REPL. The communication with the rt-thread is lock-free. The > expansion of DEFSYNTH is a big block and the compiler triggers the gc > that stops the world. A possibility to reduce the effect of the gc is > to use a separate instance of SBCL to compile the monster, avoiding > the use of the disk to save/load the fasl. I deduced there is not a > common recipe for this task, therefore I can use what I want. I still don't get why you think anything will block on writing and then reading the fasl. The writing is write-back and the read feeds out of the filesystem buffer cache. Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer <cra...@co...> http://www.cons.org/cracauer/ |