Re: [Audacity-devel] Exception safety and Nyquist Lisp

[James C. <cr...@in...>]

This merits a much fuller answer, but I'm going to say just a little here.

The extra functions for Nyquist are motivated by extra functions for 
external scripting, which in turn is for Python over a pipe or TCP/IP 
connection.

The Python will want functions for working with selections, for 
getting/setting window locations, for enumerating clips, for setting 
preferences and so on.  The big motivation there is to be able to 
generate just about every image in the manual by running a script.  A 
great workout for Audacity.  A big time saver for creating and updating 
images in the manual.

Can exceptions propagate across the pipe or TCP/IP connection? Well, I 
personally have no intention of serialising them and 'converting' them 
to Python exceptions.  So the API that the Python scripting is calling 
instead needs certain guarantees.  For my needs it looks to be that any 
unwinding of objects be handled in Audacity before a result is handed 
back to Python over the pipe.  Exactly the same logic seems appropriate 
to Nyquist.  Part of the semantics of the scripting API is that it is a 
'firewall' across which exceptions do not pass.

Is that too restrictive?  I don't think so.  The Python and Nyquist 
scripts do need a way to 'abandon ship' when a prior error (e.g. could 
not open audacity project file) means that all subsequent actions by the 
script are pointless.  I do not mandate how that is done, but instead 
provide enough control that it is possible.  I provide enough control 
that someone writing Python (or Nyquist) can request that Audacity 
report all such errors to the user - that's the default - OR ask that 
they not be reported by dialog, and that they will handle them.  Either 
way the script still gets to see the error.  On the Python side, the 
Python library can call a Python user supplied function on return from 
every call into Audacity.  The function could be set to raise Python 
exceptions if that user wants, or to stop the script, but that is up to 
them.  All that is guaranteed is that they can read results of the call 
into Audacity, which could be an error result.  Any unwinding of 
Audacity objects will already have been done at that stage.  Same for 
Nyquist.

--James.




On 8/18/2017 11:05 PM, Paul Licameli wrote:
> On Fri, Aug 18, 2017 at 4:54 PM, Roger Dannenberg <rb...@cs...> wrote:
>
>> What's the ultimate goal or desired behavior? It seems to me that the
>> most we could do or should do is print some text to the user in a dialog
>> box. Since the Audacity/Nyquist interface can already show text, it
>> seems like a simple mechanism could capture error information and
>> display it. I've never been a fan of exception handling (other than
>> simply returning values or error codes) because you never know if a
>> function call will return or not, so it's much harder to reason about
>> code (IMO). It sounds like Audacity is pretty committed to throwing
>> exceptions as a style. It's true that XLISP implements throw/catch
>> control structures, and it would be possible to catch C++ exceptions and
>> then re-throw them as Lisp exception, and even catch them at the top
>> level and re-throw them as C++ exceptions if you fall off the XLISP stack.
>>
>> -Roger
>>
> What, indeed, is the goal?  James raised mention of Nyquist in the context
> of command scripts, and I would like James to explain.
>
> There seems to be a desire to use Nyquist Lisp as a built-in command
> scripting language for the Audacity application, which goes beyond its
> original limited purposes for implementing effects and generators and
> analyzers.  I think James wants this, I believe Steve wants it too, even
> more so.  I say, if you want that expansion of what our Lisp can do, then
> you must take these other precautions.
>
> One of my big projects in Audacity 2.2.0 was to use C++ throw and catch to
> recover from disk exhaustion errors, and disk read errors too, and report
> them to the user.  In my opinion, contra Roger, exception handling is a
> very good thing.  It makes it very clean to encode the long-distance
> communication between the low level detection of resource exhaustion (which
> might strike in any of many deeply nested places) and the high level (where
> we recover from incomplete operations in just one place, by rollback of the
> undo state, and then make an error report).  Tedious explicit propagation
> of error codes through many levels of stack is avoided.  Rewriting the code
> we had was easier this way.
>
> But it also means many levels of stack in between must be written in an
> exception-aware way to avoid resource leaks or other undesired side-effects
> of operations abandoned in the middle, and I scrutinized much such code to
> be sure of its safety.  Use of smart pointers is only a part of that.  See
> also the many uses of finally().  It is not hard to adopt the "RAII"
> discipline, but you must learn it, and know the places that need it when
> you see them.  Middle level code must be safe for exceptions to go
> through.  Between throw and catch, the code must know how to "duck."
>
> If we want to bind some possibly throwing low-level operations on the
> Audacity project as Lisp functions, and then write the middle-level code
> for novel menu commands in Lisp instead of C++, then I want reassurances
> that Lisp is playing nicely with this exception handling framework that I
> implemented with much effort this past year, letting the exceptions through
> safely, neither "eating" them (not letting the high level know them) or
> "choking" on them (corrupting the Lisp runtime when they do go through).  I
> outlined how to accomplish that.
>
> And as for throw, catch, and unwind-protect in XLISP -- those are in fact
> parts of Common Lisp.  XLISP is a subset of Common Lisp (plus an object
> system), which includes all of the Common Lisp flow of control structures.
>
> PRL
>
>
>
>>
>> On 8/18/17 3:31 PM, Steve the Fiddle wrote:
>>> On 18 August 2017 at 14:52, Paul Licameli <pau...@gm...>
>> wrote:
>>>> Some idea jottings, mostly for James, and for future reference:
>>>>
>>>> If we want more Audacity functions to be callable from Nyquist Lisp, and
>>>> these functions might throw, and we want my new error reporting to the
>> user
>>>> to function just as elsewhere in Audacity, then what must be done?
>>>>
>>>> See what I did at commit 8e5975b10db0, but it's not satisfactory.
>>> It would be more helpful if you gave a link rather than expecting us
>>> to search for the repository and branch that you are referring to.
>>>
>>> Steve
>>>
>>>> Catch exceptions where Nyquist calls back into C++ land, translate to
>> error
>>>> code.
>>>> At the high level where Lisp gives control back to C++, detect the
>> error,
>>>> try to reconstruct an equivalent exception, and re-throw.
>>>> This is unsatisfactory because it doesn't generalize to all possible
>>>> exceptions.  We are less free to let the callbacks into C++ throw other
>>>> things if need be.
>>>> Re-throw is really needed.  You don't want just to absorb the exception
>> at a
>>>> low level.  You want to propagate through levels above Nyquist so they
>> can
>>>> do their proper RAII and ultimately report the cause of failure to the
>> user.
>>>> I mention in comments that the missing thing I want to use is
>>>> std::exception_ptr.
>>>>
>>>> That's reference-counting of exception objects in the C++ runtime, new
>> in
>>>> C++11.  You can keep an exception object alive even after leaving the
>> catch
>>>> block than handles it, and re-throw it later.  Read more here:
>>>> http://en.cppreference.com/w/cpp/error/exception_ptr
>>>> But on Mac at least I know we have not yet migrated to a version of the
>>>> library that includes it.  And this is connected to the question of
>>>> abandoning Snow Leopard, which did not get reexamined this release.
>>>> I don't know what minimal versions are needed on the other platforms to
>> get
>>>> us exception_ptr.
>>>>
>>>> Given exception_ptr you could re-do commit 8e5975b10db0 properly and
>> more
>>>> generally (as to possible exception types).  You could do the analogous
>> in a
>>>> few places, wrapping each foreign function, and testing for an
>> exception to
>>>> re-throw at each place where Lisp runtime returns to C++.
>>>> All this would propagate C++ exceptions, while preventing C++ stack
>>>> unwinding within the levels of stack written in C that implement the
>> Lisp
>>>> interpreter.
>>>>
>>>> But Lisp also has its exception handling -- you knew that, right?
>>>>
>>>> Relevant forms are throw, catch, and unwind-protect, the latter
>> allowing you
>>>> to use the equivalent of the finally template function that is much
>> used in
>>>> our C++ now.
>>>> You can examine the code implementing the Lisp interpreter -- I read it
>> all
>>>> a few years ago, and forgot much -- and you will see that setjmp and
>> longjmp
>>>> implement it.
>>>> If what is thrown with Lisp throw is not caught with Lisp catch, then a
>>>> top-level setjmp in the Lisp runtime stops it.
>>>>
>>>>
>>>> So: beyond the use of exception_ptr, do we want gateways between Lisp
>> and
>>>> C++ that translate exceptions?  These ideas would require a little bit
>> of
>>>> hacking the Lisp runtime, but I am sure we could figure it out.
>>>>
>>>> Where the foreign function wrapper catches and saves exception_ptr, it
>> would
>>>> also find the current jmp_buf in the Lisp runtime and longjmp it.
>>>> Levels of Lisp code could then visit their unwind_protect handlers in
>>>> response to a C++ exception, so that Lisp code could write RAII too.
>>>> We must be sure the implementation of Lisp catch (which examines the
>> Lisp
>>>> symbol associated with a Lisp throw) will never match a C++ exception
>> and
>>>> always pass control up the chain to the next jmp_buf.
>>>> Where Lisp implements its top-level setjmp of last resort (in C), it
>> could
>>>> return a code to caller indicating whether there was an unhandled
>> exception,
>>>> both in case this is a Lisp exception, or it is a C++ exception.
>>>> The C++ layer that called Lisp can detect this condition, and either
>> rethrow
>>>> the exception_ptr, or, finding it null, could construct a new C++
>> exception
>>>> object representing the Lisp exception, details of which we might also
>> have
>>>> saved in the Lisp code, using some foreign function.
>>>>
>>>> PRL
>>>>
>>>>