Re: [Iaxclient-devel] Some proposed feature additions to the library

[Steve K. <st...@st...>]

Responses inline:

Steven M. Sokol wrote:

> Steve & Company,
>
> First, thank you for the codec code.  I will be releasing a new 
> version of the free version of IAX Phone with the codec-enabled 
> library included.
>
> I'm working on a few improvements (or at least additions) to the 
> library, and I want some response.  Some of these are already coded, 
> while others are still on the drawing board.  These include:
>
> - New registration routine
> - Client-Side Call Recording
> - Multi-Path Audio Output (and possibly input)
> - Onboard Mixing for 3-way Calling
> - URL Send/URL Event
> - DTMF Received Event
>
> -= New Registration Routines =-
>
> I've created an iaxc_register2 function that creates a managed list of 
> registrations and their states.  It required a single change to the 
> existing registration structure -- the addition of a regid integer to 
> hold the registration identifier.  This value (simply an incrmenting 
> integer) is generated when the registration is created and is used to 
> identify the registration during callbacks (events).  The function 
> call returns the newly generated registration ID, or negative values 
> to indicate errors.
>
> The registration event (appended to the main event structure) returns 
> the registration ID, the registration result, and the message count 
> for the mailbox associated with the user account (if any).  A new 
> event type, IAXC_EVENT_REG is used as the event type.  The 
> registration event is generated by the iaxc_regmsg function which is 
> called from iaxc_handle_regreply.
>
> The general purpose of this change was to allow the client to display 
> registration status on an account-by-account basis, and to track 
> message counts for each account.  This is a very minor tweak (and I 
> suppose we could have returned the registration information in the 
> user message, but this seems to be a more elegant approach).

I think that the consensus seems to be that this functionality can 
probably replace the existing functionality.  I just threw that 
together, and I don't actually use registration myself :) 

I'd add a function to cancel a registration as well, once you have an ID 
and are storing the registrations.

> -= Client-Side Call Recording =-
>
> This set of routines makes use of the libsndfile project code to 
> integrate call recording into the library.  The same function can be 
> handled by the server using the monitor, but in accordance with Mark's 
> belief that the greatest amount of processing power is available at 
> the edge, it can be quite useful.  It also fits a need for a client 
> who /really/ wanted client-side recording and file playback.
>
> The code to make this happen comes, largely, from the libsndfile 
> project.  The libsndfile functions handle the creation of the file 
> header (if any) plus the proper formatting of the data for the storage 
> format.  My implementation makes use of it as an external DLL, but it 
> could readily be compiled into iaxClientLib.  It is cross-platform 
> capable in most respects.  The one issue I have not been able to make 
> fully cross-platform is timing.
>
> Windows supports "multimedia timers" which are accurate down to 1 ms.  
> This allows for accurate playback.  I don't know much about timers in 
> Linux (only that the 2.6 kernel is the first to support innate 
> high-frequency timers) and absolutely nothing about timers in Mac 
> OSX.  Thus it /may/ be best to keep this as a patch outside of the 
> main code base.  My implementation segements it with #ifdef  WIN32.
>
> The recording function captures the output audio stream (as linear 
> PCM) in the send_encoded_audio function in audio_encode.c.  It 
> captures the input audio stream (also as linear PCM) in the 
> decode_audio function.  At the present time, both streams are written 
> to separate files which can be optionally mixed using a separate 
> mixing function.
>
> Playback of files is handled in a timer callback function in the WIN32 
> section of the code.  It encodes and sends the audio using the 
> existing send_encoded_audio function.  I may add an option of playing 
> the audio back over the local audio.output channel as well.
>
> The recording and playback functions generate a series of events which 
> are added to the main event structure.  These include and event type 
> of IAXC_EVENT_FILE_IO, an event sub-type (start, stop, etc.), and io 
> sub-type (recording or playback), and a cause (timeout, end-of-file, 
> digit, or stop command).
>
> This function currently exists in the version of the library that 
> handled only GSM.  I am working on porting it into the new multi-codec 
> version.


I think this can be done in a portable way.  We already have the 
playback functionality necessary for sounds.

I'm not sure why you need millisecond-accuracy timers for this, though, 
but gettimeofday will be millisecond-precise on linux-2.6 (10ms on 
linux-2.4) and on Mac OS X.   The present gettimeofday emulation on 
Windows is currently 10ms-precise on NT-based machines, and 55ms-precise 
on 9x-based machines.

> -= Multi-Path Audio Output (and Input) =-
>
> This feature is still on the drawing board.  It services two 
> functions.  First, it allows for phones with multiple audio devices to 
> direct the audio from one call to a given device and audio from 
> another to a separate device.  The classic example of this is placing 
> and "on hold" call on one device (the speakers) while communicating on 
> another call using a headset.  The second function is a bit more 
> esoteric and relates to a specific customer request.
>
> My thought here would be to have a pool of audio objects which can be 
> dynamically assigned to the pool of calls.  Instead of having the 
> "selected" call be the sole consumer/source of audio, the call would 
> check to see if an audio object was assigned.  If so, it would make 
> use of that object.  If not, the audio would be discarded.  This still 
> allows for the use of the "selected" call as the primary (if the 
> application needs a "primary") but also allows for other calls to 
> route audio.
>
> We currently create a single audio object for the client, then assign 
> various devices (in, out, ring) to that audio object.  All calls make 
> use of that single audio object, but only when the call is the 
> "selected" call.  In order for the multi-destination system to work, 
> it would seem logical to create separate audio objects and assign them 
> each an input, output and ring device.  Then, each call could be 
> assigned a different audio object.  The function would take as 
> parameters the call ID and the object ID and would handle properly 
> passing the references, returning an error if the audio object is 
> already in use or does not exist.
>
> I suspect that this will require quite a bit of reworking of the 
> current audio driver structure.  Can the structure be expanded to 
> support multiple audio i/o devices?  Can we simply instantiate and 
> initialize several driver objects as it stants?  Do we run into any 
> kind of issue in terms of threading if we try to read and/or write on 
> multiple devices simultaneously?

The audio_portaudio driver (which is the only one, aside from audio_file 
which works at all) presently uses some statics for state; thse would 
need to be moved into a private structure. 

I'm not sure how portaudio itself would react to having several 
instantations in the same process, but I think it handles this.

There's a mutex in audio_portaudio that is static, it would need to be 
moved into the structure to make it thread safe (or, it could be kept 
static, and shared for by each instantation, as long as it was protected 
from being re-initialized, etc).

You could search the portaudio list, or look at portaudio to see about 
it's multiple-instantiation-safety.

Another way to do this would be to have more than one iaxclient 
instantiation, and to tranfer calls  between them, etc.  It is probably 
not safe to do this in a single process though, so you'd need to use a 
co-process model (i.e. like tkphone/iaxcli), to do this.

>
> It will also require at least a small change to the call structure -- 
> adding the audio object as a memeber.  The audio handler routines 
> would, obviously, need to be altered to make use of the audio object 
> associated with the call, rather than the single central audio object.
>
> -= Onboard Mixing For 3-way Calling =-
>
> I originally started to work out a routine for passing feature 
> requests back to the server over IAX primarily to enable a server-side 
> conferencing function.  Mark steared me away from that, as again he 
> feels the client should handle trivial functions like mixing for 3-way 
> calling.  As such I want to implement client native simple conferencing.
>
>                    -----------
> CALL1 SEND--------->|         |------>CALL1 RCV
>                    |   MIX   |
> CALL2 SEND--------->|    &    |------>CALL2 RCV
>                    |   ECHO  |
> AUDIO IN (MIC)----->|  CANCEL |------>AUDIO OUT (SPK)
>                    -----------
>
> I suppose I could spend a bit of time making myself familliar with the 
> code in app_meetme, but this may be a problem that other more 
> familliar with the specifics of mixing and echo cancellation would be 
> better able to address.  If nothing else, pointers would be happily 
> accepted!

Look at app_conference -- it does all of this already.

You don't need echo cancellation for this, though, meetme and 
app_conference don't do this.

In this case, you have a 3 party conference.  Call the 3 parties A, B, C.

If you do this the way meetme does it, you do this:

Mix A+B, send audio to C.
Mix A+C, send audio to B.
Mix B+C, send audio to A.

app_conference does this in a slightly more efficient way when parties 
do VAD, because in this case, most of the parties won't be sending 
audio, and you generally only have audio from one speaker.


So, you could do all this in iaxclient if you wanted to, but it would 
still be better to do it in asterisk with meetme, or app_conference, and 
have some intellegent dialplan setup combined with transfers and perhaps 
custom UI in iaxclient, and let it all be done via asterisk..

>
> -= URL Send/URL Event =-
>
> This is simply the addition of a wrapper to the innate IAX2 URL 
> function and event.  It should look much like the text event.  This 
> will make it easy for the application to either "pop" a browser, or 
> display a web page in a browser window incorporated into the client.


Sounds simple.

>
> -= DTMF Event =-
>
> This simply converts the DTMF events that are currently handled as 
> part of the user message stream and provides a separate event that can 
> be used by the client to recognize DTMF from the far end of the call.  
> This function is being added to facilitate some basic client-side 
> automation that a customer has requested.

Also sounds simple.

Also, w.r.t. Dan's comments about using Message ID's instead of strings 
for many of the callbacks, I'd support that too.  It would probably be 
best to just add a message-id, and also have the string in english, or 
to have a method inside of iaxclient which could give you a (possibly 
internationalized) string given a message-id, that way we have all the 
strings in the library for everyone to contribute to and use.

-SteveK