Pidgin

On 1/27/07, James Lockie <bjlockie@...> wrote:
> Does the  python interface to:
> self.gaim.GaimSavedstatusSetMessage(self.status, message)
> require a UTF8 string?

The Gaim API requires that all strings be UTF-8. I'm not sure how
Python deals with strings, but if it has the concept of different
encodings (as you seem to imply), it would make sense that you would
need to use UTF-8 strings in Python.

> I am trying to modify the AmarokGaim plugin so that it works with
> non-utf8 strings like "Bj=F6rk".
> It calls "message =3D message.decode('utf8')" where message might be the
> string "Bj=F6rk".
> This works for utf8 strings but fails on strings that contain special
> characters.

"strings that contain special characters"? You mean "strings that
aren't in UTF-8." I'm sure Amarok provides a mechanism for knowing the
encoding of the strings it gives you, or there's a KDE convention "all
strings use X encoding"


-s.

On 01/28/2007 06:26 AM, James Lockie wrote:

[...]
> I am trying to modify the AmarokGaim plugin so that it works with
> non-utf8 strings like "Bj=F6rk".
>
> It calls "message =3D message.decode('utf8')" where message might be th=
e
> string "Bj=F6rk".
[...]

The decode() method converts a string into a Unicode string (which is jus=
t
an internal representation and not the thing which you can pass anywhere
further). The argument specifies *current* encoding of the string. Thus i=
f
you know it's not utf-8 then you should not be giving utf-8 as the argume=
nt.

Unicode string is what you can later convert into other encodings, includ=
ing
utf-8.

In general, if you want to convert something into UTF-8, you should do:
message =3D message.decode(ENCODING_OF_MESSAGE).encode('utf-8')

The only problem is finding out what the proper value of ENCODING_OF_MESS=
AGE
is, but Sean has already mentioned how to do it.


take care,
Bartosz

Thanks.


Sean Egan wrote:
> I'm sure Amarok provides a mechanism for knowing the
> encoding of the strings it gives you

I have asked on the Amarok maiing list what string format Amarok is=20
returning. :-)



Bartosz Oler wrote:
> On 01/28/2007 06:26 AM, James Lockie wrote:
>
> [...]
>  =20
>> I am trying to modify the AmarokGaim plugin so that it works with
>> non-utf8 strings like "Bj=F6rk".
>>
>> It calls "message =3D message.decode('utf8')" where message might be t=
he
>> string "Bj=F6rk".
>>    =20
> [...]
>
> The decode() method converts a string into a Unicode string (which is j=
ust
> an internal representation and not the thing which you can pass anywher=
e
> further). The argument specifies *current* encoding of the string. Thus=
 if
> you know it's not utf-8 then you should not be giving utf-8 as the argu=
ment.
Mmm.
I didn't write the original code and I don't know python, gaim or Amarok=20
so I am trying to figure it out.
The original code passes utf8 to to decode and it works for strings with=20
non-special characters but fails on things like "Bj=F6rk".

I can print out "Bj=F6rk" to the console so the internal representation b=
y=20
python can handle special characters.

James Lockie wrote:
> Thanks.
>
>
> Sean Egan wrote:
>  =20
>> I'm sure Amarok provides a mechanism for knowing the
>> encoding of the strings it gives you
>>    =20
>
> I have asked on the Amarok maiing list what string format Amarok is=20
> returning. :-)
>  =20
I took out the decode and it works for strings without special=20
characters but I get this error for "Bj=F6rk":
message.append(signature=3Dintrospect_sig, *args)
UnicodeError: String parameters to be sent over D-Bus must be valid UTF-8

My guess is Amarok is NOT sending utf8 but decode works for some of the=20
strings but utf8 for other strings. :-(
I'll figure it out.  :-)

I was told "Bj=F6rk" can be represented by utf8 but that doesn't seem to=20
be the case.


>
>
> Bartosz Oler wrote:
>  =20
>> On 01/28/2007 06:26 AM, James Lockie wrote:
>>
>> [...]
>>  =20
>>    =20
>>> I am trying to modify the AmarokGaim plugin so that it works with
>>> non-utf8 strings like "Bj=F6rk".
>>>
>>> It calls "message =3D message.decode('utf8')" where message might be =
the
>>> string "Bj=F6rk".
>>>    =20
>>>      =20
>> [...]
>>
>> The decode() method converts a string into a Unicode string (which is =
just
>> an internal representation and not the thing which you can pass anywhe=
re
>> further). The argument specifies *current* encoding of the string. Thu=
s if
>> you know it's not utf-8 then you should not be giving utf-8 as the arg=
ument.
>>    =20
> Mmm.
> I didn't write the original code and I don't know python, gaim or Amaro=
k=20
> so I am trying to figure it out.
> The original code passes utf8 to to decode and it works for strings wit=
h=20
> non-special characters but fails on things like "Bj=F6rk".
>
> I can print out "Bj=F6rk" to the console so the internal representation=
 by=20
> python can handle special characters.
>  =20

James Lockie spake unto us the following wisdom:
> James Lockie wrote:
> > Sean Egan wrote:
> >> I'm sure Amarok provides a mechanism for knowing the
> >> encoding of the strings it gives you
> >
> > I have asked on the Amarok maiing list what string format Amarok is=20
> > returning. :-)
> >  =20
> I took out the decode and it works for strings without special=20
> characters but I get this error for "Bj=F6rk":
> message.append(signature=3Dintrospect_sig, *args)
> UnicodeError: String parameters to be sent over D-Bus must be valid UTF-8
>=20
> My guess is Amarok is NOT sending utf8 but decode works for some of the=
=20
> strings but utf8 for other strings. :-(
> I'll figure it out.  :-)

I'm guessing Amarok is not sending UTF-8 for *any* of its strings, but
those strings which happen to contain only characters in the ASCII
range are validating as UTF-8.  (ASCII strings are valid UTF-8, and
will display correctly.)  Any strings which contain extended ISO-Latin
characters (e.g., =F6) are probably in some locale-specific charset
(most likely ISO-8859-1 or ISO-8859-15) and are thus failing
validation.

Try taking a byte dump of the string you're trying to use, and see if
it doesn't say {0x42, 0x6a, 0xf6, 0x72, 0x6b}.  This is the
representation of "Bj=F6rk" in ISO-8859-{1,15}.

> I was told "Bj=F6rk" can be represented by utf8 but that doesn't seem to=
=20
> be the case.

It most certainly can.  B, j, r, and k are represented by their ASCII
values, and =F6 can be represented in several ways; this email contains
=F6 in UTF-8 as U+00F6, which UTF-8 represents as 0xc3 0xb6.

Ethan

--=20
The laws that forbid the carrying of arms are laws [that have no remedy
for evils].  They disarm only those who are neither inclined nor
determined to commit crimes.
		-- Cesare Beccaria, "On Crimes and Punishments", 1764

Ethan Blanton wrote:
> James Lockie spake unto us the following wisdom:
>  =20
>> James Lockie wrote:
>>    =20
>>> Sean Egan wrote:
>>>      =20
>>>> I'm sure Amarok provides a mechanism for knowing the
>>>> encoding of the strings it gives you
>>>>        =20
>>> I have asked on the Amarok maiing list what string format Amarok is=20
>>> returning. :-)
>>>  =20
>>>      =20
>> I took out the decode and it works for strings without special=20
>> characters but I get this error for "Bj=F6rk":
>> message.append(signature=3Dintrospect_sig, *args)
>> UnicodeError: String parameters to be sent over D-Bus must be valid UT=
F-8
>>
>> My guess is Amarok is NOT sending utf8 but decode works for some of th=
e=20
>> strings but utf8 for other strings. :-(
>> I'll figure it out.  :-)
>>    =20
>
> I'm guessing Amarok is not sending UTF-8 for *any* of its strings, but
> those strings which happen to contain only characters in the ASCII
> range are validating as UTF-8.  (ASCII strings are valid UTF-8, and
> will display correctly.)  Any strings which contain extended ISO-Latin
> characters (e.g., =F6) are probably in some locale-specific charset
> (most likely ISO-8859-1 or ISO-8859-15) and are thus failing
> validation.
>
> Try taking a byte dump of the string you're trying to use, and see if
> it doesn't say {0x42, 0x6a, 0xf6, 0x72, 0x6b}.  This is the
> representation of "Bj=F6rk" in ISO-8859-{1,15}.
>
>  =20
>> I was told "Bj=F6rk" can be represented by utf8 but that doesn't seem =
to=20
>> be the case.
>>    =20
>
> It most certainly can.  B, j, r, and k are represented by their ASCII
> values, and =F6 can be represented in several ways; this email contains
> =F6 in UTF-8 as U+00F6, which UTF-8 represents as 0xc3 0xb6.
Ah, the string is probably something other than UTF8.
I get it thanks.

I think I did this right :-)
'426c656564696e6720576f726473206279204d6f62696c65' =3D 'Bleeding Words by=
=20
Mobile'

'69742773206f6820736f20717569657420627920426af6726b' =3D 'it's oh so quie=
t=20
by Bj=F6rk'

James Lockie spake unto us the following wisdom:
> Ethan Blanton wrote:
> > It most certainly can.  B, j, r, and k are represented by their ASCII
> > values, and =F6 can be represented in several ways; this email contains
> > =F6 in UTF-8 as U+00F6, which UTF-8 represents as 0xc3 0xb6.
>
> Ah, the string is probably something other than UTF8.
> I get it thanks.
>=20
> I think I did this right :-)
> '426c656564696e6720576f726473206279204d6f62696c65' =3D 'Bleeding Words by=
=20
> Mobile'

Yes, this string is all ASCII -- I assume it works.

> '69742773206f6820736f20717569657420627920426af6726b' =3D 'it's oh so quiet
                                               ^^ =F6 in ISO-8859-{1,15}
> by Bj=F6rk'

So, if Amarok isn't telling you what encoding these strings are (and I
suspect it is, when it can tell itself, but some annotations, such as
id3v1, do not have encoding tags), your best bet is probably to simply
try whatever encoding you expect to be most common, and fall back on
replacement of the invalid characters if that doesn't work.  Something
like:

if (g_utf8_validate passes)
    use the string as is
else if (g_convert from ISO-8859-1 works)
    use the conversion
else
    gaim_utf8_salvage it

We use this sort of tactic in several places in Gaim, where strings
come in that are of some unknown encoding.

Ethan

--=20
The laws that forbid the carrying of arms are laws [that have no remedy
for evils].  They disarm only those who are neither inclined nor
determined to commit crimes.
		-- Cesare Beccaria, "On Crimes and Punishments", 1764

Ethan Blanton wrote:
> James Lockie spake unto us the following wisdom:
>  =20
>> Ethan Blanton wrote:
>>    =20
>>> It most certainly can.  B, j, r, and k are represented by their ASCII
>>> values, and =F6 can be represented in several ways; this email contai=
ns
>>> =F6 in UTF-8 as U+00F6, which UTF-8 represents as 0xc3 0xb6.
>>>      =20
>> Ah, the string is probably something other than UTF8.
>> I get it thanks.
>>
>> I think I did this right :-)
>> '426c656564696e6720576f726473206279204d6f62696c65' =3D 'Bleeding Words=
 by=20
>> Mobile'
>>    =20
>
> Yes, this string is all ASCII -- I assume it works.
>
>  =20
>> '69742773206f6820736f20717569657420627920426af6726b' =3D 'it's oh so q=
uiet
>>    =20
>                                                ^^ =F6 in ISO-8859-{1,15=
}
>  =20
>> by Bj=F6rk'
>>    =20
>
> So, if Amarok isn't telling you what encoding these strings are (and I
> suspect it is, when it can tell itself, but some annotations, such as
> id3v1, do not have encoding tags), your best bet is probably to simply
> try whatever encoding you expect to be most common, and fall back on
> replacement of the invalid characters if that doesn't work.  Something
> like:
>
> if (g_utf8_validate passes)
>     use the string as is
> else if (g_convert from ISO-8859-1 works)
>     use the conversion
> else
>     gaim_utf8_salvage it
>
> We use this sort of tactic in several places in Gaim, where strings
> come in that are of some unknown encoding.
>
> Ethan

Thank you so much.
I did
        # Try and decode message
        try:
            msg =3D message.decode('utf8')
        except:
            self.log("DecodeError: Could not decode utf8 '%s', trying ISO=
8$

            try:
                msg =3D message.decode('iso-8859-1')
            except:
                self.log("DecodeError: Could not decode '%s'" % message)
                return
which is what you suggested and it works now.

James Lockie spake unto us the following wisdom:
>         # Try and decode message
>         try:
>             msg =3D message.decode('utf8')
>         except:
>             self.log("DecodeError: Could not decode utf8 '%s', trying ISO=
8$
>=20
>             try:
>                 msg =3D message.decode('iso-8859-1')

Note that ISO-8859-1, specifically, should never fail to deocode, as
it uses all 256 codepoints.  This is not true of all encodings
(particularly multibyte encodings).

Glad to hear it works for you.

Ethan

--=20
The laws that forbid the carrying of arms are laws [that have no remedy
for evils].  They disarm only those who are neither inclined nor
determined to commit crimes.
		-- Cesare Beccaria, "On Crimes and Punishments", 1764

Anuj Verma (Kevin) spake unto us the following wisdom:
> I got interested after a long wait to see if libjingle ever gets in Gaim,
> I have gone through few basic searches, and I noticed Sean never actually
> responded on this.=20

Sean has responded to this numerous times, on various forums, but
perhaps not this mailing list.  Following the release of Gaim 2.0.0,
integration of libjingle support is slated for the 3.0.0 development
timeframe.  Jingle (nor other voice or video features) will not be
included in 2.0.0, or probably the 2.x series.

Ethan

--=20
The laws that forbid the carrying of arms are laws [that have no remedy
for evils].  They disarm only those who are neither inclined nor
determined to commit crimes.
		-- Cesare Beccaria, "On Crimes and Punishments", 1764

On 1/27/07, Anuj Verma (Kevin) <kevin.verma@...> wrote:
> I'll like to know if Sean can provide more insight on this, why Gaim is
> deprived of libjingle enhancements.

What a long-reaching followup! Turns out I had a draft in my Gmail
that I had forgotten to hit the send button on ;)

Supporting voice calling in Google Talk with libjingle doesn't make
sense in the context of Gaim unless it also provides an API
abstraction that allows all protocols to make voice calls (and video).
This is what's referred to as "gaim-vv."

Currently, the most promising avenue for that uses gstreamer to handle
most of the audio pipelining. This is one of the reasons I wanted Gaim
2.0.0 to use gstreamer to make it's little IM sounds: to see how it
would work for people.

At the time libjingle was originally released, the pieces needed for
gstreamer to work were still in their infancy and simply wouldn't work
at all.

Since then, as you mention, the work done by Collabora in working on
the Nokia 770 have vastly improved RTP streaming in Gstreamer, and it
should be possible to work on -vv. However, we really want to release
2.0.0 before adding any major new features and dependencies (and this
is pretty major).

It's not necessarily a "3.0.0" feature in that it doesn't break
backward-compatibility. Ethan's possibly pretty accurate at calling it
a "3.0.0 *timeframe*" feature, depending on how quickly we manage to
get it working.

-s.

On Jan 27, 2007, at 12:44 PM, Sean Egan wrote:

> On 1/27/07, Anuj Verma (Kevin) <kevin.verma@...> wrote:
>> I'll like to know if Sean can provide more insight on this, why  
>> Gaim is
>> deprived of libjingle enhancements.
>
> What a long-reaching followup! Turns out I had a draft in my Gmail
> that I had forgotten to hit the send button on ;)
>
> Supporting voice calling in Google Talk with libjingle doesn't make
> sense in the context of Gaim unless it also provides an API
> abstraction that allows all protocols to make voice calls (and video).
> This is what's referred to as "gaim-vv."
>
> Currently, the most promising avenue for that uses gstreamer to handle
> most of the audio pipelining. This is one of the reasons I wanted Gaim
> 2.0.0 to use gstreamer to make it's little IM sounds: to see how it
> would work for people.
>
> At the time libjingle was originally released, the pieces needed for
> gstreamer to work were still in their infancy and simply wouldn't work
> at all.
>
> Since then, as you mention, the work done by Collabora in working on
> the Nokia 770 have vastly improved RTP streaming in Gstreamer, and it
> should be possible to work on -vv. However, we really want to release
> 2.0.0 before adding any major new features and dependencies (and this
> is pretty major).
>
> It's not necessarily a "3.0.0" feature in that it doesn't break
> backward-compatibility. Ethan's possibly pretty accurate at calling it
> a "3.0.0 *timeframe*" feature, depending on how quickly we manage to
> get it working.

This is, and elb's other post, were very revealing. Will Gaim's vv  
code will be completely married to Gstreamer? If so, where does that  
leave folks who want to use VV, but are using libgaim on other  
platforms? SOL? I seem to remember reading that Gstreamer's mac  
support was fairly weak, and that they had little interest in  
improving it. Please, if I'm wrong, feel free to correct me.

-Colin

On Sat, 2007-01-27 at 17:34 -0800, Colin Barrett wrote:
> This is, and elb's other post, were very revealing. Will Gaim's vv =20
> code will be completely married to Gstreamer? If so, where does that =20
> leave folks who want to use VV, but are using libgaim on other =20
> platforms? SOL? I seem to remember reading that Gstreamer's mac =20
> support was fairly weak, and that they had little interest in =20
> improving it. Please, if I'm wrong, feel free to correct me.

I'm not sure how tied -vv and GStreamer will be, but let me pose this
question: Do you know of a better cross-platform toolkit to use in place
of GStreamer?

Richard

On Jan 27, 2007, at 5:51 PM, Evan Schoenberg wrote:

>
> On Jan 27, 2007, at 8:34 PM, Colin Barrett wrote:
>
>> I seem to remember reading that Gstreamer's mac
>> support was fairly weak, and that they had little interest in
>> improving it. Please, if I'm wrong, feel free to correct me.
>
> The current GStreamer release, 0.10.11, has this to say about OS X  
> [1]:
> ----
> 23.4. OS X
>
> GStreamer provides native video and audio output elements for OS X.  
> It builds using the standard development tools for OS X.
>
> ----
>
> what precisely that means bears investigation, but it's  
> significantly more promising than the state of things about a year  
> ago, when discussion seemed to conclude that GStreamer wouldn't  
> build on OS X.
>
> -Evan
>
> [1] http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-integration-osx.html

There's also:

http://zaheer.merali.org/articles/2006/04/27/osx-audio-with-gstreamer-0-10 
  and a bunch of other places that lead me to believe 0.10 compiles  
and runs (I found them by searching for "gstreamer os x" on google,  
sans quotes).

http://wiki.services.openoffice.org/wiki/Native_Sound_Implementation_on_Mac_OS_X 
  this page, however seems more relevant. If Gstreamer turns out not  
to work very well on OS X (we will need to do some testing and talk  
with the Gstreamer guys, probably), I would encourage you guys to at  
least attempt to abstract things out a little, so we might be able to  
shoehorn Quicktime in.

-Colin

On 1/27/07, Colin Barrett <timber@...> wrote:
> This is, and elb's other post, were very revealing. Will Gaim's vv
> code will be completely married to Gstreamer? If so, where does that
> leave folks who want to use VV, but are using libgaim on other
> platforms? SOL?

Heh, haven't we had this conversation on adium-devl before? ;)

It probably will be married to Gstreamer. It'll probably be something
along the lines of "give the PRPL a 'source' and a 'sink,' and let it
tie everything together to make things work (see: Farsight). It's
basically impossible to exact all the control needed otherwise. I've
certainly put a lot of thought into it.

> I seem to remember reading that Gstreamer's mac
> support was fairly weak, and that they had little interest in
> improving it. Please, if I'm wrong, feel free to correct me.

I recall it worked just fine, but you wanted to avoid it since
quicktime is capable of RTP streaming a bunch of audio codecs on its
own, so you wanted to be able to use it directly. Unfortunately, it
turned out not to be possible to do Jingle with it.

The core of gstreamer is pretty light-weight and entirely cross
platform. It just provides a way to stream data across different
'elements' while maintaining timing information and whatnot. Then
there are plugin elements for things like codecs and RTP streaming,
again, all platform-independent.

The platform-specific stuff all lies in data 'sinks' and 'sources,'
which tend to be relatively straightforward. On Linux, it's ass
complicated because everyone and his aunt went and invented a new way
to talk to /dev/dsp, but on Windows and Mac, the audio/video sources
and sinks are pretty straightforward, albeit not as well tested.

We'll be experimenting with getting it to work on Linux and Windows
after 2.0.0. Feel free to try messing around with it on OS X and see
how it is.

-s.

On 1/27/07, Colin Barrett <timber@...> wrote:
> I would encourage you guys to at
> least attempt to abstract things out a little, so we might be able to
> shoehorn Quicktime in.

That's the thing. Gstreamer *is* an abstraction.

I've scoured high and low to find media streaming that was:
- Cross platform
- Open-source
- Not sucky

and Gstreamer is absolutely the best alternative.

Sometimes an abstraction would be simple: you could easily have a
libgaim API that told the UI to stream PCMU audio with RTP to
10.0.0.0:8123, which is a pretty straight-forward, common case.

It's harder to say, in the case of MSN webcams, for instance, stream
video using this rare, reverse engineered codec.

And without essentially reinventing gstreamer from scratch, It's near
downright impossible to do something like Jingle, which boils down to:
here's a bunch of IP addresses, ports, and local UDP sockets. Send
STUN binding request packets out on all of the remote addresses from
each of the local sockets, and determine which one has the lowest
latency. Then use that connection to stream RTP audio with this audio
codec.

This isn't even *possible* to do if you use Quicktime, but with
Gstreamer, it's probably about 30 lines of code in the prpl. So rest
assured, even if you want to replace gstreamer with something else,
there's not much for you to #ifdef out ;)

On 1/28/07, Sean Egan <seanegan@...> wrote:
> On 1/28/07, Moritz 'Morty' Str=FCbe <morty@...> wrote:
> >  Ab bit OT: How does VLC (videolan.org) solve this? At least they have =
a lot
> > of output-filters for sound and video that work on a bunch of platforms=
??
>
> It uses SDL. All VLC needs to do is write to the screen and speakers
> as fast as humanly possible. SDL is great for that, but It has no way
> of getting images and audio from one computer to another.
>
> -s.
>