POSIX message queues backend for gammu-smsd, and more

[Peter S. <pe...@st...>]

Hi,

I've written such a backend but unfortunately lost it because of a
kernel crash. It wasn't a lot of work so I'm writing it again.

I'm using it to make an SMS transport for jabber, which I also wrote
(in Python) and lost and will rewrite.

>From (only) reading code I have some concerns with the gammu-smsd
internals, please help me understand these issues better:


1. Encoding
What a f*ing mess! Student code abound. I'm looking at the backends
for reference, and at libgammu/../coding.c to learn what the various
coding functions actually do.. (Function names are useless - oh well,
I can fix that if I want to. ;)

Question:

I wish for gammu-smsd to always handle conversion to/from utf-8.

Reading mysql.c and into coding.c it seems to me that this code:

switch (sms->SMS[i].Coding) {
case SMS_Coding_Unicode_No_Compression:
case SMS_Coding_Default_No_Compression:
        EncodeHexUnicode(buffer..,..SMS[i].Text,UnicodeLength(sms->SMS[i].Text));
        break;
case SMS_Coding_8bit:
        EncodeHexBin(buffer..,..SMS[i].Text,sms->SMS[i].Length);
default:
        break;
}

actually is redundant. EncodeHexUnicode() translates 16-bit words to
hexadecimal ascii, and EncodeHexBin() translates bytes to hexadecimal
ascii. UnicodeLength() basically returns strlen()/2. (But with it's
own, better, implementation! wtf..) Am I correct?

It's also not clear to me what actually determines Coding for a
message - and what the appropriate way is to deal with the different
values for Coding.

For now I just assume SMS_Coding_Unicode_No_Compression. For incoming
messages I call EncodeUTF8() and all seems well.

For outgoing messages it's not so straightforward, since I must be
able to generate maximum efficiency SMSes. If the utf-8 message I'm
about to send fits completely within the GSM alphabet then I refuse
to produce "Unicode" SMSes which can only take less than half the
number of characters in each message. How can I solve this right now?
None of the relevant functions in coding.c return any *output* size,
so for one thing there are tons of possible buffer overflow bugs
within gammu but more importantly this makes it impossible for me to
determine how long any encoded output is! This means that I can't try
to use GSM alphabet and get an indication that coding failed, and
only *then* switch to using "Unicode" encoding - which may also
require me to generate more messages. This entire procedure of
recoding input needs to be moved into smsd core IM-Not-So-HO. I think
there should just be a single well-defined encoding used for text
outside smsd core.

For cases when gammu isn't dealing with text I think it's important
to not lose the low-level byte control in the application. Question:

How can we know in smsd core if an SMS is (part of) a text message,
or if it is something else? Can we determine this reliably from the
UDH? That would be prefect. If not UDH, then is there another way?


2. Multipart SMSes
Big problem. They *are* independent messages in the network and in
the hardware, but at least for text noone outside gammu cares about
that, quite the opposite, so smsd should just expose them as one
single message.

On one hand I consider it a big design flaw that backends need to
know anything about multipart messages, but on the other hand it
makes sense because each part *is* separate after all. Tricky!

I think we must improve this, and ultimately provide a more "cooked"
interface to programs outside of gammu-smsd. (Regardless if they look
for files or if they use SQL or as in my case message queues.)

An obvious move is that smsd core buffers GSM_SMSMessage:s internally
until all parts have been received, and either let core join them, or
at the very least guarantee that all parts of a message will always
be within one call to SaveInboxSMS() and in correct order. The latter
is a smaller change, but requires more duplicated processing in the
backends. There are only a few backends and they are pretty simple so
I think it's OK to make changes in the internal backend API.

That will have problems for anyone already depending on the current
behavior of gammu being a fairly "thin" or "raw" interface to
hardware. There has been talk about multipart messages on the list
before and it's my impression that this *is* a hassle for everyone.
Correct?

There's another, more serious, problem: What if all parts are not
received right away?!

Since smsd would now start taking some responsibility for joining
(or call it grouping) messages, it must also be able to give up and
just call SaveInboxSMS() with an incomplete multipart SMS at some
point. We could add a simple parameter to smsdrc for this, but I have
no idea what value would be realistic.. 60 seconds? 180? SMSC can
hold messages up to 72 hours!

Delivery to the backend quickly is important so I think we want a
fairly short timeout before sending *something* to the backend, and
to the user. Maybe 30 seconds, maybe 60 seconds? If sometime later
the missing parts are recieved then the backend would also receive
those.

For persistent storage like files or SQL server then it's easy to
match with previous parts, but for backends like my posixmq that
doesn't work since messages are not saved anywhere but passed on to
the message queue immediately; previous parts of the SMS are no
longer available, and the complete message text can not be presented
to the user. I consider this a fairly significant usability problem
and I'm afraid smsd is the only place it can be solved. I suggest
that smsd core buffers old parts of multipart messages until all
parts are received, and delivers both old and received parts to each
call to SaveInboxSMS(). MySQL can easily skip over old parts, and
PosixMQ can use them.


3. Message queue config
Message queues are named similar to a path: /messagequeuename
The name can not contain / but most other VFS-allowed characters.
I'm overloading the inboxpath and outboxpath config directives right
now, and have changed core to read them if using either posixmq or
files backend. Is that OK? Or should I absolutely add all new config
params for the posixmq backend?


All for now, sorry this got long. :)


//Peter