tux-droid-user

[tuxdroid-user] Refactoring the API

[David B. <da...@ja...>]

Hi Guys,

I finally had a closer look at the API when adding the connection  
functions for the ID, sleep and wifi features. We're really going forward  
on the daemon, it gets cleaner and better everyday. The next step would be  
to dig into the API and do the same kind of cleaning and sanity job.  
Though the work here seems to be much different than for the daemon.

The python API is a huge single file that needs to be cut into pieces and  
it doesn't look like the initial architecture has been carefully chosen.  
Don't take me wrong, there's a lot of (good) functionalities in it and it  
does the job. But I see a couple of problems starting working on this one  
with the community.

So I suggest to start a new API from scratch, just taking out the good  
pieces of the current one. Here are a few reasons why.

First, Rémi is actively developping it alone, he does a lot of changes in  
every direction regularly. Unfortunately he doesn't really get into the  
open source development philosophy and this makes it hard for him to use  
svn correctly. On the other hand he has to develop some end user  
applications very quickly which explains why the API is evolving the way  
it is now. If we start changing this API, he'll get a lot of troubles  
merging all his changes every 2 weeks with ours, and we'll get the same  
trouble seing those huge commits changing nearly every line once in a  
while. For his applications, Rémi needs to keep his flexibility. On our  
side, we want to follow the style guidelines and best development  
practices that makes good OS software.

Second, I really think it will be quicker to rewrite a new api than to  
change the current one. IMHO the design structure is wrong but I still  
don't have the expertise to be able to design a good one. Just an example,  
I would expect to have a tux object with an 'leds' property, and  
'tux.leds' should return the status of the leds (left=on,right=off) while  
writing to 'tux.leds' would set them to the value. Right now you have to  
call a command with a defined parameter for the led command, and call a  
completely different status command to get the value of the leds. That  
sounds like standard structured programming to me, not OOP. (I really need  
to read a book on that to be sure though ;-) )

So I suggest we take some time to refine the functionalities of the api,  
decide on the design structure, and then implement it. That will take more  
time to get all opinions before acting but as there's already a functional  
api, we can afford to take the time to do it well.

Third, the current api is way too long. All functionalities are handled by  
the api while the daemon is basically a USB to TCP/IP interface.
There are a few problems to this. As most commands needs to cross a lot of  
interfaces before getting processed by tux and come back as a new status,  
the asynchronous communication between the daemon and the api get's overly  
complicated. And as the api is quite complex, it makes it very difficult  
to port it to different languages. I would suggest to carefully decide  
what should be moved from the api to the daemon. And going for a new api  
makes this easier.

Some strange things also. Right now all status are read from the USB and  
stored in variables. The clients get ALL status continuously through the  
api, these status are parsed to trigger a few things then dropped. So when  
a client wants to know the status of something, it still needs to do a new  
request to the daemon which will send the status again. Even if it's not a  
big load on the network, it still is a lot of useless transactions.  
Imagine a client that does only that: say "dark" when the light level gets  
under a chosen threshold. It will receive all status and they'll be  
dropped except the light level, while just one notification when the light  
gets dark would suffice. We could imagine something closer to what DBUS  
does, the client registers a function with an event that the daemon could  
offer (like low_light_event, threshold as parameter) and the daemon will  
only contact the client when the event occurs. Clients could also  
subscribe to specific status, or request them separately. Such a thing  
would change the network protocol completely.

To conclude, I think it could be a good way to keep Rémi working on the  
current api, quickly developping and testing new functionalities and  
applications, while we start thinking and working on a new clean branch to  
be a mid-term replacement solution. Once nice functionalities are getting  
into Rémi's work, we could simply copy them into our branch.

On the actions side, first we can discuss this a bit, see the pros and  
cons. Then I'll use the wiki to draw a list of all functionalities we  
currently have in the api and we'll decide what should be handled by the  
daemon and the api. There's also a virgin API page on the wiki where the  
new structure can be defined.

How does this sound to you?
David

Re: [tuxdroid-user] Refactoring the API

[neimad <ror...@gm...>]

"David Bourgeois" <da...@ja...> writes:

[...]
> some end user  applications very quickly which explains why the API is
> evolving the way  it is now. If we start changing this API, he'll get
> a lot of troubles  merging all his changes every 2 weeks with ours,
> and we'll get the same  trouble seing those huge commits changing
> nearly every line once in a  while. For his applications, R=C3=A9mi needs
> to keep his flexibility. On our  side, we want to follow the style
> guidelines and best development  practices that makes good OS
> software.

If we start a new API from scratch, maybe we could also write a
wrapper for new API exposing the same old API, so as to ease the
transition. This may not be possible for everything, though...

> Second, I really think it will be quicker to rewrite a new api than to
> change the current one. IMHO the design structure is wrong but I still
> don't have the expertise to be able to design a good one. Just an
> example,  I would expect to have a tux object with an 'leds' property,
> and  'tux.leds' should return the status of the leds
> (left=3Don,right=3Doff) while  writing to 'tux.leds' would set them to the
> value. Right now you have to  call a command with a defined parameter
> for the led command, and call a  completely different status command
> to get the value of the leds. That  sounds like standard structured
> programming to me, not OOP. (I really need  to read a book on that to
> be sure though ;-) )

Agreed.

[...]
> makes it very difficult  to port it to different languages. I would
> suggest to carefully decide  what should be moved from the api to the
> daemon. And going for a new api  makes this easier.

Haven't thought about any of this. I'll have this in mind next time I'll
wander in the code.

> Some strange things also. Right now all status are read from the USB
> and  stored in variables. The clients get ALL status continuously
> through the  api, these status are parsed to trigger a few things then
> dropped. So when  a client wants to know the status of something, it
> still needs to do a new  request to the daemon which will send the
> status again. Even if it's not a  big load on the network, it still is
> a lot of useless transactions.  Imagine a client that does only that:
> say "dark" when the light level gets  under a chosen threshold. It
> will receive all status and they'll be  dropped except the light
> level, while just one notification when the light  gets dark would
> suffice. We could imagine something closer to what DBUS  does, the
> client registers a function with an event that the daemon could  offer
> (like low_light_event, threshold as parameter) and the daemon will
> only contact the client when the event occurs. Clients could also
> subscribe to specific status, or request them separately. Such a thing
> would change the network protocol completely.

I want to add that the current API uses byte arrays for all this and
it makes things cumbersome and really hard to read and extend. Using
*real* data structures and marshalling would help immensely by making
the code shorter, cleaner, and more obvious.

> To conclude, I think it could be a good way to keep R=C3=A9mi working on
> the  current api, quickly developping and testing new functionalities
> and  applications, while we start thinking and working on a new clean
> branch to  be a mid-term replacement solution. Once nice
> functionalities are getting  into R=C3=A9mi's work, we could simply copy
> them into our branch.

Depends on how different the new API is from the old one. Could be
tricky unless this new stuff has a clear interface with the API (i.e.,
no things intertwined or buried deep down the code structure)

> On the actions side, first we can discuss this a bit, see the pros and
> cons. Then I'll use the wiki to draw a list of all functionalities we
> currently have in the api and we'll decide what should be handled by
> the  daemon and the api. There's also a virgin API page on the wiki
> where the  new structure can be defined.
>
> How does this sound to you?

Sounds good :-)

	Damien

Re: [tuxdroid-user] Refactoring the API

[David B. <da...@ja...>]

On Fri, 15 Jun 2007 19:31:57 +0200, neimad <ror...@gm...> wrote:

> "David Bourgeois" <da...@ja...> writes:
>
> [...]
>> some end user  applications very quickly which explains why the API is
>> evolving the way  it is now. If we start changing this API, he'll get
>> a lot of troubles  merging all his changes every 2 weeks with ours,
>> and we'll get the same  trouble seing those huge commits changing
>> nearly every line once in a  while. For his applications, Rémi needs
>> to keep his flexibility. On our  side, we want to follow the style
>> guidelines and best development  practices that makes good OS
>> software.
>
> If we start a new API from scratch, maybe we could also write a
> wrapper for new API exposing the same old API, so as to ease the
> transition. This may not be possible for everything, though...

Yes, I'd like to get thattoo though that should not be a requirement  
neither a priority. The project is still at its early stages and there  
isn't yet a lot of applications lying around so I don't think it will  
generate much trouble to refactor the API. I'd vote for a good design  
prior to compatibility.

>> Some strange things also. Right now all status are read from the USB
>> and  stored in variables. The clients get ALL status continuously
>> through the  api, these status are parsed to trigger a few things then
>> dropped. So when  a client wants to know the status of something, it
>> still needs to do a new  request to the daemon which will send the
>> status again. Even if it's not a  big load on the network, it still is
>> a lot of useless transactions.  Imagine a client that does only that:
>> say "dark" when the light level gets  under a chosen threshold. It
>> will receive all status and they'll be  dropped except the light
>> level, while just one notification when the light  gets dark would
>> suffice. We could imagine something closer to what DBUS  does, the
>> client registers a function with an event that the daemon could  offer
>> (like low_light_event, threshold as parameter) and the daemon will
>> only contact the client when the event occurs. Clients could also
>> subscribe to specific status, or request them separately. Such a thing
>> would change the network protocol completely.
>
> I want to add that the current API uses byte arrays for all this and
> it makes things cumbersome and really hard to read and extend. Using
> *real* data structures and marshalling would help immensely by making
> the code shorter, cleaner, and more obvious.

Yes, sure. And we can already add these structures in the daemon too.

>> To conclude, I think it could be a good way to keep Rémi working on
>> the  current api, quickly developping and testing new functionalities
>> and  applications, while we start thinking and working on a new clean
>> branch to  be a mid-term replacement solution. Once nice
>> functionalities are getting  into Rémi's work, we could simply copy
>> them into our branch.
>
> Depends on how different the new API is from the old one. Could be
> tricky unless this new stuff has a clear interface with the API (i.e.,
> no things intertwined or buried deep down the code structure)

Didn't think about cut and paste of course. More like testing new  
functionalities. If something is found to be pretty neat and interesting  
in Rémi's work, we can implement it on our branch. And when we'll get  
something functional, I'll get Rémi move to the community branch too.  
That's just a temporary situation that would free us from the strain which  
is on the curent API.

David