armory-devel Mailing List for Armory

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hello,

Will be great if people join #armory at irc.freenode.net

Lets speed this up guys.

Best Regards,

Danilo.

Hi,

I would like to contribute to the project interms of developer. I have
5Years exp as Storage Domain, Firmware Engineer. Please let me know if i can
join the project.

-- 
~Be Happy

Hi
Well, I am one of the new developers here and really wanna get started in
the server/client networking stuff(am in the beginner phase). I am unable to
get on the IRC due to some reasons(long story). Could someone help me/give
me a few tips and pointers on how to get started? I am comfortable with
C/C++ and (OOP) programming as such  network/application programming is a
new area for me.

Thanks

-- 
Varun Torka
http://lordvarun.googlepages.com

First of all, thank you very much Ananth for your response.

> 1) Dynamically linked plugins. I think it's a must both for server and
> > agents. It will enable us to better divide responsibilities, avoid
> clashes
> > in SVN and add new features without recompiling the whole app.
>
> The statistics gathering programs will be compiled into shared libs.
> The agent will be linked with these shared libs.

That's good. Though, if I understand correctly, it can be done better.
Instead of linking executables with statistics gathering modules during the
build process, we can use dlopen/dlsym (or LoadLibrary/GetProcAddress in
Windows) to link in run-time.

> 4) Networking library. Depends on all of the above.
>
> Darrik (another developer) and myself were discussing about which
> framework to use (Qt, ACE or Boost), but unfortunately he had to leave
> the project. I'm somewhat familiar with ACE and I think it is ideal
> for our requirements.

I admit I have only theoretical knowledge of ACE. It really seems to be the
right choice for the server. It also seems too heavyweight for agents.
Depending on the communication protocol we agree on, it might be better to
use two networking libraries, one for the server and another one for agents.

So far I was able to get in touch with only 3 out of 11 registered
developers (excluding Darrik who left and myself). The project seems to be
stalled. There is almost no activity, no proof that we are making any
progress. It kills peoples' motivation to participate in the project.
On SourceForge.net there's a nice tracker and a tasks system. We could
create some basic tasks and then gradually work out a design of the project
. Then divide responsibilities, assign tasks and write proper code.
Does anyone have a better idea?

Hi Robert,

My name is Ananth. I am currently working on the Server/Agent. Most of
my (little) experience is in C and UNIX system programming along with
networking, IPCs, etc and partly in C++.

Here are my comments:

> 1) Dynamically linked plugins. I think it's a must both for server and
> agents. It will enable us to better divide responsibilities, avoid clashes
> in SVN and add new features without recompiling the whole app.

The statistics gathering programs will be compiled into shared libs.
The agent will be linked with these shared libs.

> 2) Server-Agent communication protocol. Agents will be running on multiple
> platforms and probably multiple different CPUs, so we need a good data
> representation mechanism. Text format is the most cross-platform one, but
> its also more heavyweight than binary format. The proposed format (see
> https://sourceforge.net/forum/forum.php?thread_id=1751975&forum_id=689813)
> is too inflexible. Bicofino mentioned about database monitoring, which will
> definitely need lists. It is possible to do lists with the proposed format,
> but it'll look much like binary data represented as text which means waste
> of space. So why not use a binary format? I suggest we use XDR (which is
> standard and endian neutral) to transfer data through network.

XDR is a good idea. I have not programmed using it, but it would be a
great opportunity to learn.

> 3) Transport protocol. Since Armory is about monitoring, lots of small data
> packets are going to be sent from many hosts. Only a high-performance
> networking framework (like ACE) will be able to handle TCP connections.
> Moreover, if you have ever used any P2P software, you should know what
> happens to most routers/switches when there are too many TCP connections
> open.
> We can use UDP instead. It's true that some data will be lost, but things
> like CPU or memory usage are updated frequently, so no one will ever notice
> the loss of packets.

If minor loss of data is ok, then I have no problems with it.

>
> 4) Networking library. Depends on all of the above.

Darrik (another developer) and myself were discussing about which
framework to use (Qt, ACE or Boost), but unfortunately he had to leave
the project. I'm somewhat familiar with ACE and I think it is ideal
for our requirements.

>
>
> Any comments?
>
>
> Maybe we start writing down every key decision we make? There is no info
> about what is going on with the project.
>

So far, I've written the programs for getting CPU, memory, disk and
swap space stats on Linux, FreeBSD, HP-UX and AIX and a basic server
and agent in C, it is on the SVN. There are still many things to do in
the programs, but i uploaded to SVN anyway.

~Ananth

It's pretty hard to meet you all on #armory on IRC, so I thought it would be
better to use the mailing list.

First of all, I'm one of the new developers that recently joined the
project. My name is Robert Konklewski and I live in Poland. My experience
includes C/C++, Python, some networking and databases, so I'll be mostly
interested in the server/agents parts of the Armory.
It would be nice if the rest of you introduced themselves, so we all know
who we're working with.

Now I'd like to make a few comments on some of the threads in the
developers' forum.

1) Dynamically linked plugins. I think it's a must both for server and
agents. It will enable us to better divide responsibilities, avoid clashes
in SVN and add new features without recompiling the whole app.

2) Server-Agent communication protocol. Agents will be running on multiple
platforms and probably multiple different CPUs, so we need a good data
representation mechanism. Text format is the most cross-platform one, but
its also more heavyweight than binary format. The proposed format (see
https://sourceforge.net/forum/forum.php?thread_id=1751975&forum_id=689813)
is too inflexible. Bicofino mentioned about database monitoring, which will
definitely need lists. It is possible to do lists with the proposed format,
but it'll look much like binary data represented as text which means waste
of space. So why not use a binary format? I suggest we use XDR (which is
standard and endian neutral) to transfer data through network.

3) Transport protocol. Since Armory is about monitoring, lots of small data
packets are going to be sent from many hosts. Only a high-performance
networking framework (like ACE) will be able to handle TCP connections.
Moreover, if you have ever used any P2P software, you should know what
happens to most routers/switches when there are too many TCP connections
open.
We can use UDP instead. It's true that some data will be lost, but things
like CPU or memory usage are updated frequently, so no one will ever notice
the loss of packets.

4) Networking library. Depends on all of the above.

Any comments?

Maybe we start writing down every key decision we make? There is no info
about what is going on with the project.

I regret to inform you that due to my time being restricted suddenly by
other unforeseen life events, I must withdraw from this project.  I wish
you the best in a speedy development. :)

If you opt to use Qt in your project and want the code I've written thus
far, feel free to email me.  Otherwise, good luck with whatever tools
you choose.

Best wishes.

-- 
Darrik Mazey

> I've got most of the code you wrote converted to Qt, with a lot more
> other code in place (threaded server, plugin framework, etc). I was
> waiting to get a working copy before committing to sf svn, but if you
> want, I can commit most of what I have for you to look at.

Sure, do upload, we'll all take a look. But my only question is: why
use a GUI development tool for our purpose? IMHO, we just need a
simple server and client, but later we can try to multi-thread the
server. I've not programmed either in Qt or in ACE (but only know
about ACE a bit). But if you think it is ideal, i have no problems in
going ahead with it.

ana...@gm... wrote:
> the client and the statistics gathering programs are very much
> separate (please see the server and client code in the 'backend'
> directory in 'test' on sf SVN, though it can be improved much more)
>
>   
Qt provides a plugin framework for this sort of thing, allowing us to
develop and release all the stat mining functionality separately from
the client. It does this in a platform-independent way, using dll's for
plugins on windows and .so for *nix. To be truthful, I have no idea how
libraries are linked at runtime on mac, but supposedly Qt handles this
as well. ;)
>
> I am not familiar with Qt other than that it a tool to develop GUI,
> but have you taken a look at the ADAPTIVE Communication Environment
> (ACE) C++ libraries (http://www.cs.wustl.edu/~schmidt/ACE.html)? It
> has been developed written specifically to make platform dependent
> things like network programming, process and thread creation, inter
> process communications portable across OSes. There is also the BOOST
> libs (asio.sorurceforge.net), but this is not as wide as ACE in terms
> of system programming interfaces provided. Let me know what you think
> of this.
>   
I am familiar with boost but haven't used it, and I used ACE on another
project. As for Qt, while it's very good for GUIs, the libraries are
modularly linked, and it provides a lot of other platform-independent
functionality (process and thread management, ipc, etc). Personally, I'm
partial to Qt, simply because I've only used ACE once and boost not at
all, while I'm very experienced with Qt.
> But one thing is using these tools wont help in getting the
> performance statistics because they *are* highly OS specific. we cant
> make that portable.
>   
Agreed. There's only so much a toolkit can do without being a beast. ;)
At least a cross-platform toolkit removes a lot of the redundancy from
our development effort and time. What needs to be OS-specific will just
have to be. Another benefit to Qt, though, is that it provides a
platform-independent build process with convenient conditional
compilation (without #ifdefs and whatnot) for a lot of os/compiler
combinations.
> Do let me know you opinions.
>   
These are they. ;)

I've got most of the code you wrote converted to Qt, with a lot more
other code in place (threaded server, plugin framework, etc). I was
waiting to get a working copy before committing to sf svn, but if you
want, I can commit most of what I have for you to look at.

~Darrik Mazey

Darrik,

Thanks for your comments. The server and client I have written in C is
portable across all *nix, and with a couple of #ifdefs, will work on
windows too. But I agree with you on using a cross-platform tool
because it eases maintenance.

> B) Instead of writing some plugins and having some built-in
> functionality of the client, I suggested we write a client that simply
> loads whatever plugins it's told to.

the client and the statistics gathering programs are very much
separate (please see the server and client code in the 'backend'
directory in 'test' on sf SVN, though it can be improved much more)

>
> A cross-platform toolkit (while it may not support -every- possible
> platform) will allow us to save dev time on the major ones, and if need
> be, we can still #ifdef the hell out of some code for legacy systems and
> platforms not supported by our toolkit.  Qt seems like a good choice to
> me (especially since I have a lot of it written already), and it
> provides a build system with tools for abstracting cross-platform
> compilation.
>

I am not familiar with Qt other than that it a tool to develop GUI,
but have you taken a look at the ADAPTIVE Communication Environment
(ACE) C++ libraries (http://www.cs.wustl.edu/~schmidt/ACE.html)? It
has been developed written specifically to make platform dependent
things like network programming, process and thread creation, inter
process communications portable across OSes. There is also the BOOST
libs (asio.sorurceforge.net), but this is not as wide as ACE in terms
of system programming interfaces provided. Let me know what you think
of this.

But one thing is using these tools wont help in getting the
performance statistics because they *are* highly OS specific. we cant
make that portable.

Do let me know you opinions.

Thanks,
Ananth

Ananth:

My suggestion was two-fold and thus:

A) Instead of writing separate client code for each platform we propose
to run the client and server on, we use a cross-platform toolkit to
cover the vast majority of common platforms.  I suggested Qt because I
am very experienced with it.  However, there are others available, and
I'm sure most of them will do what we need them to do.  I set about
working out a Qt-based solution and have almost all the previous code
working with Qt, as well as some groundwork for abstracting the *nix
daemon process or win32 service to provide one cross-platform interface.

B) Instead of writing some plugins and having some built-in
functionality of the client, I suggested we write a client that simply
loads whatever plugins it's told to.  This keeps the interface uniform,
as well as allowing separate maintenance on even the "built-in"
functionality, keeping that separate from the client proper.  To this
end, I've also laid down some code in that direction.

A cross-platform toolkit (while it may not support -every- possible
platform) will allow us to save dev time on the major ones, and if need
be, we can still #ifdef the hell out of some code for legacy systems and
platforms not supported by our toolkit.  Qt seems like a good choice to
me (especially since I have a lot of it written already), and it
provides a build system with tools for abstracting cross-platform
compilation.

Let me know what you (or anyone else) thinks of this.  I've put quite a
bit of code in place already, and I don't want to go much further if
we're not all in agreement.  I should have a working copy of both server
and client in the svn repository by early next week (if not this weekend).

~Darrik Mazey

Ananth wrote:
> Message body follows:
>
> Hey Darrik,
> I am Ananth, one of the developers in Armory. Danilo was
> asking me to get in touch with you about using QT. Can you
> mail me whatever you wanted to discuss? Or if you could come
> online on #armory in the next 1 hour, that would be great.
> my email is ana...@gm...
>
>   

pah

Hello list.

I have a few questions regarding the planning of this project as it stands.

First and foremost, until we have something concrete in cvs, is it
possible to post a list of who's working on what, so we aren't
overlapping and responsibilities are clear?  Or if someone started a
responsibilities thread on the forum, and everyone posted with what they
were working on, that would work too.  ;)

Another question comes from Bico's description on the forum (see the
post "Plugins Development" :
http://sourceforge.net/forum/forum.php?thread_id=1751978&forum_id=689813),
the client will have a few built-in functions (disk usage, cpu, memory,
etc) and then rely on plugins for any other functionality.  My thought
was that it makes more sense to develop the client to simply load
whatever plugins and execute them, rather than mixing built-ins and
plugins.  We could write the client as just a plugin framework, and then
write the disk usage, cpu, and memory plugins.  This seems like the
cleanest method to me, and I wasn't sure if I just misunderstood the
post and this was what Bico had in mind.  This gives the obvious
advantage of not having to have two different systems for the client.
It's only function would be to load its plugins, and the plugins could
do the rest.  This would make upgrading the "builtin" functionality of
disk usage and whatnot a lot easier.

These are my thoughts.

~Darrik

Hello list.

I have a few questions regarding the planning of this project as it stands.

First and foremost, until we have something concrete in cvs, is it
possible to post a list of who's working on what, so we aren't
overlapping and responsibilities are clear?  Or if someone started a
responsibilities thread on the forum, and everyone posted with what they
were working on, that would work too. ;)

Another question comes from Bico's description on the forum (see the
post "Plugins Development" :
http://sourceforge.net/forum/forum.php?thread_id=1751978&forum_id=689813),
the client will have a few built-in functions (disk usage, cpu, memory,
etc) and then rely on plugins for any other functionality.  My thought
was that it makes more sense to develop the client to simply load
whatever plugins and execute them, rather than mixing built-ins and
plugins.  We could write the client as just a plugin framework, and then
write the disk usage, cpu, and memory plugins.  This seems like the
cleanest method to me, and I wasn't sure if I just misunderstood the
post and this was what Bico had in mind.  This gives the obvious
advantage of not having to have two different systems for the client.
It's only function would be to load its plugins, and the plugins could
do the rest.  This would make upgrading the "builtin" functionality of
disk usage and whatnot a lot easier.

These are my thoughts.

~Darrik

2007	Jan	Feb	Mar	Apr	May	Jun (12)	Jul	Aug (1)	Sep	Oct	Nov	Dec
2008	Jan	Feb	Mar	Apr	May	Jun (1)	Jul	Aug	Sep	Oct	Nov (1)	Dec

armory-devel Mailing List for Armory

armory-devel — Armory Developers