gchartman-devel

[gchartman-devel] Re[2]: todo

[<we...@le...>]

>
>
> I didn't start a real TODO list yet, because I have all
> kinds of ideas flying through my head, but I don't know
> in what order I will try to implement them.
>
> Some of the ideas that I have are:
>
> Near future:
> - add a function do_split to modules, such that a trendline
>   can rescale itself when a split is done -> automatic
>   adjustment after split.
> - make it possible for modules to be parents of other modules.
>   What I would like to have is a moving average of an RSI
>   or Momentum. The list module instances left of the chart
>   will become a tree, and when adding an instance, one is
>   asked to select a parent. It might also be possible to
>   just drag a moving average in that list from the main chart
>   to another instance which has the 'parent??capability.
>
> Less near future:
> - add a database editor, such that quotes can be entered
>   or corrected manually, including stuff like cleanly
>   deleting a stock from the database or adding a new stock.
> - add internet quote retrieval, preferably using libstocks
>   (http://libstocks.sourceforge.net) with adequate grouping
>   possibilities (download quotes for these stocks from this
>   site, either "snapshots" or complete historical quotes).
>   This will have to wait until I get flatrate internet access.
> - add a lot of modules (as soon as the interface is stable).
>   I have some books describing lots of indicators and analysis
>   techniques like Fibonacci arcs and so on.
>
> Far future:
> - Use gnome-print to print out charts.
> - Use gnome-db for compatibility with different databases.
> - Change the database format to something saner.
> - UI polishing.
> - Add esoteric analysis possibilities like cyclical analysis
>   or neural network based price predictions.
>
> I probably forgot some things... Anyway, sooner or later a
> code cleanup will have to be done. I have some ugly code
> because of bad habits (variable and function naming mainly).
> Also my spare time left for coding is not very large. I'm
> currently studying, and I only have 1 or maybe 2 hours
> per day left for coding.
>
> Anyway, tell me about your ideas!
>
> Christian
>
>

Christian, thanks for the list.. Some of the things I was looking at ,
as in modules, look like they are in your short term list. Most of
the things I wanted to do centered around the database .
Setting gnome-db fto use other databases, even though
I mainly use MySQL is something I think I would like to
work on, along with the database format.

I was thinking about the database format, and I was thinking
a flexible formatter or compiled in modules might be better.
I know a few people who have access to stock data in SQL
format, but with a different format. I think it may be
better for gchartman if it were flexible so that a person
could run from his Oracle server at work, that has a much more
complex data structure. And adding open, high, and low to the database
fields.

I have some code for a database editor/viewer I can put
in, but that might be something to look at after the database
stuff is looked at.

Greg

--
wenzi  http://www.igerg.com

We are Pentium of Borg. Division is futile. You will be approximated.
(seen in someone's .signature)

[gchartman-devel] Re[3]: todo

[<chr...@is...>]

> Christian, thanks for the list.. Some of the things I was looking at ,
> as in modules, look like they are in your short term list. Most of
> the things I wanted to do centered around the database .
> Setting gnome-db fto use other databases, even though
> I mainly use MySQL is something I think I would like to
> work on, along with the database format.

Yes, a generic database interface is a must. Gnome-db is probably the
best choice right now, because they provide gnome integration and
nifty widgets, and last but not least they support a lot of database
systems (they have a unixODBC driver, and unixODBC in turn knows
lots of databases). Problems that I see are: Gnome-db is currently
quite a moving target, and it relies on other quickly evolving
technologies (like bonobo, oaf). This makes compilation harder,
since you have to be up to date with all these things. Since I'm
rather bandwidth impaired (currently), I cannot afford to do cvs
checkouts from all those projects. Lacking documentation and
unstable snapshots don't really make things easier. I also
use some sql functions from mysql which might not be provided by
gnome-db, like soundex(), but that should not be a great problem
since they are not critical.
 
> I was thinking about the database format, and I was thinking
> a flexible formatter or compiled in modules might be better.
> I know a few people who have access to stock data in SQL
> format, but with a different format. I think it may be
> better for gchartman if it were flexible so that a person
> could run from his Oracle server at work, that has a much more
> complex data structure.

The current database format is a mess, mainly because I kludged it
together instead of designing it. I think it would be good to be able
to let the user adjust gchartman to his database. But the flexibility
that can be supported is limited by complexity. I can very well imagine
to have some things configurable, but other things not. As an
example, I would assume that all stocks with all their data are stored
in a singe table, identified by an ID column. Another table would
contain the ID, stock name, ticker symbol, etc. Now, one could let the
user adjust the name of the tables and columns used to look up data,
but the basic layout would stay the same. To support *any* kind of
database layout, one would probably end up with 1 query per data quantum,
which would make things flexible but slow. And the user probably would
have to supply the queries.

I'm not quite sure I understand what you mean by compiled in modules.
Do you mean to have different modules supporting different kinds of
database layouts? I think one should be able to adapt to a database
layout without writing code.

Another point is: *what* should be stored in the database? Currently,
everything goes into the database, videotext settings, chart configurations,
category contents etc. At some point I think it will be better to
put this data into some xml file somewhere in the users home directory.
This would be better for multi-user environments, it would not pollute
the database, and the database could even be read-only. Which brings me
to my next idea: A 'read only´ mode that could be selected at database
login time, guaranteeing that the database will not be modified.
It would be useful if one wanted to use a database at work (at a broker
house for example) without taking any risks or if one simply didn't have
the privileges.

> And adding open, high, and low to the database
> fields.

I know that these should be there. When I started populating
my database I had no data for them available, so I didn't include them
all :( stupid me
 
> I have some code for a database editor/viewer I can put
> in, but that might be something to look at after the database
> stuff is looked at.

Yes, ofcourse. There's also the gnome-db database browser widget,
which seems to be able to do some things too. Anyway, first the
database layout will have to be redesigned. I think that I
probably won't bother now to store a tree of chart items in the
database, but I will move configuration data to xml files soon.
I will modify the modules such that they store their information
in a readable way in an xml file, and not as blobs (don't know if
that is even possible).

Phew, long mail. In conclusion, I think we can start to discuss/design
the database layout, because adapting to a new layout will only get
harder the longer one waits. In my opinion, a new layout should be
done first, and then a flexible way to support different layouts
can be added. I think configuration data should not be included in
a new layout, but stored in xml files. Currently I'm making the
code more readable by gnu-ifying function and variable names.

By the way, the 'channel´ module seems to be rather broken. It doesn't
behave the way I want it to, so you better not use it too much.

No more time now,

Bye

Christian

Re: [gchartman-devel] Re[3]: todo

[<we...@ra...>]

>
> Yes, a generic database interface is a must. Gnome-db is probably the
> best choice right now, because they provide gnome integration and
> nifty widgets, and last but not least they support a lot of database
> systems (they have a unixODBC driver, and unixODBC in turn knows
> lots of databases). Problems that I see are: Gnome-db is currently
> quite a moving target, and it relies on other quickly evolving
> technologies (like bonobo, oaf). This makes compilation harder,
> since you have to be up to date with all these things. Since I'm
> rather bandwidth impaired (currently), I cannot afford to do cvs
> checkouts from all those projects. Lacking documentation and
> unstable snapshots don't really make things easier. I also
> use some sql functions from mysql which might not be provided by
> gnome-db, like soundex(), but that should not be a great problem
> since they are not critical.

I agree on the gnome, and helix code parts. I download
all the Helix code libs, etc. ( bonobo et al. ) Not only is
it a moving target, I don't think it is that stable. My Redhat
6.2 system wierded out on it.

gnome-db , is nice, but I am not sure if it is something
you can easily use without CVS , and a decent sized pipe.

>
> The current database format is a mess, mainly because I kludged it
> together instead of designing it. I think it would be good to be able
> to let the user adjust gchartman to his database. But the flexibility
> that can be supported is limited by complexity. I can very well imagine
> to have some things configurable, but other things not. As an
> example, I would assume that all stocks with all their data are stored
> in a singe table, identified by an ID column. Another table would
> contain the ID, stock name, ticker symbol, etc. Now, one could let the
> user adjust the name of the tables and columns used to look up data,
> but the basic layout would stay the same. To support *any* kind of
> database layout, one would probably end up with 1 query per data quantum,
> which would make things flexible but slow. And the user probably would
> have to supply the queries.
>

I was not thinking about supporting ANY kind of database, I was thinking
about ways to make it more flexible. If you always look at end of day prices,
that's fine. But look at some tick data suppliers, where they only supply the
price difference from the previous tick.  But more to the point, I wasn't
speaking of ANY format, but more of a way to allow people to easily use
their existing databases. For example, after you do a query in gchartman,
it assumes the stock_name is in column[0].

I was just thinking of how to be flexible, the ability to support every
database and
supporting MOST users are two separate things. Looking for MOST users would
be better.


>
> I'm not quite sure I understand what you mean by compiled in modules.
> Do you mean to have different modules supporting different kinds of
> database layouts? I think one should be able to adapt to a database
> layout without writing code.
>

I should not have used the word "modules", my mistake.  what I
thought about was a data access set of routines. on the outside
generic functions, such as get stock list() for functions like
init_stock_combo() and others.  If your particular rig
was special, you could change what was needed there. Since
it would be a C file, it would have to be compiled it.

also , it would make an easier upgrade path to gnome-db


>
> Another point is: *what* should be stored in the database? Currently,
> everything goes into the database, videotext settings, chart configurations,
> category contents etc. At some point I think it will be better to
> put this data into some xml file somewhere in the users home directory.
> This would be better for multi-user environments, it would not pollute
> the database, and the database could even be read-only. Which brings me
> to my next idea: A 'read only??mode that could be selected at database
> login time, guaranteeing that the database will not be modified.
> It would be useful if one wanted to use a database at work (at a broker
> house for example) without taking any risks or if one simply didn't have
> the privileges.
>

I think global settings should go in the database. However , I think a global
xml file would be better, I don't know how high that is on the priority list.
User configuration files should DEFINITELY go in the users home
directory.

I would think that making a .gchartman directory in the users
directory would probably be best.

The read only idea is a good one. My database here allows certain log ins
only in read mode. they don't have the permissions to modify the database.
That is on the database side. The problem with that is that the user
interface must reflect the certain things are unchangeable. Putting it
in gchartman itself instead of the database does eliminate this problem.

I am wondering if it gchartman would have to check to see if it had
write privileges, and how would it go about it.

>
> > And adding open, high, and low to the database
> > fields.
>
> I know that these should be there. When I started populating
> my database I had no data for them available, so I didn't include them
> all :( stupid me
>
>

I have plenty of US data. let me know if you want any :-)

>
>
> Yes, ofcourse. There's also the gnome-db database browser widget,
> which seems to be able to do some things too. Anyway, first the
> database layout will have to be redesigned. I think that I
> probably won't bother now to store a tree of chart items in the
> database, but I will move configuration data to xml files soon.
> I will modify the modules such that they store their information
> in a readable way in an xml file, and not as blobs (don't know if
> that is even possible).
>
> Phew, long mail. In conclusion, I think we can start to discuss/design
> the database layout, because adapting to a new layout will only get
> harder the longer one waits. In my opinion, a new layout should be
> done first, and then a flexible way to support different layouts
> can be added. I think configuration data should not be included in
> a new layout, but stored in xml files. Currently I'm making the
> code more readable by gnu-ifying function and variable names.
>

def. a long email. but good stuff...

I would like to discuss the database layout also, and I think the sooner
the better. I think we should look at how it will make programming easier
and stay on the path to gnome-db. but as for flexible ways, i wonder if
we can also just store it in the xml file. what I think , is that most data
formats
will be very , very similar. the major differences will be in data format
, column position, time format, and SQL calls. I think a good part could
be abstracted out..

what xml parser are you thinking about using ?

what do you mean by 'gnu-ifying' names ?

>
> By the way, the 'channel??module seems to be rather broken. It doesn't
> behave the way I want it to, so you better not use it too much.
>
>

thanks.

--
wenzi   we...@we... http://www.igerg.com

Him:    "Your skin is so soft.  Are you a model?"
Her:    "No,"  [blush]  "I'm a cosmetologist."
Him:    "Really? That's incredible... It must be very tough to handle
        weightlessness."
                -- "The Jerk"

[gchartman-devel] Re[4]: todo

[<we...@we...>]

> FROM: ist.luDATE: 10/13/2000 10:34:56SUBJECT:  Re[2]: [gchartman-devel] Re[3]: todo
> > I agree on the gnome, and helix code parts. I download
> > all the Helix code libs, etc. ( bonobo et al. ) Not only is
> > it a moving target, I don`t think it is that stable. My Redhat
> > 6.2 system wierded out on it.
> >
> > gnome-db , is nice, but I am not sure if it is something
> > you can easily use without CVS , and a decent sized pipe.
>
> for me, gnome-db (0.1.0) is not stable at all. The included applications
> regularly seg-fault, especially anything using their db browser widget.
> As long as their applications crash I`m not even trying to use it.
>

true, true. I do use the table widget and that seems pretty stable
except for
cutting and pasting ops are not working.



> > I was not thinking about supporting ANY kind of database, I was thinking
> > about ways to make it more flexible. If you always look at end of day prices,
> > that`s fine. But look at some tick data suppliers, where they only supply the
> > price difference from the previous tick.
>
> That`s very interesting. Basically a chart can be displayed of any data stored
> in QuoteV (which I renamed to quote_vector). So it should be possible to support
> different kinds of data sources. If everyone had free broadband internet access,
> why store any data at all? It`s all on the net! Just get it from a website/data
> provider
> when it`s needed. Another possibility: support streaming data. A piece of
> code could use a reverse-engineered livecharts.com protocol for example, and
> provide a new quote_vector once a minute. By the way, recently I stumbled over
> the FreeMarket project at SourceForge. It`s pretty interesting, they use a Corba
> based architecture and have streaming data support.
>

exactly ! Say I have a custom indicator I wrote or say the output of a
neural net program, etc, etc.And then stored it in my MySql database.
I then could use it in gchartman.

Supporting streaming data would require a change in QuoteV to suppot a
time marker ( as opposed to end of day ). Other than that everything
else is the same. Instead of the date, followed by price, it would be
time+date followed by price.

I also saw the free market project. it looks like they want to make a
linux clone of Omega Tradestation. It is interesting.

> > But more to the point, I wasn`t
> > speaking of ANY format, but more of a way to allow people to easily use
> > their existing databases.
>
> I think it will be sufficient to have configurable column and table names,
> and to make some features optional, so that one can fall back to end-of-day
> data only if nothing else works. It may also be possible to use sql views to
> increase compatibility.
>

That is pretty much what I meant. configurable column and table names
and the SQL staments to retrieve the data. If you have a relational
database , you can use alias in the SQL statments to change the column
names, i believe. I will have to check.


> > I should not have used the word "modules", my mistake.  what I
> > thought about was a data access set of routines. on the outside
> > generic functions, such as get stock list() for functions like
> > init_stock_combo() and others.  If your particular rig
> > was special, you could change what was needed there. Since
> > it would be a C file, it would have to be compiled it.
> > also , it would make an easier upgrade path to gnome-db
>
> Yes, I also thought about it. And it would not be a big deal to put these
> data access routines into real modules (which I think would be good).
>

true.

> > I think global settings should go in the database. However , I think a global
> > xml file would be better, I don`t know how high that is on the priority list.
> > User configuration files should DEFINITELY go in the users home
> > directory.
>
> Currently I`m working on 2 things:
>
> turning the list of chart instances into a tree & making modules generate a new
> (or modifying the) quote_vector.
>
> and
>
> putting configuration data into xml files in the users home directory,
> 1 for global configs, 1 for local configs, 1 for categories, 1 for
> videotext settings,
>
> So the priority for getting configuration data out of the database is pretty
> high.
>



> > I would think that making a .gchartman directory in the users
> > directory would probably be best.
>
> It`s already there :) Videotext pages are stored in ~/.gchartman/videotext
> by alevt-cap. Once capturing is done they are read from there and parsed by
> gchartman.
>

Cool. But I don't have that directory. I don't use videotext., so it
probably never got created.

> > The read only idea is a good one. My database here allows certain log ins
> > only in read mode. they don`t have the permissions to modify the database.
> > That is on the database side. The problem with that is that the user
> > interface must reflect the certain things are unchangeable. Putting it
> > in gchartman itself instead of the database does eliminate this problem.
>
> I think a simple `read-only` toggle in the login window should do the trick.
>
> > I am wondering if it gchartman would have to check to see if it had
> > write privileges, and how would it go about it.
>
> Hm... easiest way: just try it and check for success. not very elegant...
>

not elegant, but works. Just remind them that they don't have write
privs.

> > I would like to discuss the database layout also, and I think the sooner
> > the better. I think we should look at how it will make programming easier
> > and stay on the path to gnome-db. but as for flexible ways, i wonder if
> > we can also just store it in the xml file.
>
> Do you mean the data itself? I`ve thought about it. There`s two possibilites:
> - rely on gnome-db. AFAIK they want to provide a way to export any supported
>   database into an xml file.
> - write all the necessary data access routines, which should not be *that* hard,
>   once all data accessing code is put away into functions.
>
> I think the db layout is now rather important, too.
>

No, I don't mean about exporting xml data. Most of the time, the data
will be going INTO the database, not exporting. If someone really needs
to export in xml, maybe that exercise should be left for themsleves.

I meant define what the data access routine interface should be. I can
write them, but I wouldn't want to write something that would be
incompatible with what others are doing. I think some thought needs to
go into defining what the interface should be, before writing them.

>
> > what I think , is that most data
> > formats will be very , very similar. the major differences will be in data
> format,
> > column position, time format, and SQL calls. I think a good part could
> > be abstracted out..
>
> Hopefully.
>
> > what xml parser are you thinking about using ?
>
> I`ve experimented with libxml aka gnome-xml. The api is pretty ok, and it has
> one big advantage: Just about everyone has it. It also seems to be fast enough.
> Of course any data has to be sprintf`ed to strings prior to saving, and it
> must be parsed upon loading, but I think that is inevitable and still faster
> than a (potentially slow) database access.
>

I still use Jim Clark's expat. Probably because it is so old, and there
wern't too many parsers around back then. As if it was that long ago. I
think I will have to muck about with libxml, see what it is like.

I think speed should not really be that much of an issue. For xml , I
think I would go for ease of use, availabilty and  stability over speed.
xml is xml.

> > what do you mean by `gnu-ifying` names ?
>
> Turning variable names like `QuoteV` into names like `quote_vector`. It`s
> longer,
> but it`s also clearer. All variable and function names are now lowercase,
> underscore-
> seperated word sequences. Types are first-letter-uppercase-no-separator (like
> ChartInstance).
> This is the same naming scheme as in gtk. It also means that if you wrote any
> code,
> you`ll have to change some things probably.
>

I ususlly use that style for filenames. So I wouldn't have to make any
change. I got my shop to standardize on that form. The advantages of
writing the style guide ;-)

wenzi

[gchartman-devel] Re[5]: todo

[<chr...@is...>]

[snip]

> Supporting streaming data would require a change in QuoteV to suppot a
> time marker ( as opposed to end of day ). Other than that everything
> else is the same. Instead of the date, followed by price, it would be
> time+date followed by price.

Or maybe one could have multiple threads and some sort of inter-process
communication
to signal that new data has arrived and that a new quote_vector is ready for
display.
 
> I also saw the free market project. it looks like they want to make a
> linux clone of Omega Tradestation. It is interesting.

I don't know Tradestation, guess it's some piece of professional software.
 
[snip]
 
> That is pretty much what I meant. configurable column and table names
> and the SQL staments to retrieve the data. If you have a relational
> database , you can use alias in the SQL statments to change the column
> names, i believe. I will have to check.
 
Yes, I think aliases are possible. Having user-adjustable sql statements is ok
as long as there are not too many, and sensible defaults have to be provided of
course.

[snip]
 
> > > I would think that making a .gchartman directory in the users
> > > directory would probably be best.
> >
> > It`s already there :) Videotext pages are stored in ~/.gchartman/videotext
> > by alevt-cap. Once capturing is done they are read from there and parsed by
> > gchartman.
> >
> 
> Cool. But I don't have that directory. I don't use videotext., so it
> probably never got created.

Right. I modified this now: ~/.gchartman is now always created and
~/.gchartman/videotext
only when you actually use videotext.

[snip]
 
> > > I would like to discuss the database layout also, and I think the sooner
> > > the better. I think we should look at how it will make programming easier
> > > and stay on the path to gnome-db. but as for flexible ways, i wonder if
> > > we can also just store it in the xml file.
> >
> > Do you mean the data itself? I`ve thought about it. There`s two
possibilites:
> > - rely on gnome-db. AFAIK they want to provide a way to export any supported
> >   database into an xml file.
> > - write all the necessary data access routines, which should not be *that*
hard,
> >   once all data accessing code is put away into functions.
> >
> > I think the db layout is now rather important, too.
> >
> 
> No, I don't mean about exporting xml data. Most of the time, the data
> will be going INTO the database, not exporting. If someone really needs
> to export in xml, maybe that exercise should be left for themsleves.
> 
> I meant define what the data access routine interface should be. I can
> write them, but I wouldn't want to write something that would be
> incompatible with what others are doing. I think some thought needs to
> go into defining what the interface should be, before writing them.

Yes, one should really think about things before coding away. Wish I had known
that 1 year ago :) Actually I knew it but I didn't care :)
But it won't be long until I will be brainwashed into not knowing any other way,
because I now have lots of courses on these things.

[snip]

> > > what xml parser are you thinking about using ?
> >
> > I`ve experimented with libxml aka gnome-xml. The api is pretty ok, and it
has
> > one big advantage: Just about everyone has it. It also seems to be fast
enough.
> > Of course any data has to be sprintf`ed to strings prior to saving, and it
> > must be parsed upon loading, but I think that is inevitable and still faster
> > than a (potentially slow) database access.
> >
> 
> I still use Jim Clark's expat. Probably because it is so old, and there
> wern't too many parsers around back then. As if it was that long ago. I
> think I will have to muck about with libxml, see what it is like.
> 
> I think speed should not really be that much of an issue. For xml , I
> think I would go for ease of use, availabilty and  stability over speed.
> xml is xml.

I don't know expat, but then I'm new to xml anyway. libxml seems pretty
straight-
forward to me, and it has the availability advantage.
 
Currently I'm working on the tree-of-chart-items-stuff, which means quite a few
changes,
and on the xml stuff. It will take 1-2 weeks at least until the next release,
because I'd
like to have *all* configuration data out of the database for it. But I can
upload
an intermediate version in a few days if you want it, or I can mail it to you.

Christian

[gchartman-devel] Re[6]: todo

[<we...@we...>]

> > Supporting streaming data would require a change in QuoteV to suppot a
> > time marker ( as opposed to end of day ). Other than that everything
> > else is the same. Instead of the date, followed by price, it would be
> > time+date followed by price.
>
> Or maybe one could have multiple threads and some sort of inter-process
> communication
> to signal that new data has arrived and that a new quote_vector is ready for
> display.
>

Here is an idea, one process justs collects data and places it in the
data base. gchartman updates once a min. I am not even sure if we need
IPC ( inter process communication ) if it just updates once a min.
Wasteful yes, but easy.

> I don`t know Tradestation, guess it`s some piece of professional software.
>

Yeah, it is a real popular/well known PC software. They are ( FreeMarket
) seem to be writing a clone of their backtesting language.

[snip]


>
>
>
>
> Yes, one should really think about things before coding away. Wish I had known
> that 1 year ago :) Actually I knew it but I didn`t care :)
> But it won`t be long until I will be brainwashed into not knowing any other way,
> because I now have lots of courses on these things.
>

Most of the time, I write the first one to throw away. Not really to
throw away. Sometime you need to write something just to help you get
ideas on what you want to design. So I build, see what I like , and then
rebuild after I became more familiar with what I want. Still using most
of the code I wrote from the first one.

>
> Currently I`m working on the tree-of-chart-items-stuff, which means quite a few
> changes,
> and on the xml stuff. It will take 1-2 weeks at least until the next release,
> because I`d
> like to have *all* configuration data out of the database for it. But I can
> upload
> an intermediate version in a few days if you want it, or I can mail it to you.
>
> Christian
>

I am going on vaccation for a couple of weeks, so I will be on my laptop
instead of my development machine. I will be on email , but not
compiling, etc.

wenzi