Local Storage/Caching of data

Peter Kane
  • Peter Kane

    Peter Kane - 2011-12-02

    Is there a progamming architecure philosophy for what data is stored locally and what data is loaded on request?
    eg. Manually entered dividends are stored in xml files. What should be done if there were a source of dividend data on-line?
    Or a source of fundamental information (PE ratio, etc)?

    If more data is to be allowed locally, how should it be stored?
    Options seem to be more xml files, a run-time database like hsqldb, or a database server: derby, oracle etc…

    There is at least one feature request for cached price data.

  • yccheok

    yccheok - 2011-12-03

    Hi peterjkane,

    Sorry. I don't quite get what you are trying to express. However, let me make some clarification on the current stored data format in JStock.

    1) Currently, we are storing most of the information (Dividends, Deposit, Buy information, Sell information, Stock database…) in XML format. However, our long term strategy is convert all these data into plain text CSV format. This is because, we want the information can be exchanged easily in between android and desktop. I found the data exchange in between android and desktop, will be much more easier, if we use the data in CSV format instead of XML.

    2) I have no plan to use a true database system like postgresql, mysql, … This is because I do not see such need to full fill user requirement. All the requirement can be achieved by using data stored in flat file format.

    3) It is quite difficult to get dividend information on-line. As, I cannot find a reliable source to obtain such information. I will rather let the users key in manually.

  • Peter Kane

    Peter Kane - 2011-12-04

    Pardon my questions and lack of a full understanding of jstock. I'd like to contribute more coding and some documentation.
    I'm on a steep learning curve.

    As the moment I am trying to explore what information jstock stores, where, and in what format. jstock as an open, widely used system has some differences to my experiences in coding for personal or corporate use.

    2) if file-based local storage is being used, then both xml and csv have advantages. I'm OK with csv, particularly if you think it supports an android version better - I have an android phone and am interested in that product.

    1) WRT dividends (particularly Australian ASX), maybe that should be a separate thread. It appears that in jstock dividend payments are stored as total dollar amounts for each portfolio only.
    The Australian system  of dividends recognises tax that a company has paid, and the dividend amount is broken up into a) franked (tax-paid) and b) unfranked (tax not paid) amounts and also has a c) franking credit amount (which you can deduct from your personal tax), as well as a d) witholding tax amount.
    So, to use jstock to keep Australian tax records, extra information is required against each dividend.
    I think there should be a way to do this, but it means extra fields against the dividend record. Are you OK in principle with this? If so I can provide a proposal.
    (I can also send an example dividend statement if you like, but I don't know how to attach it to this posting)

    3) WRT fundamental data (extra topic, also maybe should be in a separate thread) - My other package screen-scraped and obtained dividend and other information (PE, DE, other fundamental indicators) from a variey of sources, and stored them for local analysis. I think it would be good if jstock could do this. Maybe allow people to enter this information manually, and allow them to load information from csv/xml as well. I used storage in a database, and had SQL statements to generate local reports.
    I use this to analyse stocks in my watchlist

    4) (related topic) - as a public open-source system do you need approval from the data provided (yahoo, google etc.,)? I have been downloading information from the ASX and other sites in a variety of formats (html, csv, etc), and loading it into my system. Is this OK for jstock? alternatives might be to allow people to do this themselves but provide a loader for csv/xml, or maybe to develop and support some sort of plug-ins for data sourcing from the internet. This might be a good architectural development for prices as well.

    - Peter

  • yccheok

    yccheok - 2011-12-06

    From legal point of view, please refer to http://jstock.sourceforge.net/help_faq.html#legal. However, I would say, that is only for reference purpose. As I doubt the replied email from customer service, can act as a lawful document to proof JStock is using legal way to obtain its stock data.

    If you are interested in contributing to coding, perhaps you want to consider some tickets at bug tracker. There are few new items coming in recently.