Re: [Xbrlapi-developer] Difficulties working with the rendering example

[Matthew D. <ro...@gm...>]

Hi Geoff,

I think that the problems I am having with my large datastore are caused by
aspect models and linkroles being based on stores, not instances.  As a
result, when I invoke these functions, the program loads every relationship
in the data store, and not just those for a particular instance.
 Understandably, this crushes the system.

Is there a way to build aspect models and use linkroles from a particular
instance only?  I could not find a way in the documentation, but I could
easily have missed it.  If not, is there an easy way to create a kind of
"virtual store", which pulls documents and persisted relationships for a
particular instance out of the main store, but can be treated as a store
object?  If not, I will look into how to do this, and would appreciate any
guidance that you might have.


Regards,
Matt

On Mon, Jul 11, 2011 at 5:29 PM, Matthew DeAngelis <ro...@gm...>wrote:

> Hi Geoff,
>
> Using the new material with a fresh, single-item BDBXML database works as
> expected.  Using the new material with my existing database (which contains
> over 500 instance and related documents) continues to have (some) of the
> same problems.  As such, I'm going to guess that the slowdown and tendency
> for memory overruns is due to database performance.  I will try rebuilding
> my database again to see if there is anything wrong with it.
>
> If it turns out that the size is the problem, I will have to look into
> tuning BDBXML databases.  Have you noticed scaling issues with these
> databases in the past?  From your experience, do you think that an eXist
> database would fare better with a large volume of data?
>
> Thanks for helping me troubleshoot.  Now that I can at least work with a
> single document quickly, I am starting to make progress.
>
>
> Regards,
> Matt
>
> On Sun, Jul 10, 2011 at 12:41 AM, Geoff Shuetrim <ge...@ga...> wrote:
>
>> Matthew,
>>
>> I have tried running the Rendering example using a fresh BDBXML data
>> store.  The data took a few minutes to load (less than 10) and then then
>> rendering process was done in about 1-2 minutes.  I used 2 GB of memory and
>> did not come up against any constraints.
>>
>> Note that the freemarker template does need a couple of small changes to
>> get it to work with the updated freemarker library and to reflect the change
>> in the name of the serializeToString method of the Store interface.  To get
>> the fixed version of the Freemarker template, you can update from SVN or
>> just download it directly from *http://tinyurl.com/3aubjlj
>>
>> *The rendering result is in a file in SVN "result.html" that is also
>> accessible from the SVN browse facility.
>>
>> Without more information I am not going to be able to troubleshoot your
>> problem.  Give it a go with the revised freemarker template in SVN and let
>> me know if the problems persist.  If they do, you might need to do some of
>> your own timing evaluations to point us in the right direction.
>>
>> Regards
>>
>> Geoff Shuetrim
>>
>>
>>
>>
>> On 9 July 2011 05:50, Matthew DeAngelis <ro...@gm...> wrote:
>>
>>> Hi Geoff,
>>>
>>> Following your suggestion, I have started to work with aspects and
>>> dimensions.  I have been using your rendering example
>>> (org.xbrlapi.data.bdbxml.examples.render, Run.java) as a template, since it
>>> uses fact sets and aspect values in a number of ways.  After some problems
>>> with my own code, I attempted to run your Run.java example as is, on
>>> http://www.sec.gov/Archives/edgar/data/93751/000119312510036481/stt-20091231.xml,
>>> which is in the data store.  It takes a long time to run (upwards of a half
>>> hour), and fails with an Out of Memory Error after line 394 (once it starts
>>> building childConcepts).  My memory is set to 2G (out of 4G system memory),
>>> which I should think would be sufficient for a single instance document.
>>>
>>> As mentioned above, I have also been running into problems with my own
>>> code, attempting to load all of the Facts in a particular instance into a
>>> FactSet.  Specifically, if I retrieve all Facts from an instance, and
>>> attempt to use addFacts() to add those Facts to the FactSet, I get an Out of
>>> Memory Error.  I have taken to loading the Facts one by one into the
>>> FactSet, which bypasses the memory error, but is extremely slow: each Fact
>>> takes many minutes (while writing, I have watched one take over 10 minutes).
>>>  This does not make sense to me: if all of the Facts can be retrieved from
>>> the instance in a few seconds, I do not see why adding the Facts to the
>>> FactSet should take so long.  In an attempt to speed the process up, I ran
>>> your utility to persist relationships
>>> (org.xbrlapi.data.bdbxml.examples.utilities,
>>> PersistAllRelationshipsInStore.java), but this did not help.  I am beginning
>>> to suspect that there is something wrong with my installation, but I am not
>>> sure how to figure out what.
>>>
>>> Do you have any idea why these functions are eating so much memory and
>>> CPU on my system?  I don't think that I will be able to use dimensions and
>>> aspects for my project (which will analyze over 500 annual reports) if I
>>> cannot solve these issues.
>>>
>>>
>>> Regards,
>>> Matt
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> All of the data generated in your IT infrastructure is seriously
>>> valuable.
>>> Why? It contains a definitive record of application performance, security
>>> threats, fraudulent activity, and more. Splunk takes this data and makes
>>> sense of it. IT sense. And common sense.
>>> http://p.sf.net/sfu/splunk-d2d-c2
>>> _______________________________________________
>>> Xbrlapi-developer mailing list
>>> Xbr...@li...
>>> https://lists.sourceforge.net/lists/listinfo/xbrlapi-developer
>>>
>>>
>>
>