ArrayIndexOutOfBoundsException

[Didier]

Hello,

I have found this sofware very interesting, but

Very often, I have this sort of message with "Genetic Programming" :

Exception in thread "Thread-6" java.lang.ArrayIndexOutOfBoundsException: -2147481508
        at java.util.ArrayList.get(ArrayList.java:324)
        at org.mov.quote.EODQuoteCache.getQuotesForDate(EODQuoteCache.java:305)
        at org.mov.quote.EODQuoteCache.getQuoteCacheQuote(EODQuoteCache.java:287)
        at org.mov.quote.EODQuoteCache.getQuote(EODQuoteCache.java:183)
        at org.mov.quote.EODQuoteBundle.getQuote(EODQuoteBundle.java:126)
        at org.mov.analyser.gp.GPQuoteBundle.getQuote(GPQuoteBundle.java:92)
        at org.mov.parser.expression.LagExpression.evaluate(LagExpression.java:57)
        at org.mov.parser.expression.GreaterThanExpression.evaluate(GreaterThanExpression.java:37)
        at org.mov.analyser.PaperTrade.buyTrades(PaperTrade.java:386)
        at org.mov.analyser.PaperTrade.paperTrade(PaperTrade.java:678)
        at org.mov.analyser.gp.Individual.paperTrade(Individual.java:244)
        at org.mov.analyser.gp.GeneticProgramme.nextIndividual(GeneticProgramme.java:192)
        at org.mov.analyser.GPModule.geneticProgramme(GPModule.java:327)
        at org.mov.analyser.GPModule.access$3(GPModule.java:231)
        at org.mov.analyser.GPModule$3.run(GPModule.java:193)
        at java.lang.Thread.run(Thread.java:619)

I haved checked, but I will not find alone!!

I have done lots of tests and the problem seems to be located in the LagExpression :

"int lag = (int)getChild(1).evaluate(variables, quoteBundle, symbol, day);
           throw EvaluationException.LAG_OFFSET_EXCEPTION;
        int quoteKind = ((QuoteExpression)getChild(0)).getQuoteKind();"

The lag must be <0 but sometime the lag is >0 !!

I am not enough good in Java tu understand well "(int)getChild(1).evaluate" and "EvaluationException.LAG_OFFSET_EXCEPTION".

Could you explain me the signification, or give the problem, or better, the anwer ... ?

Best Regards

Didier

[Mark Hummel]

Hi Didier,

Have you been able to replicate the problem at all? If so, can you provide the settings and symbols which trigger the problem?

The index -2147481508 is quite close to -2^31, which could indicate integer overflow. However, without being able to replicate the problem, there's no way to determine what type of expression child(1) is.

Thanks,
Mark.

[Didier]

Hi Mark,

Thank you for your help

I have replicated the problem many times.

It's inside many classes of org.mov.parser.expression

The problem is for exemple  (for LagExpression Class):

     Buy : (percent(dayofweek(), capital))<abs(lag(volume, -27+(percent(-4, min(volume, transactions, -31-order)))))

     When it's going in the Class LagExpression we have a bad calculation of lag :

    LagExpression.evaluate :
        lag= -2147483648 !!!! --> PROBLEM
        day = -2140  --> OK
        child1 =-27+(percent(-4, min(volume, transactions, -31-order))) --> OK
        child0 = volume --> OK

Is it easier for you to understand the problem ?

Thanks

Didier

[Mark Hummel]

Hi Didier,

There may well be a problem with the LagExpression class, or it could be in the inputs to the LagExpression methods.  What I was asking for was the parameters to the Genetic Programming that trigger the problem.
Specifically, the start and end date ranges, which symbols (e.g All-ordinaries, S+P500 etc), the number of stocks and initial capital, and the GP parameters. If you can provide those, I should be able to see the same behaviour and track down the root cause.

Thanks,

Mark.

[Didier]

Hi Mark,

Please find one exemple of parameters.

Sometime it's going to the end with no problem, most of the time it's stopping before the end.

I have imported from yahoo.finance.com :
            RNO.PA
            01/01/1993
            14/03/2008

GP :
In the order :

1)
            3/01/2000
            1/03/2007
            RNO.PA
            Unordored
2)
            50000
            100
            Number of stocks 1
3)
            by Tomorrow open
            by tomorrow open

4)
            120
            300
            60
            3
            180
5)
            -
            100
            0.0
            10
6)
            28.52
            71.42

            19.89
            19.89
            20.08
            20.07
            20.07
            0
            0
            0
            0

            33.34
            33.33
            33.33

            25
            25
            25
            25

            11.12
            11.11
            11.11
            11.11
            11.11
            11.11
            11.11
            11.11
            11.11

            0.01
            5.90
            5.90
            5.90
            0.01
            5.88
            5.88
            5.88
            5.88
            0.01
            5.88
            0
            0
            0
            0
            5.87
            5.87
            5.87
            5.87
            5.87
            5.87
            5.87
            5.87

            0.01
            5.56
            5.56
            5.56
            0.01
            5.56
            5.56
            5.56
            5.56
            5.56
            5.56
            0.02
            5.56
            5.56
            5.55
            5.55
            5.54
            5.54
            5.54
            5.54

            0
            20
            20
            20
            0
            20
            0
            0
            20
            0
            0

            0
            25
            25
            25
            0
            25
            0
            0
            0

To run :

    java -Xss1024k -Xms2024m -Xmx2024m -jar venice.jar

    With Linux Unbuntu 7.10 - Gutsy Gibbon

PS : I make a graphics with symbol RNO.PA to be quicker!!

Thanks

Didier

[Didier]

Hi Mark,

I have done some small mistakes in my last message :

You can simulate the problem with :

I have imported from yahoo.finance.com :
            RNO.PA
            01/01/1993
            14/03/2008

GP :
In the order :

1)
            2/01/2003
            1/03/2007
            RNO.PA
            Unordered
2)
            50000
            100
            Number of stocks 1
3)
            by Tomorrow open
            by tomorrow open

4)
            120
            300
            60
            3
            180
5)
            -
            100
            0,0
            10
6)
            28,58
            71,42

            19,89
            19,89
            20,08
            20,07
            20,07
            0
            0
            0
            0

            33,34
            33,33
            33,33

            25
            25
            25
            25

            11,12
            11,11
            11,11
            11,11
            11,11
            11,11
            11,11
            11,11
            11,11

            0,01
            5,90
            5,90
            5,90
            0,01
            5,99
            0,01
            5,88
            5,88
            5,88
            5,88
            0,01
            5,88
            0
            0
            0
            0
            5,87
            5,87
            5,87
            5,87
            5,87
            5,87
            5,87
            5,87

            0,01
            5,56
            5,56
            5,56
            0,01
            5,56
            5,56
            5,56
            5,56
            5,56
            5,56
            0,02
            5,56
            5,56
            5,55
            5,55
            5,54
            5,54
            5,54
            5,54
            5,54

            0
            20
            20
            20
            0
            20
            0
            0
            20
            0
            0

            0
            25
            25
            25
            0
            25
            0
            0
            0

PS : before run, please,make a graphic with RNO.PA symbol to be quicker to run !!

Could you, please, inform me if you have the same problem as me ?

Thanks

Didier

[Mark Hummel]

Hi Didier,

I have not yet managed to replicate the index out of bounds problem - all I am seeing at the moment are assertions triggered from date methods that aren't implemented yet. My attempts to implement them causes date offset problems - so I'm still working on it.

Are you using version 0.71b? Also, which version Java are you using?

Thanks,
Mark.

[Didier]

Hi Mark,

I'm using version 0.71b where I have done some small modifications to import data, but I have the same problem with original version  0.71b (I have tested this morning, the problem has happened à the second running of GP!).

I'm using :
     (java -fullversion)
     java full version "1.6.0_03-b05"

I think you are on the good way because

the problem appear in the calculation of : offset, lag, period and days.

in avgExpression, BBUExpression , ema..., GreaterThanEqual..., GreaterThan..., lagExpression, LessThan..., MomentumExpression, NotEqualExpression, OBVExpression, StandardDeviation, SumExpression

This is the message of the original version (in the console) with the example before (RNO.PA ...):

Le Marchand de Venise, 0.71 beta / 25/Fev/2007
----------------------------------------------
Copyright (C) 2003-7, Andrew Leppard (aleppard@picknowl.com.au)
Référez vous au fichier COPYING.txt en ce qui concerne la licence.

(<unknown>:23014): Gtk-WARNING **: Attempting to add a widget with type GtkButton to a GtkComboBoxEntry (need an instance of GtkEntry or of a subclass)

(<unknown>:23014): Gtk-CRITICAL **: gtk_widget_realize: assertion `GTK_WIDGET_ANCHORED (widget) || GTK_IS_INVISIBLE (widget)' failed

(<unknown>:23014): Gtk-CRITICAL **: gtk_paint_box: assertion `style->depth == gdk_drawable_get_depth (window)' failed

(<unknown>:23014): Gtk-CRITICAL **: gtk_paint_box: assertion `style->depth == gdk_drawable_get_depth (window)' failed

(<unknown>:23014): Gtk-CRITICAL **: gtk_paint_box: assertion `style->depth == gdk_drawable_get_depth (window)' failed

(<unknown>:23014): Gtk-CRITICAL **: gtk_paint_box: assertion `style->depth == gdk_drawable_get_depth (window)' failed

Exception in thread "Thread-11" java.lang.ArrayIndexOutOfBoundsException: -2147481489
        at java.util.ArrayList.get(ArrayList.java:324)
        at org.mov.quote.EODQuoteCache.getQuotesForDate(EODQuoteCache.java:303)
        at org.mov.quote.EODQuoteCache.getQuoteCacheQuote(EODQuoteCache.java:287)
        at org.mov.quote.EODQuoteCache.getQuote(EODQuoteCache.java:183)
        at org.mov.quote.EODQuoteBundle.getQuote(EODQuoteBundle.java:126)
        at org.mov.analyser.gp.GPQuoteBundle.getQuote(GPQuoteBundle.java:90)
        at org.mov.parser.expression.LagExpression.evaluate(LagExpression.java:56)
        at org.mov.parser.expression.LessThanEqualExpression.evaluate(LessThanEqualExpression.java:37)
        at org.mov.parser.expression.IfExpression.evaluate(IfExpression.java:51)
        at org.mov.parser.expression.GreaterThanExpression.evaluate(GreaterThanExpression.java:37)
        at org.mov.analyser.PaperTrade.buyTrades(PaperTrade.java:386)
        at org.mov.analyser.PaperTrade.paperTrade(PaperTrade.java:678)
        at org.mov.analyser.gp.Individual.paperTrade(Individual.java:232)
        at org.mov.analyser.gp.GeneticProgramme.nextIndividual(GeneticProgramme.java:191)
        at org.mov.analyser.GPModule.geneticProgramme(GPModule.java:327)
        at org.mov.analyser.GPModule.access$300(GPModule.java:43)
        at org.mov.analyser.GPModule$3.run(GPModule.java:193)
        at java.lang.Thread.run(Thread.java:619)

Thanks

Didier

PS: May I do something to help you ? (I have started learning Java to do some small mofications in MOV !!)

[Mark Hummel]

Hi Didier,

Progress report on the problem - I've discovered a cause, but I am still considered a solution. I was right, it is a case of integer overflow/underflow. What's happening is that you have an expression, say minExpression which when it evaluates it doesn't find any quotes for the particular symbol, and returns the initial value - Double.MAX_VALUE. If minExpression is a child of LagExpression, when lagExpression evaluates you get a lag of
-INTEGER.MAX_VALUE. Now the problem arises when you take the lag and add it an offset (e.g -1000). When you take Integer.MAX_VALUE - 1000, the result underflows and becomes a positive integer, and subsequently triggers the assertions.  This situation probably occurs in reverse with other expression trees yielding a negative index given two positive operands.

Ok, so that's the cause, but what is the solution? Possibilities:
1) Halve the default values so instead of returning INTEGER.MAX_VALUE, we return INTEGER.MAX_VALUE / 2 - reducing the possibility of overflow (unless both operands are within epislon of the maximum)
2) Detect when overflow has occurred and correct for it.
3) Throw an exception in the evaluation method instead of returning the default value.

I'm not sure I like either of the first two options - seem a bit kludgy. Especially 2) - you'd have this sort of code which would look a bit funny on casual inspection.
if (op1 < 0 && op2 < 0)
   if (op1 + op2 > 0) {
     //detected overflow, do something about it
  }
}

The problem with 3), is I don't know what the appropriate response is for all the situations and what to do about it in the long term.

Hopefully I'll be able to post a fix, but if you want to get past this bug in the short term edit the getQuote method in GPQuoteBundle.java. Here is the code you want to change:

//Date is within range
else
return getQuote(symbol, quoteType, tpday+offset)

and replace with:

//Date is within range
else {
  int dateOffset = today + offset;
  if (today < 0 && offset < 0)
     if (dateOffset > 0) {
        dateOffset *= -1;
     ]
  }
  return getQuote(symbol, quoteType, dateOffset);
}

Thanks for your help,
Regards,
Mark.

PS You are welcome to post any patches, but I don't know if anyone else will look at them. Alternatively, you can send them directly to me. In other words, "Patches Welcome" ;)

[Didier]

Hi Mark,

Thank you very much for the answer.

I have read with attention your answer, but I have one question :

You said "What's happening is that you have an expression, say minExpression which when it evaluates it doesn't find any quotes for the particular symbol,.."

Why it doesn't find any quote ? Is-it because there is no quote in the database (Week-end ..)? or because it try to find quote out of the bounds of the database ( try to find quote in 1/03/1932 for example ?) ?

In the last case, maybe we can find a solution to generate sell an buy rules in the bounds of the database ? (It will not be easy!!)

Thanks
Didier

[Mark Hummel]

Hi Didier,

That's a good question.

After a fair bit of digging, I can confirm that it's the latter case - it's accessing quotes out of the database range.

In the case of the MinExpression, you have the loop:

(for int i = offset - days + 1; i <= offset; i++) {
....
}

In the cases where no value is being set, offset is set to the greatest lower bound of the database. ie the offset of the earliest date. So the loop never reaches a date where there is data. Offset is a parameter of a child of the
MinExpression tree. At this point, I don't know if that value is generated as part of the GP process, or if it's derived from the given date range. In the former case, we'll have to deal with expressions evaluating with default values; if it's the latter it's possibly a bug and we may have to do as you suggest. I'll need to spend a little more time tracking the source of the offset.

Thanks,
Mark.

[Mark Hummel]

Hi Didier,

A followup to my previous post - I was able to determine that the offset was a result of the GP process. To solve the problem of the default values causiing overflow, I modified the Min/Max expressions to throw an EvaluationException instead of returning the default value. If the GP process encounters this exception at the top level, it prunes the result for that individual and tries another. This fixes the problem raised by the assertions, allows the GP to run to completion, and doesn't seem to change the end result of the GP process significantly.

I've checked in the fix to CVS; please try the latest version and let me know if you can run the GP to completion and what other bugs you encounter.

Thanks,

Mark.

[Didier]

Hi Mark,

Thank you very much for what you have done.

I will be able to test your last version, Monday.
I will inform you, if I found other bugs.

An other question about the speed of GP program :
When I take only one quote (RNO.PA for example), and when I run GP program, it's slow.
If I make a graphics with RNO.PA before running GP program, the running is quick !!

If I use 2 quotes in GP program (exemple RNO.PA and BNP.PA), even if I make a graphic of them before running GP program, the running is still slow !!

So I imagine that there is a solution to increase the speed.
Do you have a idea ?

Thanks,

Didier

[Didier]

Hi Mark,

I have tested the current version, the running is quicker (good news for me --> so my last demand about speed is out of date!!) :

1) In readme.txt, I think :

line 84 : cvs -z3 -d:pserver:anonymous@mov.cvs.sourceforge.net:/cvsroot/mov co -P venice

has to be changed by :

cvs -z3 -d:pserver:anonymous@mov.cvs.sourceforge.net:/cvsroot/mov co -P mov

2) I have a problem to make venice.jar :

with the command "ant doc" (and "ant jar") I have this result :

Buildfile: build.xml

doc:
     [xslt] Transforming into /home/didier/venice071c/mov/src/org/mov/help/doc
     [xslt] Processing /home/didier/venice071c/mov/doc/manual.xml to /home/didier/venice071c/mov/src/org/mov/help/doc/manual.html
     [xslt] Loading stylesheet /home/didier/venice071c/mov/doc/manual-help.xsl
     [xslt] : Error! Unrecognized XSLTC extension 'org.apache.xalan.xslt.extensions.Redirect:open'
     [xslt] Failed to process null

I don't know why ?

So, I use "ant run" to test venice

3) The example with RNO.PA is running very well now, but I have discovered a other bug ( I think it's the same type of problem)

You can simulate this problem with :

import from yahoo.finance.com :
RNO.PA, BNP.PA and LG.PA
01/01/1993
14/03/2008

GP :
In the order :

1)
2/01/2003
1/03/2007
RNO.PA,BNP.PA,LG.PA
Unordered
2)
50000
100
Number of stocks 1
3)
by Tomorrow open
by tomorrow open

4)
120
300
60
3
180
5)
-
100
0,0
10
6)
All "set as default"
-------------------------------
And I have this type of message in console :

(<unknown>:8753): Gtk-CRITICAL **: gtk_paint_box: assertion `style->depth == gdk_drawable_get_depth (window)' failed
     [java] Exception in thread "Thread-29" java.lang.AssertionError
     [java]     at org.mov.quote.EODQuoteCache.getQuotesForDate(EODQuoteCache.java:298)
     [java]     at org.mov.quote.EODQuoteCache.getQuoteCacheQuote(EODQuoteCache.java:287)
     [java]     at org.mov.quote.EODQuoteCache.getQuote(EODQuoteCache.java:183)
     [java]     at org.mov.quote.EODQuoteBundle.getQuote(EODQuoteBundle.java:126)
     [java]     at org.mov.analyser.gp.GPQuoteBundle.getQuote(GPQuoteBundle.java:90)
     [java]     at org.mov.quote.QuoteBundleFunctionSource.getValue(QuoteBundleFunctionSource.java:90)
     [java]     at org.mov.quote.QuoteFunctions.obv(QuoteFunctions.java:436)
     [java]     at org.mov.parser.expression.OBVExpression.evaluate(OBVExpression.java:70)
     [java]     at org.mov.parser.expression.LessThanEqualExpression.evaluate(LessThanEqualExpression.java:37)
     [java]     at org.mov.analyser.PaperTrade.sellTrades(PaperTrade.java:313)
     [java]     at org.mov.analyser.PaperTrade.paperTrade(PaperTrade.java:671)
     [java]     at org.mov.analyser.gp.Individual.paperTrade(Individual.java:232)
     [java]     at org.mov.analyser.gp.GeneticProgramme.nextIndividual(GeneticProgramme.java:191)
     [java]     at org.mov.analyser.GPModule.geneticProgramme(GPModule.java:330)
     [java]     at org.mov.analyser.GPModule.access$300(GPModule.java:44)
     [java]     at org.mov.analyser.GPModule$3.run(GPModule.java:196)
     [java]     at java.lang.Thread.run(Thread.java:619)

I will inform you, if I have some other informations about this bug.

Thanks,

Didier

[Mark Hummel]

Hi Didier,

1) Yes, you're correct. Good catch.
2) I suspect you don't have Xalan-J installed or at least not where ant can find it. You need Xalan for the extensions. The base distribution of ant doesn't include it, it seems.
3) There's still something going here. I'm still looking into this, but I'm sure I'll find more bugs.

Thanks for all your feedback,
Mark.

[Mark Hummel]

Ok,

There's another fix in CVS for this problem.

If you;re interested, the bug was in the check: (Math.abs(offset) > Math.abs(window), which is supposed to prevent dates too far into the past from being accessed. Unfortunately, Math.Abs(Integer.MIN_VALUE) returns Integer.MIN_VALUE, which is a negative number, so the check fails. I wish that it had occurred to me to check this line earlier, I could have saved a lot of time. Oh well.

The reason why offset gets so large is because of expressions like these:  (avg(volume, 36, -37))*(obv(36, 0, macd(volume, -20))). For many stocks, volume is a number in the millions, and when nested expressions which are essentially chains of multiplications are generated, numbers of these sizes are inevitable.

Thanks again,

Mark.

[Didier]

Hi Mark,

I'm testing the last version, and it's still running : NO bug.

Thank you very much for what you have done.

I will inform you if I found others running bugs.

I think, there is some graphics bug (when we ask some graph). Next week, I will take the time send a new item in the forum about this point.

Thank you very much again, now I will start to use venice for what it has been designed.

Didier.

[Mark Hummel]

Hi Didier,

While you are most welcome, I couldn't have managed it had you not replicated the bug and persevered with the problem.  So thank you for sticking with it and asking the right questions.

Regards,

Mark.

[Marc]

Hi Didier and Mark,

I have the same xalan-j issue. Could you walk me through how to fix it? I am trying to use the ant script w/in NetBeans on the target "doc", hoping that I'll have locally the same thing as the javadoc online here http://mov.sourceforge.net/api/index.html

I will download the latest release from http://xml.apache.org/xalan-j/downloads.html#latest-release

But then what do I do with it?

Thanks,
Marc

Transforming into C:\JAVAFILES\mov\src\org\mov\help\doc
Processing C:\JAVAFILES\mov\doc\manual.xml to C:\JAVAFILES\mov\src\org\mov\help\doc\manual.html
Loading stylesheet C:\JAVAFILES\mov\doc\manual-help.xsl
: Error! Unrecognized XSLTC extension 'org.apache.xalan.xslt.extensions.Redirect:open'
Failed to process null

[Marc]

Cool! works. thanks a lot.

some little comments:

1) the doc target in the provided build.xml actually creates the help file for the application (and does not create the javadoc that I was looking for). It'll be accessible via Help -> Contents. If you don't run this target, it'll just show "???" and "no help file"

2) in order to create a javadoc on my hard drive (and have it "on the road" w/o internet connection), I used the javadoc target that NetBeans creates for you in nbbuild.xml as you create a project with existing java files

[Mark Hummel]

'api' is the target that builds the javadoc.  (ie "ant api")

[Mark Hummel]

Hi Marc,

You need to place the jar files where ant can find them - I placed them in the same directory as ant.jar. On my system that's /usr/share/ant/lib.

Regards,

Mark.