geniustrader-devel — participate in geniustrader development, ask for help, suggest improvements, etc

Re: [GT] inaccurate RSI data

[Robert A. S. <ra...@ac...>]

Thomas Graham wrote:
> Hi all,
> 
> Please check attached chart, left hand side is generated by Yahoo data;
> Right hand side is so call "commercial site", all indicator lines seems
> very close except RSI 14 which have big offset. Is there have a problem
> with RSI calculation?

most instances of comparisons like these are due to differences in the
initial conditions used to 'prime' the indicator. as you see gt has no
values for rsi until late july whereas the "commercial site" has data.

the other aspect, especially for the more complex indicators, is overall
input data equivalence -- with gt you know you are using data from whatever
source you are feeding it from -- free yahoo data or some other provider.
with a "commercial site" you usually have no info regarding their data
source(s). while it's relatively straight forward to compare price data,
volume data is other challenge. i don't think i've ever seen two different
sites (take yahoo and google) agree on volume.

on the other hand the attached I:RSI module might be an improvement (or not).

ras

> 
> here is my graph config:
> --title=Stock of %c
> --add=Curve(Indicators::SMA 10, [255,0,0])
> --add=Curve(Indicators::SMA 20, [255,240,0])
> --add=Curve(Indicators::SMA 50, [194,198,236])
> --add=Curve(Indicators::SMA 100, [255,200,0])
> --add=Curve(Indicators::SMA 150, [255,175,175])
> --add=New-Zone(110)
> --add=Text("RSI (14)", 0, 100)
> --add=Curve(Indicators::RSI 14, [255,0,0])
> --add=New-Zone(110)
> --add=Text("MACD", 0, 100)
> --add=Curve(Indicators::MACD/1, [255,0,0])
> --add=Curve(Indicators::MACD/2, [0,255,0])
> 
> [image: Inline image 1]
> 
> 
> ------------------------------------------------------------------------
> 
> ------------------------------------------------------------------------------
> Keep yourself connected to Go Parallel: 
> TUNE You got it built. Now make it sing. Tune shows you how.
> http://goparallel.sourceforge.net

Re: [GT] [system-traders] Problem to use scan.pl

[Thomas G. <lkt...@gm...>]

one step forward, I found data in DB is the cause of problem, check the
compare between yesterday and today data in the current DB:

sqlite> select * from stockprices where symbol = '0590.HK' AND date =
'2012-11-30';
0590.HK|2012-11-30|23.15|23.0|22.2|23.45|23.25|0.1|23.2|23.25|5979040
sqlite> select * from stockprices where symbol = '0590.HK' AND date =
'2012-11-29';
0590.HK|2012-11-29||21.5|21.4|23.45|23.15||||18117100



On Fri, Nov 30, 2012 at 9:47 PM, Thomas Graham <lkt...@gm...> wrote:

> Robert, I found out this is a bug.
>
> Try this step to reproduce it:
> 1. beancounter backpopulate T --prevdate (Any date) --date 'today'
> 2. run scan.pl. works fine
> 3. beancounter update --forceupdate today
>
>
> tons of error:
> Use of uninitialized value $age in numeric gt (>) at
> /usr/share/perl5/Finance/BeanCounter.pm line 1743.
> Overriding date for 0175.HK (GEELY AUTO) to 20121130
>
>
> 4.  ./scan.pl --timeframe=day --start=2012-07-26 --end=2012-11-26
> /mnt/stocks/newstockscode.txt 2012-11-26 /mnt/trading/system.gt
>  Use of uninitialized value $d in pattern match (m//) at
> ../GT/DateTime.pm line 284, <> line 4.
> Use of uninitialized value $m in subtraction (-) at ../GT/DateTime.pm line
> 285, <> line 4.
>
>
>
> On Fri, Nov 30, 2012 at 9:24 PM, Robert A. Schmied <ra...@ac...> wrote:
>
>> aloha thomas
>>
>> thanks for the cc -- the list is not forwarding things correctly
>>
>>
>> Thomas Graham wrote:
>>
>>> Another problem after this:
>>> $ ./scan.pl --timeframe=day --start=2012-01-26 --end=2012-11-26
>>> /mnt/stocks/newstockscode.txt 2012-11-26 /mnt/trading/system.gt
>>> Use of uninitialized value $d in pattern match (m//) at ../GT/DateTime.pm
>>> line 284, <> line 5.
>>> Use of uninitialized value $m in subtraction (-) at ../GT/DateTime.pm
>>> line
>>> 285, <> line 5.
>>> Month '-1' out of range 0..11 at ../GT/DateTime.pm line 285
>>> Use of uninitialized value $d in pattern match (m//) at ../GT/DateTime.pm
>>> line 284, <> line 5.
>>> Use of uninitialized value $m in subtraction (-) at ../GT/DateTime.pm
>>> line
>>> 285, <> line 5.
>>>
>>>
>>>
>> this sort of warnings indicate a problem with the values being supplied
>> from
>> the prices database source.
>>
>> <snip>
>>
>>
>>>>  Thomas Graham wrote:
>>>>>
>>>>>
>>>>>  sqlite> .tables
>>>>>> beancounter  fxprices     portfolio    stockprices
>>>>>> cash         indices      stockinfo
>>>>>>
>>>>>>
>>>>>> sqlite> select * from stockprices where symbol='0001.HK';
>>>>>> 0001.HK
>>>>>> |2012-08-08||109.168546617916|****106.979204753199|109.**
>>>>>> 268062157221|108.87||||4971300
>>>>>> 0001.HK
>>>>>> |2012-08-09||109.266318631863|****108.967776777678|110.**
>>>>>> 858541854185|110.56||||7319600
>>>>>> 0001.HK
>>>>>> |2012-08-10||109.763678899083|****107.474862385321|110.**
>>>>>> 36076146789|108.47||||5611200
>>>>>> 0001.HK
>>>>>> |2012-08-13||109.967106460419|****109.071446769791|111.**
>>>>>> 161319381256|109.37||||6163000
>>>>>>
>>>>>>
>> humm -- the fields with the asterisks look strange or are the asterisk
>> pairs
>> some sort of email/gmail escaping -- i see them in the font paths below
>> too
>>
>> my postgresql beancounter tuple fields are:
>>
>> symbol |    date    | previous_close | day_open | day_low | day_high |
>> day_close | day_change | bid | ask |  volume
>> gt only uses 'symbol | date | day_open | day_low | day_high | day_close |
>> volume'
>> if i remember correctly.
>> if the asterisks are really entered in the db it looks like your prices
>> source for 0001.HK is feeding things beancounter and/or yahooquote cannot
>> correctly parse and archive.
>>
>> interesting. 0001.HK here in states returns what looks to be reasonable
>> data
>>
>>
>> does your prices database include data for all dates in your date range?
>> --start=2012-01-26 --end=2012-11-26
>> many times missing price data can manifest itself in very strange behavior
>> and or messages.
>>
>> what happens if you try a very simple graphic.pl chart for 0001.HK with
>> this same date range?
>>    %     graphic.pl --start=2012-01-26 --end=2012-11-26 0001.HK>/tmp/tg_0001.hk.png
>>
>> ras
>>
>>
>>>>>> ~/.gt/options
>>>>>> DB::module bean
>>>>>> DB::bean::dbname /mnt/trading/stocks.db
>>>>>> DB::bean::db SQLite
>>>>>> Path::Font::Arial    /usr/share/fonts/truetype/****
>>>>>> msttcorefonts/arial.ttf
>>>>>> Path::Font::Courier  /usr/share/fonts/truetype/****
>>>>>> msttcorefonts/couri.ttf
>>>>>> Path::Font::Times    /usr/share/fonts/truetype/****
>>>>>> msttcorefonts/times.ttf
>>>>>>
>>>>>>
>>>>>>
>>>>>
>> << big snip >>
>>
>>
>>
>
>
> --
> Thomas G Lau
> Tel: 93239670
>



-- 
Thomas G Lau
Tel: 93239670

Re: [GT] [system-traders] Problem to use scan.pl

[Thomas G. <lkt...@gm...>]

Robert, I found out this is a bug.

Try this step to reproduce it:
1. beancounter backpopulate T --prevdate (Any date) --date 'today'
2. run scan.pl. works fine
3. beancounter update --forceupdate today


tons of error:
Use of uninitialized value $age in numeric gt (>) at
/usr/share/perl5/Finance/BeanCounter.pm line 1743.
Overriding date for 0175.HK (GEELY AUTO) to 20121130


4.  ./scan.pl --timeframe=day --start=2012-07-26 --end=2012-11-26
/mnt/stocks/newstockscode.txt 2012-11-26 /mnt/trading/system.gt
Use of uninitialized value $d in pattern match (m//) at ../GT/DateTime.pm
line 284, <> line 4.
Use of uninitialized value $m in subtraction (-) at ../GT/DateTime.pm line
285, <> line 4.



On Fri, Nov 30, 2012 at 9:24 PM, Robert A. Schmied <ra...@ac...> wrote:

> aloha thomas
>
> thanks for the cc -- the list is not forwarding things correctly
>
>
> Thomas Graham wrote:
>
>> Another problem after this:
>> $ ./scan.pl --timeframe=day --start=2012-01-26 --end=2012-11-26
>> /mnt/stocks/newstockscode.txt 2012-11-26 /mnt/trading/system.gt
>> Use of uninitialized value $d in pattern match (m//) at ../GT/DateTime.pm
>> line 284, <> line 5.
>> Use of uninitialized value $m in subtraction (-) at ../GT/DateTime.pm line
>> 285, <> line 5.
>> Month '-1' out of range 0..11 at ../GT/DateTime.pm line 285
>> Use of uninitialized value $d in pattern match (m//) at ../GT/DateTime.pm
>> line 284, <> line 5.
>> Use of uninitialized value $m in subtraction (-) at ../GT/DateTime.pm line
>> 285, <> line 5.
>>
>>
>>
> this sort of warnings indicate a problem with the values being supplied
> from
> the prices database source.
>
> <snip>
>
>
>>>  Thomas Graham wrote:
>>>>
>>>>
>>>>  sqlite> .tables
>>>>> beancounter  fxprices     portfolio    stockprices
>>>>> cash         indices      stockinfo
>>>>>
>>>>>
>>>>> sqlite> select * from stockprices where symbol='0001.HK';
>>>>> 0001.HK
>>>>> |2012-08-08||109.168546617916|****106.979204753199|109.**
>>>>> 268062157221|108.87||||4971300
>>>>> 0001.HK
>>>>> |2012-08-09||109.266318631863|****108.967776777678|110.**
>>>>> 858541854185|110.56||||7319600
>>>>> 0001.HK
>>>>> |2012-08-10||109.763678899083|****107.474862385321|110.**
>>>>> 36076146789|108.47||||5611200
>>>>> 0001.HK
>>>>> |2012-08-13||109.967106460419|****109.071446769791|111.**
>>>>> 161319381256|109.37||||6163000
>>>>>
>>>>>
> humm -- the fields with the asterisks look strange or are the asterisk
> pairs
> some sort of email/gmail escaping -- i see them in the font paths below too
>
> my postgresql beancounter tuple fields are:
>
> symbol |    date    | previous_close | day_open | day_low | day_high |
> day_close | day_change | bid | ask |  volume
> gt only uses 'symbol | date | day_open | day_low | day_high | day_close |
> volume'
> if i remember correctly.
> if the asterisks are really entered in the db it looks like your prices
> source for 0001.HK is feeding things beancounter and/or yahooquote cannot
> correctly parse and archive.
>
> interesting. 0001.HK here in states returns what looks to be reasonable
> data
>
>
> does your prices database include data for all dates in your date range?
> --start=2012-01-26 --end=2012-11-26
> many times missing price data can manifest itself in very strange behavior
> and or messages.
>
> what happens if you try a very simple graphic.pl chart for 0001.HK with
> this same date range?
>    %     graphic.pl --start=2012-01-26 --end=2012-11-26 0001.HK>/tmp/tg_0001.hk.png
>
> ras
>
>
>>>>> ~/.gt/options
>>>>> DB::module bean
>>>>> DB::bean::dbname /mnt/trading/stocks.db
>>>>> DB::bean::db SQLite
>>>>> Path::Font::Arial    /usr/share/fonts/truetype/****
>>>>> msttcorefonts/arial.ttf
>>>>> Path::Font::Courier  /usr/share/fonts/truetype/****
>>>>> msttcorefonts/couri.ttf
>>>>> Path::Font::Times    /usr/share/fonts/truetype/****
>>>>> msttcorefonts/times.ttf
>>>>>
>>>>>
>>>>>
>>>>
> << big snip >>
>
>
>


-- 
Thomas G Lau
Tel: 93239670

Re: [GT] Kirshenbaum Bands

[Thomas W. <tho...@un...>]

Delix,

the POD says:

"Kirshenbaum Bands are similar to Bollinger Bands, in that they measure
market volatility. However, rather than use Standard Deviation of
a moving average for band with, they use Standard Error of linear
regression lines of the Close. This has the effect of measuring
volatility around the current trend, instead of measuring volatility for
changes in trend."

What this refers to is that the signal lines are not N standard
deviations below or above the MA, but instead they use a different
distance based on the SE as described in the POD.

The key lines in the code are:

    # Kirshenbaum Band Sup is equal to the value of the exponential moving
    # average + standard error
    my $kbsup_value = $ema_value + ($n * $se_value);
   
    # Kirshenbaum Band Inf is equal to the value of the exponential moving
    # average - standard error
    my $kbinf_value = $ema_value - ($n * $se_value);
 
Th.

On 11/29/2012 10:44 AM, Delix wrote:
> Sorry guys, it's me again....
> My next question :
> the pod of the KirshenbaumBands modul says it's using a linear
> regression line instead of a MA, but then the EMA is used in the
> calculation (?)
> Once again : is this a misunderstanding ?

Re: [GT] RSquare indicator

[Robert A. S. <ra...@ac...>]

Shinaberry Derek wrote:
> Some commentary interspersed below.
> 
> Derek
> 
> On Nov 25, 2012, at 18:50 , Robert A. Schmied wrote:
> 
> <snip>
> 
>> so now all that remains is to see if there is a way to resolve the
>> initial confusion issue that started this in the first place:
>> 
>> i've hacked the following extensively revised pod into package
>> GT::Indicators::RSquare -- doubtless i've mis-worded things,
>> possibly made errors and generally made a mess of it (the pod) --
>> anyone one want to rework, correct, clarify or ???.
>> 
>> 
>> =pod
>> 
>> =head1 GT::Indicators::RSquare
>> 
>> =head2 Synopsis
>> 
>> I:RSquare(14, "{I:Prices CLOSE}")
>> 
>> =head2 Overview
>> 
>> This function calculates the R-Squared coefficient. Originally
>> defined as a technical indicator in "The New Technical Trader" by
>> Tushar S. Chande and Stanley Kroll. The intent is to use
>> correlation against a monotonically increasing line as a means of
>> quantifying the strength of a trend.
>> 
>> =head2 Calculation
>> 
>> For GT I:RSquare is computed using
>> 
>> {I:BPCorrelation( 14, {I:Generic:Cum 1}, {I:Prices CLOSE} ) where -
>> 14 corresponds to RSquare arg #1 - {I:Generic:Cum 1} [and is fixed]
>>  - {I:Prices CLOSE} is RSquare argument #2
>> 
>> as a reference metastock computes RSquare of an Indicator as
>> follows:
>> 
>> Pwr(Corr(Cum(1),C,14,0),2) where Corr() (correlation) is computed: 
>> - Cum(1) is count of periods so far and is fixed. - C is current
>> time period close price data series and can be altered, 
>> corresponding to the 2nd RSquare argument. - 14 is (default) time
>> periods of interest and can be altered, corresponding to the 1st
>> RSquare argument. - 0 is unknown as of this writing and is fixed.
> 
> 
> A little searching shows this Metastock parameter to be a value,
> which is the number of periods that the data series in the second
> argument, i.e. C, is shifted to the right before calculating the
> correlation coefficients.

interesting. does it's absence from the gt implementation impose
serious limits on the utility of the gt implementation?

if so could it easily be added or worked around in some other manner?

> 
> 
>> - Pwr() simply squares the computed Corr() value.
>> 
>> =head2 Discussion
>> 
>> The RSquare of an Indicator shows how well the price approximates a
>> linear regression line. It uses BPCorrelation to compute this.
>> 
>> To use RSquare you must choose a basis on which you want to compare
>> the indicator to.  By default the basis is time (i.e., the numbers
>> of periods so far). In other words, for your correlation you have
>> the indicator on the Y axis and the periods on the X axis.
> 
> 
> The sentence "In other words, ..." muddies the picture of this for
> me.  It almost makes it sound like we're figuring the correlation
> between the two axes or something.  What we're figuring is the
> stepwise correlation between two time series.  Each of these time
> series can be plotted in an X-Y plane, where the X axis is time.  The
> Y value for the indicator time series is whatever the value of that
> indicator is at any given point in time, X.  The Y value of the other
> time series, which is the time basis that we're figuring the
> correlation of the indicator to, is equal to the value of time at any
> given point in time, X.  Or in other words, for the time basis time
> series, Y = X.
> 
> I'll grant you that this is a bit long winded and maybe it is only
> clearer to me because this happens to be how I make sense of it.  To
> my thinking, we could just strike the sentence that bothers me and
> leave it at that.  If you think more description is better than less,
> then I offer up some version of my explanation.  If you strongly
> think it is fine as is, I wouldn't fight you to the death over it.
> 

as thomas weigert wrote both the indic and that explanation i figured it
would clarify, but frankly it shed little additional light for me as well.

i like to see your (any ones) description of what the values of
   %      display_indicator.pl  --timeframe=day \
   --start=2012-08-29 --end=2012-11-26 I:RSquare AAPL
really imply.

the range appears to be 0..1; is less better or is more better?

maybe a set of graphic sys-sig-indics designed to plot the results?

how it might by used in a trading scheme, one could also refer to
Tushar S. Chande and Stanley Kroll.


ras

> 
>> =cut
>> 
>> 
>> 
>> [it might be a good idea to cc me explicitly because the list
>> forwarding to me appears to be rather sporadic of late.]

yea! the list is really funky -- i've not been getting any of the
messages started by delix, but that might not be the root cause,
just a coincidence.

>> 
>> 
>> aloha
>> 
>> ras
> 
> 
> 

Re: [GT] RSquare indicator

[Shinaberry D. <gen...@sh...>]

Some commentary interspersed below.

Derek

On Nov 25, 2012, at 18:50 , Robert A. Schmied wrote:

<snip>
> 
> so now all that remains is to see if there is a way to resolve the initial
> confusion issue that started this in the first place:
> 
> i've hacked the following extensively revised pod into package GT::Indicators::RSquare
> -- doubtless i've mis-worded things, possibly made errors and generally made a mess
> of it (the pod) -- anyone one want to rework, correct, clarify or ???.
> 
> 
> =pod
> 
> =head1 GT::Indicators::RSquare
> 
> =head2 Synopsis
> 
> I:RSquare(14, "{I:Prices CLOSE}")
> 
> =head2 Overview
> 
> This function calculates the R-Squared coefficient.
> Originally defined as a technical indicator in 
> "The New Technical Trader" by Tushar S. Chande and Stanley Kroll.
> The intent is to use correlation against a monotonically increasing
> line as a means of quantifying the strength of a trend.
> 
> =head2 Calculation
> 
> For GT I:RSquare is computed using
> 
> {I:BPCorrelation( 14, {I:Generic:Cum 1}, {I:Prices CLOSE} )
> where
>  - 14 corresponds to RSquare arg #1
>  - {I:Generic:Cum 1} [and is fixed]
>  - {I:Prices CLOSE} is RSquare argument #2
> 
> as a reference metastock computes RSquare of an Indicator as follows:
> 
>  Pwr(Corr(Cum(1),C,14,0),2)
>  where Corr() (correlation) is computed:
>   - Cum(1) is count of periods so far and is fixed.
>   - C is current time period close price data series and can be altered,
>     corresponding to the 2nd RSquare argument.
>   - 14 is (default) time periods of interest and can be altered,
>     corresponding to the 1st RSquare argument.
>   - 0 is unknown as of this writing and is fixed.

A little searching shows this Metastock parameter to be a value, which is the number of periods that the data series in the second argument, i.e. C, is shifted to the right before calculating the correlation coefficients.

>   - Pwr() simply squares the computed Corr() value.
> 
> =head2 Discussion
> 
> The RSquare of an Indicator shows how well the price
> approximates a linear regression line. It uses BPCorrelation
> to compute this.
> 
> To use RSquare you must choose a basis on which you want to
> compare the indicator to.  By default the basis is time
> (i.e., the numbers of periods so far). In other words,
> for your correlation you have the indicator on the Y axis
> and the periods on the X axis.

The sentence "In other words, ..." muddies the picture of this for me.  It almost makes it sound like we're figuring the correlation between the two axes or something.  What we're figuring is the stepwise correlation between two time series.  Each of these time series can be plotted in an X-Y plane, where the X axis is time.  The Y value for the indicator time series is whatever the value of that indicator is at any given point in time, X.  The Y value of the other time series, which is the time basis that we're figuring the correlation of the indicator to, is equal to the value of time at any given point in time, X.  Or in other words, for the time basis time series, Y = X.

I'll grant you that this is a bit long winded and maybe it is only clearer to me because this happens to be how I make sense of it.  To my thinking, we could just strike the sentence that bothers me and leave it at that.  If you think more description is better than less, then I offer up some version of my explanation.  If you strongly think it is fine as is, I wouldn't fight you to the death over it.

> 
> =cut
> 
> 
> 
> [it might be a good idea to cc me explicitly because the list forwarding
> to me appears to be rather sporadic of late.]
> 
> 
> aloha
> 
> ras

Re: [GT] RSquare indicator

[Shinaberry D. <gen...@sh...>]

For what it is worth, I think these are reasonable descriptions given my limited working knowledge of German.  Unfortunately, my knowledge of German is far too limited to be able to catch anything but glaring errors in your text.  I would never catch any subtle problems with it.

Derek

On Sun, 25 Nov 2012 19:48:14 +0100
Delix <del...@t-...> hatte geschrieben :

This is the German version of my manual entries for the BPCorrelation
and RSquare moduls.
Hope somebody is able to check it -- any comment is welcome.

==========================================
\section{BPCORRELATION (Bravais-Pearson Correlation Coefficient)}

\paragraph{\textit{Typ}\quad Basis}

\paragraph{\textit{Beschreibung}}
\newline
Dieser Koeffizient aus der statistischen Analyse berechnet die
Korrelation zweier Datenreihen, die hier bei GT beide jeweils die Zeit
-also in der Regel die Börsenhandelstage - als Bezugsachse haben. Es
wird also z.B. für das Datum X die Korrelation von Höchst- und
Tiefstkursen der vorangegangenen 14 Tage errechnet. Ein Wert von 1
ergibt sich dabei, falls beide Kurse völlig parallel verlaufen sind,
ein Wert von -1 bei genau entgegengesetztem Verlauf (d.h. die zweite
Datenreihe fällt im Vergleich zum Vortag um denselben Betrag, um den
die erste steigt) und ein Wert von Null zeigt an, daß es keinen
linearen Zusammenhang zwischen den beiden Kursverläufen gibt.

\paragraph{\textit{Parameter}}
\newline
Der erste Parameter legt die Länge der Periode an, für die der Verlauf
der beiden Datenreihen für das jeweilige Datum verglichen werden soll.
Mögliche Werte können sowohl eine Konstante als auch die Werte einer
Zeitreihe sein. Die beiden anderen Parameter geben die beiden zu
vergleichenden Datenreihen an.

\paragraph{\textit{Aufruf}}
\newline
I:BPCorrelation IBM ´20 {I:Prices HIGH} {I:Prices CLOSE}´

\paragraph{\textit{Bewertung}}
\newline
Interessant sind so berechnete Korrelationsverläufe etwa bei der Suche
von Outperformern innerhalb eines Index' oder auch beim Vergleich
zwischen unterschiedlichen Märkten. Recht bekannte Beispiele sind da
die negative Korrelation zwischen Renten- und Aktienmärkten oder auch
die zwischen in US-Dollar notierten Rohstoffen und Euro/US-Dollar
Wechselkurs.
===============================================

\section{RSQUARE (R-squared coefficient}

\paragraph{\textit{Typ}\quad Trendbestimmung}

\paragraph{\textit{Beschreibung}}
\newline
Dieser Koeffizient aus der Statik hat im Deutschen die Bezeichnungen
Bestimmtheitsmaß und Determinationskoeffizient. Allerdings gibt es da
leichte Variantionen in der Definition, deshalb sollte man sich bei
mehr Interesse an der Mathematik die entsprechenden Wikipediaseiten
oder ähnliches ansehen.
\newline
Dieses Modul berechnet den R\square einfach als das Quadrat eines
Spezialfalls des Bravais-Pearson Koeffizienten, der wiederum im Modul
BPCorrelation berechnet und auch dort beschrieben  wird. Letztlich
vergleicht das Modul eine gewählte (zeitabhängige) Datenreihe mit einer
vorgegebenen gleichmäßig ansteigenden Zeitreihe. Diese Zeitreihe erhält
man schlicht durch Durchzählen von 1 bis zur vorgegebenen
Periodenlänge. Betrachtet man also z.B. die Höchstkurse einer Periode
von 14 Tagen so vergleicht der R\square für jeden Tag die Höchstkurse
der vorangegangenen 14 Tage mit einer "Linie", die gleichmäßig von 1
bis 14 steigt. \newline Bewegt sich also die betrachtete Datenreihen im
Periodenzeitraum gleichmäßig linear nach oben, ergibt das einen Wert
von 1. Wegen der Quadratur gilt das auch bei einer gleichmäßig linearen
Bewegung nach unten. Ein Wert von Null gibt an, daß keine lineare
Bewegung erkennbar ist. Niedrige R\square Werte zeigen also eine
zufällige Bewegung der Datenreihe an, hohe eine stark lineare - also
einen Trend.

\paragraph{\textit{Parameter}}
\newline
Der erste Parameter gibt die Periodenlänge der betrachteten Zeitreihe
an, der zweite legt die Datenreihe fest.

\paragraph{\textit{Aufruf}}
\newline
I:RSquare IBM ´14 {I:Prices CLOSE}´

\paragraph{\textit{Bewertung}}
\newline
Prinzipiell kann der Indikator wohl schon bei der Bestimmung helfen, ob
eine trendlose Phase vorliegt oder nicht. Durch die Quadratur erreicht
man aber nicht nur ein besseres Herausarbeiten der Extremsituationen,
sondern verliert man eben auch die im ursprüglichen
Bravais-Pearson-Koeffizient noch vorhandene Unterscheidung von
positiver und negativer Korrelation.

==================================================
-- 
Delix <lin...@t-...>

------------------------------------------------------------------------------
Keep yourself connected to Go Parallel: 
DESIGN Expert tips on starting your parallel project right.
http://goparallel.sourceforge.net

Re: [GT] RSquare indicator

[Delix <del...@t-...>]

On Sun, 25 Nov 2012 19:48:14 +0100
Delix <del...@t-...> hatte geschrieben :

This is the German version of my manual entries for the BPCorrelation
and RSquare moduls.
Hope somebody is able to check it -- any comment is welcome.

 ==========================================
\section{BPCORRELATION (Bravais-Pearson Correlation Coefficient)}

\paragraph{\textit{Typ}\quad Basis}

\paragraph{\textit{Beschreibung}}
\newline
Dieser Koeffizient aus der statistischen Analyse berechnet die
Korrelation zweier Datenreihen, die hier bei GT beide jeweils die Zeit
-also in der Regel die Börsenhandelstage - als Bezugsachse haben. Es
wird also z.B. für das Datum X die Korrelation von Höchst- und
Tiefstkursen der vorangegangenen 14 Tage errechnet. Ein Wert von 1
ergibt sich dabei, falls beide Kurse völlig parallel verlaufen sind,
ein Wert von -1 bei genau entgegengesetztem Verlauf (d.h. die zweite
Datenreihe fällt im Vergleich zum Vortag um denselben Betrag, um den
die erste steigt) und ein Wert von Null zeigt an, daß es keinen
linearen Zusammenhang zwischen den beiden Kursverläufen gibt.

\paragraph{\textit{Parameter}}
\newline
Der erste Parameter legt die Länge der Periode an, für die der Verlauf
der beiden Datenreihen für das jeweilige Datum verglichen werden soll.
Mögliche Werte können sowohl eine Konstante als auch die Werte einer
Zeitreihe sein. Die beiden anderen Parameter geben die beiden zu
vergleichenden Datenreihen an.

\paragraph{\textit{Aufruf}}
\newline
I:BPCorrelation IBM ´20 {I:Prices HIGH} {I:Prices CLOSE}´

\paragraph{\textit{Bewertung}}
\newline
Interessant sind so berechnete Korrelationsverläufe etwa bei der Suche
von Outperformern innerhalb eines Index' oder auch beim Vergleich
zwischen unterschiedlichen Märkten. Recht bekannte Beispiele sind da
die negative Korrelation zwischen Renten- und Aktienmärkten oder auch
die zwischen in US-Dollar notierten Rohstoffen und Euro/US-Dollar
Wechselkurs.
===============================================

\section{RSQUARE (R-squared coefficient}

\paragraph{\textit{Typ}\quad Trendbestimmung}

\paragraph{\textit{Beschreibung}}
\newline
Dieser Koeffizient aus der Statik hat im Deutschen die Bezeichnungen
Bestimmtheitsmaß und Determinationskoeffizient. Allerdings gibt es da
leichte Variantionen in der Definition, deshalb sollte man sich bei
mehr Interesse an der Mathematik die entsprechenden Wikipediaseiten
oder ähnliches ansehen.
\newline
Dieses Modul berechnet den R\square einfach als das Quadrat eines
Spezialfalls des Bravais-Pearson Koeffizienten, der wiederum im Modul
BPCorrelation berechnet und auch dort beschrieben  wird. Letztlich
vergleicht das Modul eine gewählte (zeitabhängige) Datenreihe mit einer
vorgegebenen gleichmäßig ansteigenden Zeitreihe. Diese Zeitreihe erhält
man schlicht durch Durchzählen von 1 bis zur vorgegebenen
Periodenlänge. Betrachtet man also z.B. die Höchstkurse einer Periode
von 14 Tagen so vergleicht der R\square für jeden Tag die Höchstkurse
der vorangegangenen 14 Tage mit einer "Linie", die gleichmäßig von 1
bis 14 steigt. \newline Bewegt sich also die betrachtete Datenreihen im
Periodenzeitraum gleichmäßig linear nach oben, ergibt das einen Wert
von 1. Wegen der Quadratur gilt das auch bei einer gleichmäßig linearen
Bewegung nach unten. Ein Wert von Null gibt an, daß keine lineare
Bewegung erkennbar ist. Niedrige R\square Werte zeigen also eine
zufällige Bewegung der Datenreihe an, hohe eine stark lineare - also
einen Trend.

\paragraph{\textit{Parameter}}
\newline
Der erste Parameter gibt die Periodenlänge der betrachteten Zeitreihe
an, der zweite legt die Datenreihe fest.

\paragraph{\textit{Aufruf}}
\newline
I:RSquare IBM ´14 {I:Prices CLOSE}´

\paragraph{\textit{Bewertung}}
\newline
Prinzipiell kann der Indikator wohl schon bei der Bestimmung helfen, ob
eine trendlose Phase vorliegt oder nicht. Durch die Quadratur erreicht
man aber nicht nur ein besseres Herausarbeiten der Extremsituationen,
sondern verliert man eben auch die im ursprüglichen
Bravais-Pearson-Koeffizient noch vorhandene Unterscheidung von
positiver und negativer Korrelation.

==================================================
-- 
Delix <lin...@t-...>

Re: [GT] RSquare indicator

[Delix <del...@t-...>]

On Sun, 25 Nov 2012 09:50:10 -0800
"Robert A. Schmied" <ra...@ac...> hatte geschrieben :


> well the mailing list has yet again failed to forward the following to me
> from Derek in response to thomas' comments (that i also reposted) so i
> once again copy, rewrap at 80char/line and forward to the list:

Actually, I had no problems to receive the original messages
strait from the mailing list server. May be this is a specific issue of
your account or spam filter ???

> so now all that remains is to see if there is a way to resolve the initial
> confusion issue that started this in the first place:
> 
> i've hacked the following extensively revised pod into package GT::Indicators::RSquare
> -- doubtless i've mis-worded things, possibly made errors and generally made a mess
> of it (the pod) -- anyone one want to rework, correct, clarify or ???.

Thanks a lot to all for the feedback and input. I asked because of the
manual entry for this indicator, but never tried to use it (so far).
Guess I got it -- or most of it  ;-))

> 
> 
> =pod
> 
> =head1 GT::Indicators::RSquare
> 
> =head2 Synopsis
> 
> I:RSquare(14, "{I:Prices CLOSE}")
> 
> =head2 Overview
> 
> This function calculates the R-Squared coefficient.
> Originally defined as a technical indicator in 
> "The New Technical Trader" by Tushar S. Chande and Stanley Kroll.
> The intent is to use correlation against a monotonically increasing
> line as a means of quantifying the strength of a trend.
> 
> =head2 Calculation
> 
> For GT I:RSquare is computed using
> 
>  {I:BPCorrelation( 14, {I:Generic:Cum 1}, {I:Prices CLOSE} )
>  where
>   - 14 corresponds to RSquare arg #1
>   - {I:Generic:Cum 1} [and is fixed]
>   - {I:Prices CLOSE} is RSquare argument #2
> 
> as a reference metastock computes RSquare of an Indicator as follows:
> 
>   Pwr(Corr(Cum(1),C,14,0),2)
>   where Corr() (correlation) is computed:
>    - Cum(1) is count of periods so far and is fixed.
>    - C is current time period close price data series and can be altered,
>      corresponding to the 2nd RSquare argument.
>    - 14 is (default) time periods of interest and can be altered,
>      corresponding to the 1st RSquare argument.
>    - 0 is unknown as of this writing and is fixed.
>    - Pwr() simply squares the computed Corr() value.
> 
> =head2 Discussion
> 
> The RSquare of an Indicator shows how well the price
> approximates a linear regression line. It uses BPCorrelation
> to compute this.

And this is the point where I get stuck : I understood the code in the
way, that it approximates a monotiously increasinging line. That makes
no sense for an obiously downward trend.
I know, at this point I don't understand the procedure of the
BPCorrelation correctly. But I still can't catch the correct meaning of
it. Probably, I'm just unable to change my point of view...... 

> 
> To use RSquare you must choose a basis on which you want to
> compare the indicator to.  By default the basis is time
> (i.e., the numbers of periods so far). In other words,
> for your correlation you have the indicator on the Y axis
> and the periods on the X axis.

So what I get is a pair of values with the a number between 1 and 14 as
the x value and the computed R² as the y value -- right ?
> 
> =cut
> 
> 
> 
> [it might be a good idea to cc me explicitly because the list forwarding
> to me appears to be rather sporadic of late.]
> 
> 
> aloha
> 
> ras
> 
> 

I'll try in the next few weeks to write a desription of the
BPCorrelation and R², based on the new pod for the manual. Perhaps
somebody finds a way and the time to check the outcome  :-)))


-- 
Delix <lin...@t-...>

Re: [GT] RSquare indicator

[Robert A. S. <ra...@ac...>]

Shinaberry Derek wrote:
> Guten Abend Delix,
> 
> I come to the same conclusion as you did as to the the default
> behaviour of the RSquare indicator.  I can't fathom why you would
> ever want to use such a default, but it's possible that I'm missing
> something.  To my thinking, it probably shouldn't provide any default
> at all.  You generally want to calculate a correlation coefficient
> between two completely different time series, e.g. a stock and broad
> index or one sector index to a different sector index.  Looking for
> correlation between a stock's closing price and its moving average
> seems like a potentially dangerously misleading exercise to me.  The
> relationship between a MA and its price series is already known by
> definition, so why would you need to use statistics to look for a
> correlation between them?
> 
> Perhaps, the default should be removed from the code to preclude any
> potential confusion for future users?
> 
> Cheers, Derek
> 
> On Nov 17, 2012, at 20:41 , Delix wrote:
> 
> 
>> I have a problem to understand the meaning of this indicator with
>> the given defaults. My interpretation of the code is : the
>> BPCorrelation compares the number of the data records (= CUM 1) 
>> with the CLOSE prices. Is this correct or a misunderstanding ? I
>> would expect a comparison of a MA with the prices...... -- Delix
>> <lin...@t-...>
> 
> 
> 
> ------------------------------------------------------------------------------
>  Monitor your physical, virtual and cloud infrastructure from a
> single web console. Get in-depth insight into apps, servers,
> databases, vmware, SAP, cloud infrastructure, etc. Download 30-day
> Free Trial. Pricing starts from $795 for 25 servers or applications! 
> http://p.sf.net/sfu/zoho_dev2dev_nov
> 


well the mailing list has yet again failed to forward the following to me
from Derek in response to thomas' comments (that i also reposted) so i
once again copy, rewrap at 80char/line and forward to the list:

From: Shinaberry Derek <geniustrader@sh...> - 2012-11-25 12:47
Well, this discussion piqued my curiosity, so I went off and did some googling. 
This may be old ground for everyone else, but I wasn't aware that R-squared had
a definition as a technical analysis indicator.  I was only familiar with its
use in statistics in general.

It's use as a technical indicator was outlined in "The New Technical Trader" by
Tushar S. Chande and Stanley Kroll.  It hadn't occurred to me to use correlation
against a monotonically increasing line as a means of quantifying the strength
of a trend.

So, the way it is coded matches the definition in Chande and Kroll.  Due to my
newfound enlightenment, I therefore rescind my suggestion of removing the
default for RSquare.pm.

Perhaps next time I'll use Google before I open my mouth and insert my foot.

Cheers, Derek



so now all that remains is to see if there is a way to resolve the initial
confusion issue that started this in the first place:

i've hacked the following extensively revised pod into package GT::Indicators::RSquare
-- doubtless i've mis-worded things, possibly made errors and generally made a mess
of it (the pod) -- anyone one want to rework, correct, clarify or ???.


=pod

=head1 GT::Indicators::RSquare

=head2 Synopsis

I:RSquare(14, "{I:Prices CLOSE}")

=head2 Overview

This function calculates the R-Squared coefficient.
Originally defined as a technical indicator in 
"The New Technical Trader" by Tushar S. Chande and Stanley Kroll.
The intent is to use correlation against a monotonically increasing
line as a means of quantifying the strength of a trend.

=head2 Calculation

For GT I:RSquare is computed using

 {I:BPCorrelation( 14, {I:Generic:Cum 1}, {I:Prices CLOSE} )
 where
  - 14 corresponds to RSquare arg #1
  - {I:Generic:Cum 1} [and is fixed]
  - {I:Prices CLOSE} is RSquare argument #2

as a reference metastock computes RSquare of an Indicator as follows:

  Pwr(Corr(Cum(1),C,14,0),2)
  where Corr() (correlation) is computed:
   - Cum(1) is count of periods so far and is fixed.
   - C is current time period close price data series and can be altered,
     corresponding to the 2nd RSquare argument.
   - 14 is (default) time periods of interest and can be altered,
     corresponding to the 1st RSquare argument.
   - 0 is unknown as of this writing and is fixed.
   - Pwr() simply squares the computed Corr() value.

=head2 Discussion

The RSquare of an Indicator shows how well the price
approximates a linear regression line. It uses BPCorrelation
to compute this.

To use RSquare you must choose a basis on which you want to
compare the indicator to.  By default the basis is time
(i.e., the numbers of periods so far). In other words,
for your correlation you have the indicator on the Y axis
and the periods on the X axis.

=cut



[it might be a good idea to cc me explicitly because the list forwarding
to me appears to be rather sporadic of late.]


aloha

ras

Re: [GT] German Manual

[Delix <del...@t-...>]

On Sun, 11 Nov 2012 11:47:41 -0800
"Robert A. Schmied" <ra...@ac...> hatte geschrieben :

> Delix wrote:
> > Just for information :
> > I'm working occassionally on a German manual for GT. Progress is and
> > will be very slow. So far I finished the description of about half the
> > indicators. I 'll continue only the LaTex version, but at the moment
> > there is also a version in the Abiword format available.
> > The files can be found here :
> > https://gitorious.org/geniustrader-handbuch/deutsch/trees/Indikatoren
> > Maybe somebody wants to do some translation training and we can get
> > other versions of the manual, too .....
> 
> aloha delix
> 
> this is an interesting project.  as i'm am illiterate in any human
> language but [american] english [some probably question even that
> assertion] i can imagine instances where translating from english
> to another language might be less than helpful or just plain problematic.
> so if anyone involved in this effort encounters pod or other explanatory
> text that should be improved upon (in english) you are encouraged
> to report it.

Actually, at the beginning I wanted to continue with Oliver Bresserts
attempts to write an English manual. But soon I realized my English
wasn't good enough to handle all the financial, mathematical and
programming- related expression. So, I had a look at the excellent
lexicon of www. tradesignalonline.com which is available in English and
German. But I still think I better stick with the German version ;-)


> 
> delix -- since you have not provided a readme (in english) nor any
> english explanatory description of your approach, methodology, source
> material, etc i can only guess at the final end result -- a single
> large volume containing the bulk of the gt toolkit and application
> scripts pod. could you explain your design and plans so interested
> others can better integrate and align their efforts with yours.

Actually, I was planning to write a manual -- that means an extended
version of some parts the wiki including the infos from the pods an
additinally some hints and comments on the underlying financial and
mathematical tools and methods. For example, the indicators modules are
not described in the wiki yet and many of them have extremly short pods.
I had massive problems to figure out, what the file names really mean
or even more, what the specific indicator in the GT version finally
indicates......
I would skip the installation and setup parts of the wiki in
the manual.

However, I'm well aware this would take ages.......
So, at the moment I writing text files for the single indicators,
organized by the following scheme :
- name
- type (for example "Moving average", "volatility",...)
- description (what it is based on, how it works and how it can be
interpreted)
- mandatoric and optional parameters
- invocation
- my personal validations, comments and comparison with other indicators

Every module gets it own .tex file and represents a LaTeX \subsection.
At the end all files are added with \includefile{...} command to the
basic manual.tex file. The final output can be a HTML-file or a .pdf --
haven't decided yet.

Any suggestions, comments and help is welcome. But as I wrote
before : progress will be veeeery sslooooowwwwwly....   



> 
> 
> regards
> 
> ras
> 
> complete aside: how does locale and the .po files (found in many
> gnu/opensource projects) interrelate in order to provide on-the-fly
> multi-language support? is this stuff supported by standard (but old)
> perl for pod?

.po files a file with formats accordingly to the translation program
poedit. This is probably the most often used translation tools in
GNU/Linux nowadays.
The main body of he files is similar to the plain locale files, but
there is a (small) header added which makes the administration pretty
easy - regarding version control, authors etc.



> 
> 
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_nov


-- 
Delix <lin...@t-...>

Re: [GT] German Manual

[Robert A. S. <ra...@ac...>]

Delix wrote:
> Just for information :
> I'm working occassionally on a German manual for GT. Progress is and
> will be very slow. So far I finished the description of about half the
> indicators. I 'll continue only the LaTex version, but at the moment
> there is also a version in the Abiword format available.
> The files can be found here :
> https://gitorious.org/geniustrader-handbuch/deutsch/trees/Indikatoren
> Maybe somebody wants to do some translation training and we can get
> other versions of the manual, too .....

aloha delix

this is an interesting project.  as i'm am illiterate in any human
language but [american] english [some probably question even that
assertion] i can imagine instances where translating from english
to another language might be less than helpful or just plain problematic.
so if anyone involved in this effort encounters pod or other explanatory
text that should be improved upon (in english) you are encouraged
to report it.

delix -- since you have not provided a readme (in english) nor any
english explanatory description of your approach, methodology, source
material, etc i can only guess at the final end result -- a single
large volume containing the bulk of the gt toolkit and application
scripts pod. could you explain your design and plans so interested
others can better integrate and align their efforts with yours.


regards

ras

complete aside: how does locale and the .po files (found in many
gnu/opensource projects) interrelate in order to provide on-the-fly
multi-language support? is this stuff supported by standard (but old)
perl for pod?

[GT] German Manual

[Delix <del...@t-...>]

Just for information :
I'm working occassionally on a German manual for GT. Progress is and
will be very slow. So far I finished the description of about half the
indicators. I 'll continue only the LaTex version, but at the moment
there is also a version in the Abiword format available.
The files can be found here :
https://gitorious.org/geniustrader-handbuch/deutsch/trees/Indikatoren
Maybe somebody wants to do some translation training and we can get
other versions of the manual, too .....
-- 
Delix <lin...@t-...>

Re: [GT] Thoughts On Fleshing Out The "manage_portfolio.pl <portfolio> report performance" Command

[Robert A. S. <ra...@ac...>]

Shinaberry Derek wrote:
> GT'ers...
> 
> I've been doing some thinking about how I'd would like to fill out the "report" command of the manage_portfolio.pl script.  The "historic" and "positions" sub-commands already seem to work reasonably well.  I've tweaked the section for "analysis" to allow the call to GT::Report::PortfolioAnalysis to operate more smoothly.  But most of my thinking has gone into the heretofore empty section for the "performance" sub-command.
> 
> For my money, pun intended, returns in percentage terms are how I like to evaluate overall performance as opposed to raw nominal monetary terms.  I thought about creating sub sub-commands for different sorts of rates of return, but that seemed like too many levels deep.  It doesn't seem too bad to report different sorts of rates of return all in a single report.  I'm contemplating the following different rates:
> 
> 	overall rate of return (raw ROI),
> 	annualized rate of return (CAGR),
> 	DWRR (IRR; portfolio context) [1],
> 	DWRR (IRR; trades only context) [1],
> 	TWRR (portfolio context) [1],
> 	TWRR (trades only context) [1]
> 
> [1] Since DWRR/TWRR analyze cash flows, it is possible to analyze just the cash flows from the trades (trades only context) or also include the starting and ending values of the containing portfolio as cash flows (portfolio context).  The portfolio context would take into account the lack of return on any idle money not utilized by the series of trades.
> 


question: what about 'idle money' held in an account that draws an interest rate from the broker?

question: how about a way to add and withdraw 'idle money' held in an account during the time period of interest.

question: do you plan on integrating the use of margin money, and if so to what extent?


> I also thought it would be valuable to be able to see performance data on sub-components of the actual portfolio.  My initial list of ways to specify what to include or exclude from the performance report are:
> 
> 	time period (include only; I suppose you could want to exclude a time period, but that leads to discontinuities which could be problematic),
> 	security symbol (include or exclude; via a list or a regexp),
> 	source (a.k.a. system; include or exclude; via a list or a regexp)
> 
> One last way of slicing up the performance report data that seemed valuable was to be able to specify grouping in order to have subtotal calculations by group in addition to overall performance calculations.  My list of ways to group include:
> 
> 	year,
> 	quarter,
> 	month,
> 	security symbol (a.k.a. code),
> 	source (a.k.a. system)
> 
> So I am proposing the following as the command line interface for the report performance sub-command:
> 
> manage_portfolio.pl <portfolio> report [options] performance
> 
> where the available options options are:
> 
> 	--start <date>
> 	--end <date>
> 	--include-code [code1, code2, ... | regexp]
> 	--exclude-code [code1, code2, ... | regexp]
> 	--include-source [source1, source2, ... | regexp]
> 	--exclude-source [source1, source2, ... | regexp]
> 	--group-by=[year|quarter|month|code|source]
> 
> --start will replace the currently present, but unutilized, --since option.  Specifying a date will restrict the report to transactions that occur on or after that date and are chosen by the selection criteria options specified, if any.  If not specified, the start date will be the date of the first transaction chosen by the selection criteria options specified, if any, or the first transaction in the portfolio if no selection criteria are specified.
> 
> --end will replace the current present, but unutilized, --until option.  Specifying a date will restrict the report to transactions that occur on or before that date and are chosen by the selection criteria specified, if any.  If not specified, the end date will be the date of the last transaction chosen by the selection criteria options specified, if any, or the last transaction in the portfolio if no selection criteria are specified.
> 
> --include-code narrows the scope of transactions selected for the report to only those security codes specified.  The specification of security codes to include can take the form of either a comma separated list of security codes or a regular expression that will be matched against all the security codes present in the portfolio.  If the --include-code option is specified, the --exclude-code option is not allowed and an error will result if both are specified.  If neither --include-code nor --exclude-code are specified, then all transactions in the portfolio will be reported on.
> 
> --exclude-code narrows the scope of transactions selected for the report to those security codes which are not specified in the option.  The specification of security codes to exclude can take the form of either a comma separated list of security codes or a regular expression that will be matched against all the security codes present in the portfolio.  If the --exclude-code option is specified, the --include-code option is not allowed and an error will result in both are specified.  If neither --include-code nor --exclude-code are specified, then all transactions in the portfolio will be reported on.
> 
> --include-source narrows the scope of transactions selected for the report to only those that match the sources specified.  The specification of sources to include can take the form of either a comma separated list of sources or a regular expression that will be matched against all the sources present in the portfolio.  If the --include-source option is specified, the --exclude-source option is not allowed and an error will result if both are specified.  If neither --include-source nor --exclude-source are specified, then all transactions in the portfolio will be reported on.
> 
> --exclude-source narrows the scope of transactions selected for the report to those sources which are not specified in the option.  The specification of sources to exclude can take the form of either a comma separated list of sources or a regular expression that will be matched against all the sources present in the portfolio.  If the --exclude-source option is specified, the --include-source option is not allowed and an error will result in both are specified.  If neither --include-source nor --exclude-source are specified, then all transactions in the portfolio will be reported on.
> 
> --group-by=<group-type> causes the transactions selected for the report to be subdivided into groups of the specified group type.  A subtotal for each group type will be reported in addition to the overall total for all transactions.  The available group types are: year, quarter, month, code, or source.
> 
> 
> And finally, some thoughts about the details of implementing the above described reporting interface...
> 
> o I'm planning on using the modules Finance::Math::IRR and Finance::Performance::Calc for DWRR and TWRR respectively rather than reinvent the wheel.  Not sure whether to make then hard requirements or to try and fail gracefully if they aren't present on the user's system.
> 

since these are not yet in the known list of perl dependencies but also just for better coding i strongly encourage failing gracefully with a useful message about what caused it and what the user can to do about it.

Finance::Performance::Calc should not be too much bother as it has no additional dependencies.

but Finance::Math::IRR is a bother because it also needs Math::Polynom and that module needs Data::Float.


> o In running some test scenarios, I discovered several situations that might need some tweaks made to allow getting transactions into GT in the proper fashion.  They aren't really problems when it comes to reporting performance, but rather have to do with grouping transactions into positions.  There isn't currently any way to specify that a transaction is part of a specific position.
> 
> 	1) The obvious problem here is trading the same security in multiple systems.  This one can be solved by using the --source option to specify which system the transaction belongs to.  
> 


this is a 'typical' problem when trying to extend the gt backtest portfolio into a practical real-world trading transaction record.

this is likely oversimplifying, missing the obvious, or just flat out wrong, but it appears to me that gt backtest portfolios track positions and account history details using GT::Portfolio::Order data objects, and each 'position' will have a unique 'id', and the 'position' is closed when the associated 'position' for 'id' share qty reaches zero based on a CloseStrategy order.

there isn't a 'known' way to simulate/emulate/create same (e.g. CloseStrategy orders) when creating a real-world trading portfolio via any current gt script app wben importing or creating a gt portfolio account file. creating such data object(s) is probably is not a significant difficulty, but it just is not being done by any of the apps i'm familiar with.


> 	2) A second problem I discovered was with some day trading transactions where I closed out a position in one direction and opened a new position in the opposite direction all in a single transaction.  When imported into GT, everything stayed in a single position until the balance of shares actually hit zero rather than just passed through zero.  Some additional logic to handle this doesn't seem like too big an issue.
> 


also see comment above

this is the time-frame versus time-period problem and how gt records positions (but not transactions) in a portfolio.  from my experience you will really want/need to gather and process data in terms of real-world 'transactions' and then perform accounting analysis once at the end of day based on the account 'transactions'.

this approach might/could also lend itself to adding on/in support for miscellaneous corporate actions like stock splits and dividend payouts.


> 	3) A third situation arose when wanting to rollover an expiring futures or options position without actually closing out the position.  I handle these sorts of trades by using a security code symbol that indicates the underlying security and market without the time sensitive part.  So to GT it looks like I'm selling and buying the same code at nearly the same time.  If it happens that the transaction for the new security occurs before the transaction for the expiring security, everything works out fine, because it appears to GT that you are increasing the position size and then decreasing it a short time later.  But if the transactions happen in the opposite order, then the position is closed out and a new position is opened.  I'm not quite sure how to best handle this situation.  I'm not sure you could handle it without having some sort of a priori knowledge.  Maybe a special new option that tells GT to not close a position even though the size drops to zero.  Or maybe si
mply noting that it is an issue and requires the proper ordering of input transaction on the part of the operator.  I'm open to any ideas that anyone might have about this.
> 


extending gt to handle options without integrated support for option chain data might be hi-risk, but since i'm unfamiliar with this sort of trading i'm not likely to be able to see the trees or the forest.


> o To be able to pick arbitrary start and/or end dates for reporting requires the ability to evaluate portfolio values in the middle of positions.  For example, a start date may come after a position was opened, but before it was closed.  Alternatively, an end date may be specified where a, since closed, position is still open.  This will require access to price data for the period that a position was open.  This could be problematic if the list of unique symbols is quite large, since the database access mechanism seems to want to cache entire price histories rather than allow one off price queries.  This is understandable given GT's primary purpose, but presents a potential stumbling block to this particular desire to extend GT in slightly different directions.
> 


much of the analysis issues would likely be resolved if they were performed on a generated transaction record object that contained only the necessary data-elements, but the overall 'data hoggyness' of gt would still remain, but that's not really an issue given todays multi-gigabyte ram banks and multi-terabyte filesystems; right?! [kinda tongue in check]


> o Also problematic is the notion of a portfolio report once you start slicing and dicing with include or exclude filters.  What exactly do you use for the starting and ending portfolio value?  It seems to make sense to just use the actual portfolio value as of the date of the start of the report.  That is simple enough if the start date is just the simplest default case, which is the date of the first transaction in the portfolio.  But if the transaction list is filtered from the portfolio, what do you use for the starting value of the portfolio?  It could be the actual value of the portfolio including all transactions that occurred prior to the starting date.  It could be a theoretical portfolio value constructed from the original initial value of the portfolio and the application of any transactions in the filtered list that occurred prior to the starting date.  It could also be some hypothetical value specified via some command line option specifically for situations lik
e this.
> 
> o I'm not sure it's worth trying to do anything about programmatically, but might be worth some commentary in the pod to warn about the following scenario.  If a start date is specified that is significantly before the date of the first transaction, it could have a significant impact on any DWRR or CAGR calculation since there could be a lot of "dead" time during the report time frame.  The same is true if an end date is specified that is significantly later than the date of the last transaction.  I'm not even sure what you might try to do programmatically to address this.
> 
> o At the moment, I am planning on reporting both DWRR and TWRR in both contexts, i.e. trades only and portfolio.  I suppose this could be offered up as a command line option to select one or the other for the report rather than including both.  I'm not sure whether showing both might be confusing or not.
> 


not explicitly mentioned here are your plans for the output format: the ascii text format is generally fixed, but using HTML::Mason and a corresponding template could/would allow 'customizations' anyone wants/needs and is willing to do the work required to create the appropriate template provided adequate disclosure of data object names is provided. or at least that is what i think the purpose/intent/use of and for HTML::Mason is.


> o Open and closed positions are stored separately in the portfolio data structure.  Ideally, the performance report would including both open and closed positions.  I have no idea how difficult that might turn out to be in practice.  I'd like to reserve the right to not include open positions in any first cut that might make it to the trunk head.
> 


i encourage your undertaking to improve and enhance the current gt script app manage_portfolio.pl but given all the changes you want to make i'd prefer to see you pick a new name for it. this way you can make whatever changes you want without trying to maintain any backward compatibility. plus it gives everyone the ability to easily perform comparisons between the new improved version and the current model.

as a case in point -- just by require/use of the two (4) additional prerequisite modules the new manage_portfolio.pl may no longer function on some perl platforms without upgrade (my antique v5.6.2 for example). that is something that i think should be avoided (e.g. breaking something that works in order to improve it).


> 
<snip>


keep the list posted with your efforts, meaning post your source code to the list in whatever form you find the easiest to use and manage. interested users are encouraged to evaluate and comment.


aloha

ras

[GT] Thoughts On Fleshing Out The "manage_portfolio.pl <portfolio> report performance" Command

[Shinaberry D. <gen...@sh...>]

GT'ers...

I've been doing some thinking about how I'd would like to fill out the "report" command of the manage_portfolio.pl script.  The "historic" and "positions" sub-commands already seem to work reasonably well.  I've tweaked the section for "analysis" to allow the call to GT::Report::PortfolioAnalysis to operate more smoothly.  But most of my thinking has gone into the heretofore empty section for the "performance" sub-command.

For my money, pun intended, returns in percentage terms are how I like to evaluate overall performance as opposed to raw nominal monetary terms.  I thought about creating sub sub-commands for different sorts of rates of return, but that seemed like too many levels deep.  It doesn't seem too bad to report different sorts of rates of return all in a single report.  I'm contemplating the following different rates:

	overall rate of return (raw ROI),
	annualized rate of return (CAGR),
	DWRR (IRR; portfolio context) [1],
	DWRR (IRR; trades only context) [1],
	TWRR (portfolio context) [1],
	TWRR (trades only context) [1]

[1] Since DWRR/TWRR analyze cash flows, it is possible to analyze just the cash flows from the trades (trades only context) or also include the starting and ending values of the containing portfolio as cash flows (portfolio context).  The portfolio context would take into account the lack of return on any idle money not utilized by the series of trades.

I also thought it would be valuable to be able to see performance data on sub-components of the actual portfolio.  My initial list of ways to specify what to include or exclude from the performance report are:

	time period (include only; I suppose you could want to exclude a time period, but that leads to discontinuities which could be problematic),
	security symbol (include or exclude; via a list or a regexp),
	source (a.k.a. system; include or exclude; via a list or a regexp)

One last way of slicing up the performance report data that seemed valuable was to be able to specify grouping in order to have subtotal calculations by group in addition to overall performance calculations.  My list of ways to group include:

	year,
	quarter,
	month,
	security symbol (a.k.a. code),
	source (a.k.a. system)

So I am proposing the following as the command line interface for the report performance sub-command:

manage_portfolio.pl <portfolio> report [options] performance

where the available options options are:

	--start <date>
	--end <date>
	--include-code [code1, code2, ... | regexp]
	--exclude-code [code1, code2, ... | regexp]
	--include-source [source1, source2, ... | regexp]
	--exclude-source [source1, source2, ... | regexp]
	--group-by=[year|quarter|month|code|source]

--start will replace the currently present, but unutilized, --since option.  Specifying a date will restrict the report to transactions that occur on or after that date and are chosen by the selection criteria options specified, if any.  If not specified, the start date will be the date of the first transaction chosen by the selection criteria options specified, if any, or the first transaction in the portfolio if no selection criteria are specified.

--end will replace the current present, but unutilized, --until option.  Specifying a date will restrict the report to transactions that occur on or before that date and are chosen by the selection criteria specified, if any.  If not specified, the end date will be the date of the last transaction chosen by the selection criteria options specified, if any, or the last transaction in the portfolio if no selection criteria are specified.

--include-code narrows the scope of transactions selected for the report to only those security codes specified.  The specification of security codes to include can take the form of either a comma separated list of security codes or a regular expression that will be matched against all the security codes present in the portfolio.  If the --include-code option is specified, the --exclude-code option is not allowed and an error will result if both are specified.  If neither --include-code nor --exclude-code are specified, then all transactions in the portfolio will be reported on.

--exclude-code narrows the scope of transactions selected for the report to those security codes which are not specified in the option.  The specification of security codes to exclude can take the form of either a comma separated list of security codes or a regular expression that will be matched against all the security codes present in the portfolio.  If the --exclude-code option is specified, the --include-code option is not allowed and an error will result in both are specified.  If neither --include-code nor --exclude-code are specified, then all transactions in the portfolio will be reported on.

--include-source narrows the scope of transactions selected for the report to only those that match the sources specified.  The specification of sources to include can take the form of either a comma separated list of sources or a regular expression that will be matched against all the sources present in the portfolio.  If the --include-source option is specified, the --exclude-source option is not allowed and an error will result if both are specified.  If neither --include-source nor --exclude-source are specified, then all transactions in the portfolio will be reported on.

--exclude-source narrows the scope of transactions selected for the report to those sources which are not specified in the option.  The specification of sources to exclude can take the form of either a comma separated list of sources or a regular expression that will be matched against all the sources present in the portfolio.  If the --exclude-source option is specified, the --include-source option is not allowed and an error will result in both are specified.  If neither --include-source nor --exclude-source are specified, then all transactions in the portfolio will be reported on.

--group-by=<group-type> causes the transactions selected for the report to be subdivided into groups of the specified group type.  A subtotal for each group type will be reported in addition to the overall total for all transactions.  The available group types are: year, quarter, month, code, or source.


And finally, some thoughts about the details of implementing the above described reporting interface...

o I'm planning on using the modules Finance::Math::IRR and Finance::Performance::Calc for DWRR and TWRR respectively rather than reinvent the wheel.  Not sure whether to make then hard requirements or to try and fail gracefully if they aren't present on the user's system.

o In running some test scenarios, I discovered several situations that might need some tweaks made to allow getting transactions into GT in the proper fashion.  They aren't really problems when it comes to reporting performance, but rather have to do with grouping transactions into positions.  There isn't currently any way to specify that a transaction is part of a specific position.

	1) The obvious problem here is trading the same security in multiple systems.  This one can be solved by using the --source option to specify which system the transaction belongs to.  

	2) A second problem I discovered was with some day trading transactions where I closed out a position in one direction and opened a new position in the opposite direction all in a single transaction.  When imported into GT, everything stayed in a single position until the balance of shares actually hit zero rather than just passed through zero.  Some additional logic to handle this doesn't seem like too big an issue.

	3) A third situation arose when wanting to rollover an expiring futures or options position without actually closing out the position.  I handle these sorts of trades by using a security code symbol that indicates the underlying security and market without the time sensitive part.  So to GT it looks like I'm selling and buying the same code at nearly the same time.  If it happens that the transaction for the new security occurs before the transaction for the expiring security, everything works out fine, because it appears to GT that you are increasing the position size and then decreasing it a short time later.  But if the transactions happen in the opposite order, then the position is closed out and a new position is opened.  I'm not quite sure how to best handle this situation.  I'm not sure you could handle it without having some sort of a priori knowledge.  Maybe a special new option that tells GT to not close a position even though the size drops to zero.  Or maybe simply noting that it is an issue and requires the proper ordering of input transaction on the part of the operator.  I'm open to any ideas that anyone might have about this.

o To be able to pick arbitrary start and/or end dates for reporting requires the ability to evaluate portfolio values in the middle of positions.  For example, a start date may come after a position was opened, but before it was closed.  Alternatively, an end date may be specified where a, since closed, position is still open.  This will require access to price data for the period that a position was open.  This could be problematic if the list of unique symbols is quite large, since the database access mechanism seems to want to cache entire price histories rather than allow one off price queries.  This is understandable given GT's primary purpose, but presents a potential stumbling block to this particular desire to extend GT in slightly different directions.

o Also problematic is the notion of a portfolio report once you start slicing and dicing with include or exclude filters.  What exactly do you use for the starting and ending portfolio value?  It seems to make sense to just use the actual portfolio value as of the date of the start of the report.  That is simple enough if the start date is just the simplest default case, which is the date of the first transaction in the portfolio.  But if the transaction list is filtered from the portfolio, what do you use for the starting value of the portfolio?  It could be the actual value of the portfolio including all transactions that occurred prior to the starting date.  It could be a theoretical portfolio value constructed from the original initial value of the portfolio and the application of any transactions in the filtered list that occurred prior to the starting date.  It could also be some hypothetical value specified via some command line option specifically for situations like this.

o I'm not sure it's worth trying to do anything about programmatically, but might be worth some commentary in the pod to warn about the following scenario.  If a start date is specified that is significantly before the date of the first transaction, it could have a significant impact on any DWRR or CAGR calculation since there could be a lot of "dead" time during the report time frame.  The same is true if an end date is specified that is significantly later than the date of the last transaction.  I'm not even sure what you might try to do programmatically to address this.

o At the moment, I am planning on reporting both DWRR and TWRR in both contexts, i.e. trades only and portfolio.  I suppose this could be offered up as a command line option to select one or the other for the report rather than including both.  I'm not sure whether showing both might be confusing or not.

o Open and closed positions are stored separately in the portfolio data structure.  Ideally, the performance report would including both open and closed positions.  I have no idea how difficult that might turn out to be in practice.  I'd like to reserve the right to not include open positions in any first cut that might make it to the trunk head.

[GT] update versions on trunk-head for: GT/BackTest.pm and Scripts/backtest_multi.pl

[Robert A. S. <ra...@ac...>]

gt'ers

these (just committed) revised files are the result of dereks work
along with a bit of reformatting, some pod text and the like by me.

the basic command line (csh) for evaluation looks like this:
    ./backtest_multi.pl \
      --opt='DB::module=Text' \
      --option='DB::text::directory=/usr/local/src/genius_trader/sample_data' \
      --start=2001-10-05 --end=2002-10-04 \
      --output-directory=/var/tmp --set=bt_multi \
      --broker=NoCosts \
      ./bt_multi_sample_codes_ex1.txt \
      ./bt_multi_3_sys.txt

where the input market file (./bt_multi_sample_codes_ex1.txt) contains:
13000
13330
12040

and the systems file (./bt_multi_3_sys.txt) contains:
SY:G \
 { S:G:CrossOverUp   \
    { I:SMA 20 {I:Prices CLOSE} } { I:SMA 60 {I:Prices CLOSE} } } \
 { S:G:CrossOverDown \
    { I:SMA 20 {I:Prices CLOSE} } { I:SMA 60 {I:Prices CLOSE} } } \
\
 | CS:OppositeSignal \
 | MM:Portfolio::FixedFractional 15

SY:TTS | CS:PartialStop 7 | MM:VAR 20 4

SY:TFS 50 7 | CS:SY:TFS 50 | CS:Stop:Fixed 6 \
| MM:VAR 10 2 | MM:PositionSizeLimit 100


testing conducted:

my testing has not been comprehensive but i've noticed some anomalies
when using a week timeframe (and near term dates and market symbols
GE VOD VZ and T), and have yet to determine the root cause(s). the
anomalies have to do with missing trades at end of the time period
of interest and actual trade dates within the time period of interest.


utility of ./backtest_multi.pl:

without a graphic of portfolio holdings and value over time i don't
fully grok the trading events history list. it is sorted based on
holding dates of each position. maybe there needs to be additional
line(s) that reflect current holdings (if any) and total portfolio
value (cash plus holdings market value) after each trade?

i also don't fully understand how the multiple systems interact (if
they interact at all) during the analysis.



aloha

ras

Re: [GT] epoch-fix comments was: Re: Problems With Rev 713 GT/DateTime.pm - Fixing the UNIX Epoch Problem

[Derek S. <gen...@sh...>]

On Aug 24, 2012, at 21:25 , Robert A. Schmied wrote:

> aloha derek
> 
> i reviewed subject patch and applied it to
> ../GT/DateTime.pm
> and
> ../GT/Prices.pm
> and skipped the changes to ../GT/DateTime/*.pm as these files
> are no longer used.
> 
> this was done in my production-development environment, so i could
> evaluate the change using my available testing files.
> 
> 
> results were an increase in number of tests failing for every date
> considered. this shouldn't be considered an indictment of your changes
> as it is possible my testing has errors.
> 
> my baseline date test generally fail some (1..13) of the 120 odd tests,
> with your patch applied it appeared all dates tested failed between 13
> and 15 tests.  this suggests to me that it's my testing that might be
> failing *not* your date conversion.
> 
> also note that for the most part these tests are atypical of most of
> date conversions performed by routine gt operations, so it's very likely
> none of these failures are that critical ...
> 
> as i don't have the time or energy right now to investigate in any
> depth i've reverted the changed files and will keep this as an item
> on the to-do list for later when i get a round tuit.
> 
> 
> aloha
> 
> ras
> 
> ps -- when i get you the tarballs (3 tarballs, about 3mb total)
> you can perform your own testing on this. -- ras
> 
<big snip>
Robert,

Here is an updated patch file for GT/DateTime.pm and GT/Prices.pm.  I've also included a copy of the test output showing 123 passed tests and a patch file for datetime.t.

My original patch did fix the whole 50 year window problem that resulted in old quotes being unusable, what it didn't account for was timezone and DST issues.

It turns out that localtime and timelocal adjust for timezone and daylight savings time, while gmtime and timegt do not.  So, the update uses the later over the former to ignore those effects.

The changes to the test code were minimal.  Just a call to tell Date::Manip to ignore timezones and commented out the code that attempted to deal with daylight savings time, since it is no longer an issue.  I didn't attempt to tidy the test code in any way.  Just brute forced what was required to pass the tests.

Give it a go and see if it's worthy to commit.

Cheers,
Derek

Re: [GT] detail findings on GT/BackTest.pm was Re: Problems With Rev 714 GT/BackTest.pm - Fixing The UNIX Epoch Problem

[Shinaberry D. <gen...@sh...>]

On Aug 27, 2012, at 19:14 , Robert A. Schmied wrote:

> Shinaberry Derek wrote:
>> Hi Robert,
>> I did some spelunking through the -devel list archives and the svn
>> repository to see what I could discover about your description of the
>> intended design goal of GT/BackTest.pm not depending on GT/Tools.pm,
>> script apps needing to invoke find_calculator. I think four of the
>> six issues I raised involve changes made to move GT/BackTest.pm and
>> Scripts/backtest_multi.pl towards that goal.
> 
> as did i along with a bit of version comparison and evaluations and i've
> discovered that some place along the way some 'refactoring' and some
> 'harmonizing' has happened to trunk-head Scripts/backtest_multi.pl and
> i completely missed it.
> 
> so i have to conclude that the gt user community think the original xml
> based control file form employed by Scripts/backtest_multi.pl isn't the
> way to proceed. commentary welcome. i tend to agree -- without a way to
> manage the xml control file this approach was destined to fail.
> 
> so the Scripts/backtest_multi.pl pod isn't completely incorrect as i wrote
> earlier, just partly. backtest_multi.pl does now read plain text input
> control files: the first a list of codes, the second a list of trading
> systems.  however, the pod references to backtest_many.pl should really be
> backtest_multi.pl to give the casual reader confidence that the pod just
> isn't a cut+paste job applied without any thought. all that and more have
> been corrected in the pod.
> 
> and the files Scripts/backtest_multi.pl and GT/BackTest.pm have had other
> required changes to make them operational, at least on my sample of 1 system,
> a non-common system at that. all this without wedging find_calculator into
> GT/BackTest.pm just because it could be (wedged in).
> 
> i've attached the two changed files for your evaluation and comment and
> will commit, in due course, possibly in slightly different form as my
> additional evaluations, tests, and pod tweakage proceed and as commentary
> is received, discussed and factored in.
> 
I call your BackTest.pm and backtest_multi.pl and raise you these versions.

I was just about ready to post my take on these files to address the issues I
raised in this thread when yours appeared in my inbox.

I merged your version into mine to arrive at these.  It achieves the stated goal of
of calling find_calculator from backtest_multi.pl and also optimizes the code in
order to not have to have some essentially duplicated code from find_calculator
in GT::BackTest::backtest_multi and some timeframe related code from
get_timeframe_data in backtest_multi.pl.

When I ran a sample system through my own hacked code tree and these versions
in the trunk head, the results were identical.

Give 'em a look and see what you think.

Cheers,
Derek

ps - I think I've got an update to the DateTime.pm patch and your private test code
that, I think, may finally put that to bed.  I hope.  It essentially deals with timezone and
daylight savings time issues by not dealing with them.  In other words, it just takes
whatever date/time it's given and never attempts to adjust it for either timezone or
daylight savings time.  There is one more thing I want to check, but it's going to have
to wait until tomorrow.
> 
> aloha
> 
> ras
> 
> ps -- i don't know if these changes address any of the issues raised by
> this thread, but they do (should) correct the *non-operational* issue(s) -- ras
> 
> 
> 
<snip>

Re: [GT] detail findings on GT/BackTest.pm was Re: Problems With Rev 714 GT/BackTest.pm - Fixing The UNIX Epoch Problem

[Robert A. S. <ra...@ac...>]

Shinaberry Derek wrote:
> Hi Robert,
> 
> I did some spelunking through the -devel list archives and the svn
> repository to see what I could discover about your description of the
> intended design goal of GT/BackTest.pm not depending on GT/Tools.pm,
> script apps needing to invoke find_calculator. I think four of the
> six issues I raised involve changes made to move GT/BackTest.pm and
> Scripts/backtest_multi.pl towards that goal.

as did i along with a bit of version comparison and evaluations and i've
discovered that some place along the way some 'refactoring' and some
'harmonizing' has happened to trunk-head Scripts/backtest_multi.pl and
i completely missed it.

so i have to conclude that the gt user community think the original xml
based control file form employed by Scripts/backtest_multi.pl isn't the
way to proceed. commentary welcome. i tend to agree -- without a way to
manage the xml control file this approach was destined to fail.

so the Scripts/backtest_multi.pl pod isn't completely incorrect as i wrote
earlier, just partly. backtest_multi.pl does now read plain text input
control files: the first a list of codes, the second a list of trading
systems.  however, the pod references to backtest_many.pl should really be
backtest_multi.pl to give the casual reader confidence that the pod just
isn't a cut+paste job applied without any thought. all that and more have
been corrected in the pod.

and the files Scripts/backtest_multi.pl and GT/BackTest.pm have had other
required changes to make them operational, at least on my sample of 1 system,
a non-common system at that. all this without wedging find_calculator into
GT/BackTest.pm just because it could be (wedged in).

i've attached the two changed files for your evaluation and comment and
will commit, in due course, possibly in slightly different form as my
additional evaluations, tests, and pod tweakage proceed and as commentary
is received, discussed and factored in.


aloha

ras

ps -- i don't know if these changes address any of the issues raised by
this thread, but they do (should) correct the *non-operational* issue(s) -- ras



<snip>

> 
> Cheers,
> Derek
> 
> On Aug 22, 2012, at 21:29 , Robert A. Schmied wrote:
> 
> 

< snip >

Re: [GT] detail findings on GT/BackTest.pm was Re: Problems With Rev 714 GT/BackTest.pm - Fixing The UNIX Epoch Problem

[Robert A. S. <ra...@ac...>]

Shinaberry Derek wrote:
> Hi Robert,
> 
> I did some spelunking through the -devel list archives and the svn repository to see what I could discover about your description of the intended design goal of GT/BackTest.pm not depending on GT/Tools.pm, script apps needing to invoke find_calculator.  I think four of the six issues I raised involve changes made to move GT/BackTest.pm and Scripts/backtest_multi.pl towards that goal.
> 
> This is more than a bit long winded.  I hope you'll forgive the length, but there was a lot of ground for me to cover.  There are some interspersed comments and then much longer commentary towards the end.
> 
> Cheers,
> Derek
> 
> On Aug 22, 2012, at 21:29 , Robert A. Schmied wrote:
> 
> 

< large snip >

> 
> So, my conclusions...
> 
> I'm not sure whether demanding that BackTest.pm NOT use Tools.pm is a worthy design goal or not.  I can say that backtest_multi.pl is a bit of the odd one out and that oddness is what drives BackTest.pm to use Tools.pm.
> 
> I'd like to restore the trunk head to a functioning backtest_multi.pl, BackTest.pm combination fairly quickly as I think it has value.
> 
> I think I see a fairly simple way to move the call to find_calculator back to the script app and out of BackTest.pm.
> 
> If it's alright with you, I'll tab a stab at clearing up the issues and post proposed versions for Scripts/backtest_multi.pl and GT/BackTest.pm to the list.
> 

derek


fine with me, but [there's always a but]

my looksees this am have caused me to 're-learn' lots of stuff forgotten yet again
about this pos backtest_multi.pl -- among them that the pod text lies -- the input
file needs to be one xml file -- i've yet to determine how that file might be created
to start with. the sample files is called test.xml in Scripts. note that this one
seems to expect database using Text with cusip entries for 500340 578580 840400 and
604843 none of which are provided in the sample database. [counterpart file in the
ras hack dist uses T VZ VOD and GE].

also, the trunk-head backtest_multi.pl appears broken in that it expects another input
file, but i've yet to determine if that is a broken bit or oliver (orig author) had
something else in mind. and right now i've got other fish to fry so i'm not going
to do much more on this today. [the ras hack dist does not expect this 2nd input file
and to some extent the backtest_multi.pl pod is more correct. in addition this version
does work but of course it's using a different toolkit (primarily GT::BackTest)]

by all the above i'm sort-of saying

     "feel free to have a go at fixing it but i expect it's going
      to snowball on you into a task larger than anticipated".

in addition, feel free to expand the task has you see fit -- for example how does
one 'manage' the new xml based input file(s) for backtest_multi.pl, correcting,
improving the pod text, replacing XML::Simple with XML::Lib, etc.

good luck, enjoy,

aloha


ras



>>aloha
>>
>>ras
>>
>>
>>< snip of the rest >
> 
> 
> 

Re: [GT] revised backtest.pl for evaluation and comment

[Shinaberry D. <gen...@sh...>]

Looks good to me.  Glad to see you came up with a good way to figure out from the finalized objects whether they were running on defaults in order to warn the user about it.

Derek

On Aug 24, 2012, at 21:35 , Robert A. Schmied wrote:

> gt'ers
> 
> 
> attached is an alternate to the flawed backtest.pl as derek has
> noted in this thread.
> 
> i'll commit it in a few days unless there's more commentary ...
> 
> note: i have not looked at backtest_m*.pl to see if there is any
> common code in those files that also be changed, so commentary
> regarding them is also welcome.
> 
> aloha
> 
> ras
> 
> 
> 
> Robert A. Schmied wrote:
>> Shinaberry Derek wrote:
>>> On Aug 22, 2012, at 17:47 , Robert A. Schmied wrote:
>>> 
>>> 
>>> 
>>>> Shinaberry Derek wrote:
>>>> 
>>>> 
>>>>> There are, I think, four potential problems with this revision.
>>>>> Each issue is broken out below with its own snippet of code for
>>>>> documentation.  Thoughts or oversights on my part? Derek The
>>>>> following code snippet was removed and while maybe not a major
>>>>> deal, I'm not sure it was wise to remove it. Since there is no
>>>>> default Close Strategy installed, it probably is wise to require
>>>>> a close strategy when the system name is supplied from the
>>>>> command line as there is no other source for a close strategy in
>>>>> this case. r709 Scripts/backtest.pl:263-265
>>>> 
>>>> aloha derek
>>>> 
>>>> i'm not following either your (svn) version numbers or (some of)
>>>> your line number references at least for the csname issue.
>>>> 
>>>> my research shows svn r709 to be related to the old website move to
>>>> sourceforge. recent changes directly related to Scripts/backtest.pl
>>>> are: 716  	40d 20h  	ras2010  	/trunk/Scripts/backtest.pl  	set
>>>> default timeframe to 'day', include terse cryptic sub usage that
>>>> was missing
>>>> 
>>>> 715 	135d 14h 	ras2010 	/trunk/Scripts/backtest.pl 	incorporate
>>>> command line argument --initial_value to set value of initial
>>>> amount of money for investment this value can also be set from
>>>> configuration key "backtest::initial_value".
>>>> 
>>>> 658 	1469d 21h 	joao 	/trunk/Scripts/backtest.pl 	Fix the
>>>> max-loaded-items option in scripts
>>>> 
>>>> 
>>> 
>>> Hi Robert,
>>> 
>>> By revision 709, I meant to refer to any file under the trunk as it
>>> appeared immediately following the commit for revision 709.  Revision
>>> 658 was the last commit to modify backtest.pl prior to revision 709,
>>> so revisions 658 through 714 of backtest.pl are all identical.  The
>>> tarball I originally downloaded from the website happens to
>>> correspond to revision 709, so that's why that particular revision
>>> number has popped up here.  The way I conceptualize svn is that every
>>> revision is a snapshot of the entire repository regardless of when
>>> the last commit actually modified any given file.  I hope that clears
>>> things up; sorry for the confusion.  Anyway it looks like you managed
>>> to find the code in question in spite of my attempts to obfuscate.
>>> ;-)
>>> 
>>> Cheers, Derek
>>> 
>>> 
>>>> ras
>>>> 
>>>> 
>>>> 
>>>> i do see the deathnell message requiring at least one closestrat be
>>>> part of a trading system has been removed. this was because by
>>>> default a missing closestrat will always result in the 'do nothing'
>>>> default closestrat CS:NeverClose. (iirc this is enforced by
>>>> add_position_manager).
>>>> 
>>>> this isn't obvious based on the data emitted by the app unless you
>>>> study the output results including the trade history chart.
>>>> 
>>> 
>>> I suppose we could have a philosophical debate on the usefulness of
>>> backtesting a buy and hold system, i.e. CS:NeverClose.  Especially
>>> when the backtest report will provide a comparison with buy and hold.
>>> ;-)
>>> 
>>> What you describe is true enough though.  It looks like although the
>>> code provides a routine to install a default CS, backtest.pl never
>>> avails itself of that.  The result being that with no CS installed,
>>> no CS code is ever called, which results in identical behavior to explicitly installing CS:NeverClose.  You say po-TA-toe and I say
>>> po-TAH-toe; let's call the whole thing off. :-)
>>> 
>> you are correct.
>>>> here's a working minimal command line: ras [ 266 ] %
>>>> ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
>>>> 
>>>> n.b. i've a non-standard tweaked ./backtest.pl that does a better
>>>> job describing the actual trading system being used, but it
>>>> (probably) requires, in a perl sense, other non-standard ras hacks
>>>> to the gt toolkit. contact me directly for more info.
>>>> 
>>>> here's that same example with my hack ... ras [ 270 ] %
>>>> ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG backtest.pl: warning: setting default CloseStrategy to NeverClose .
>>>> .
>>>> 
>>>> 
>>>> 
>>>>> - if ($system && ! scalar(@csname)) { -       die "You must give
>>>>> at least one --close-strategy argument !\n"; - } It looks like
>>>>> some redundant code found its way in here, but even if you fix
>>>>> the redundancy I'm not sure the code is kosher.  It appears the
>>>>> intent is to warn the user if the default Money Management rule
>>>>> is installed, but the default_money_management subroutine
>>>>> explicitly returns nothing, so testing it yields nothing. Maybe
>>>>> it works in an alternative experimental branch? r709
>>>>> Scripts/backtest.pl:385-398 foreach (@mmname) { $pf_manager->add_money_management_rule( create_standard_object(split (/\s+/, "MoneyManagement::$_"))); } + my $mmbasic_added = $pf_manager->default_money_management_rule(
>>>>> +  create_standard_object("MoneyManagement::Basic")); + warn
>>>>> "$prog_name: warning: added mm::basic to system\n" +  if
>>>>> $mmbasic_added && ( $verbose || $debug ); + $pf_manager->default_money_management_rule( create_standard_object("MoneyManagement::Basic")); + warn
>>>>> "$prog_name: warning: added mm::basic to system\n" +  if
>>>>> $mmbasic_added && ( $verbose || $debug );
>>>> 
>>>> ok these line refs match with r716 the intent is to install the
>>>> minimal moneymanager (basic) in the absence of one. i believe it
>>>> does work as intended; refer to attached example output line 1. (the -rh.out attachment from ras hacked backtest.pl includes
>>>> on-going diagnostics but gives identical results otherwise)
>>>> 
>>>> however, i didn't debug the actual code, just the results, but the
>>>> code does look a bit redundant ... added to 'todo list' thinking
>>>> out loud without any testing ::: lines 395-398 are redundant
>>> 
>>> 
>>> Let me clarify.  My nit isn't that MM:Basic won't be installed; I
>>> agree with you that it will be installed as your output example
>>> demonstrates.  What I think doesn't work as expected is the "warn if
>>> $mmbasic_added".
>> you are correct.
>>> I think $mmbasic_added will always be undef.  The way
>>> default_money_management_rule works, you can never know whether the
>>> provided MM is installed or not without knowing a priori that no
>>> other MMs have been added.  In this particular case, it might be more
>>> feasible to test whether @mmname is empty and if so, then install the
>>> default and issue the warning.
>> yep and yep. i still think this sort of user warning info is appropriate
>> for new users ...
>> yet another fix-it on the todo list
>> with this code fragment in backtest.pl (forgive the line number walking)
>>   385	foreach (@mmname)
>>   386	{
>>   387	    $pf_manager->add_money_management_rule(
>>   388		create_standard_object(split (/\s+/, "MoneyManagement::$_")));
>>   389	}
>>   390	# my $mmbasic_added = $pf_manager->default_money_management_rule(
>>   391	#  create_standard_object("MoneyManagement::Basic"));
>>   392	# #warn "$prog_name: warning: added mm::basic to system\n"
>>   393	# # if $mmbasic_added && ( $verbose || $debug );
>>   394	#    395	# warn "$prog_name: mmbasic_added is \"$mmbasic_added\"\n";
>>   396	#    397	$pf_manager->default_money_management_rule(
>>   398	       create_standard_object("MoneyManagement::Basic"));
>>   399	#warn "$prog_name: warning: added mm::basic to system\n"
>>   400	# if $mmbasic_added && ( $verbose || $debug );
>>   401	#    402	# warn "$prog_name: mmbasic_added is \"$mmbasic_added\"\n";
>> i'm getting MM::Basic automatically? with this command line:
>>     %    ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
>> and no MM::Basic with this command line:
>>     %    ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" -mm=Martingale GOOG
>> pos backtest.pl will not read a file containing a trading system
>> [i think having that feature would be a big improvement]
>> so if there's a global alias defined:
>>   Aliases::Global::EMA_OPT[]    SY:Generic \
>>     { S:Generic:CrossOverUp \
>>       { I:EMA #1 } \
>>       { I:EMA #2 } \
>>     } \
>>  | CS:Generic { S:Generic:CrossOverDown { I:EMA #3 } { I:EMA #1 } } \
>>  | MM:FixedSum 5000
>> the command line:
>>      %    ./backtest.pl --iv=100000 --broker="SelfTrade" 'EMA_OPT[ 30, 100, 80 ]' GOOG
>> results in no explicit addition of MM::Basic so i'm ready to accept
>> the above fragment as a suitable fix for this issue. this means
>> the commented out lines 390..396 and 399..402 can be removed.
>> if there is some way to parse the $pf_manager object to evaluate
>> if MM::Basic alone is there and inform the user of that (unless
>> the user requested it via -mm) that would be a good thing i think,
>> but certainly not critical ...
>>> Does that make any sense?  I hope that makes sense.  Hmm...I'm not
>>> sure that makes sense.
>>> 
>>> 
>>>>> Seems to have a similar problem to the money management code
>>>>> above in that set_order_factory has no explicit return value.
>>>>> Thus, the implicit return value will be the value of the order
>>>>> factory passed in as an argument, which in this case will always
>>>>> be undef.  So, unless I'm missing something, this is a quite
>>>>> roundabout way of saying "if ( ! defined undef )", which is
>>>>> always true by definition. r709 Scripts/backtest.pl:400-414 if
>>>>> ($ofname) { $sys_manager->set_order_factory( create_standard_object(split (/\s+/, "OrderFactory::$ofname"))); } else { -     if ( $verbose || $debug ) { +     my $of =
>>>>> $sys_manager->set_order_factory(); +     if ( ! defined $of && (
>>>>> $verbose || $debug ) ) { my $msg = join "", "$prog_name: notice:
>>>>> ", "without explicit orderfactory the implicit default is\n", "OF::MarketPrice. ", "this equates to market open price day after
>>>>> the order.", "\n"; warn "$msg"; } } The else clause was added,
>>>>> but is it really a good idea to force the portfolio to be stored?
>>>>> The description of the --store flag says nothing about a default
>>>>> if the option is unspecified. If it is realy important to store
>>>>> the portfolio, maybe it shouldn't be an option.
>>>> 
>>>> valid point ... added to 'todo list'
>>> 
>>> 
>>> Not sure whether your valid point comment applied to the order
>>> factory issue above or the else clause issue below. If it didn't
>>> apply to the order factory issue, then I don't think I saw any
>>> comments about it.
>> applies to the else clause below, without any testing i commented
>> out the else part in my svn repo backtest.pl file.
>> i must have glazed over on the orderfactory issue, you are correct
>> that it is also non-functional code.
>> but the intent was to inform the user that the default orderfactory
>> is OF::MarketPrice.
>> this trading system component is also not 'explicitly' present in the
>> listed system even though i do think it is implicit in the tool kit code
>> when no orderfactory is spec'ed in the trading system.
>>    %     ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
>>    ## Analysis of SY:TFS 50 7|MM:Basic
>>    History of the portfolio :
>>    ...
>> bottom lines
>> looks like an update of Scripts/backtest.pl is needed to remove the
>> cruft that has been accumulating. i've put that on my to-do list,
>> but there are other things there that are more important, to me anyway,
>> so please be patient. after all this code is, for the most part,
>> non-functional, so while it may waste cpu and memory, it does no
>> other harm so the update isn't deemed critical. on the other hand
>> if anyone wants to hack away and submit an improved version please
>> do so.
>> additionally, much of this code might also be found in Scripts/backtest_*.pl
>> so those files also deserve a good going over as well.
>> aloha
>> ras
>>> 
>>>> thinking out loud without any testing ::: lines 452 and 353 are not
>>                                                       453 ^^^
>>>> appropriate but might be conditioned with $debug or something
>>>> similar.
>>>> 
>>>> 
>>>> 
>>>>> r709 Scripts/backtest.pl:447-451 if ($store_file) { $analysis->{'portfolio'}->store($store_file); + } else { +
>>>>> $analysis->{'portfolio'}->store("./bt_portfolio.xml"); }
>>> 
>>> <snip>
>>> 
>>> 
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference Exclusive live event will cover all
>>> the ways today's security and threat landscape has changed and how IT
>>> managers can respond. Discussions will include endpoint security,
>>> mobile security and the latest in malware threats.
>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> 
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 
> #!/usr/bin/env perl
> #
> #!/bin/sh -- # perl, to stop looping
> # the following must be on two lines
> eval 'exec perl -S $0 ${1+"$@"}'
> if 0;
> 
> #!/usr/bin/perl -w
> 
> # Copyright 2000-2002 Raphaël Hertzog, Fabien Fulhaber
> # This file is distributed under the terms of the General Public License
> # version 2 or (at your option) any later version.
> 
> use lib '..';
> 
> use strict;
> use vars qw($db);
> 
> use GT::Prices;
> use GT::Portfolio;
> use GT::PortfolioManager;
> use GT::Calculator;
> use GT::Report;
> use GT::BackTest;
> use GT::Eval;
> use GT::Conf;
> use GT::DateTime;
> use GT::Tools qw(:conf :timeframe);
> use GT::Graphics::DataSource;
> use GT::Graphics::Driver;
> use GT::Graphics::Object;
> use GT::Graphics::Graphic;
> use GT::Graphics::Tools qw(:axis :color);
> 
> # perl prerequisites
> use GD;
> use Pod::Usage;
> use Getopt::Long;
> 
> use Data::Dumper;
> 
> =head1 ./backtest.pl [ options ] <code>
> 
> =head1 ./backtest.pl [ options ] <system_alias> <code>
> 
> =head1 ./backtest.pl [ options ] "<full_system_name>" <code>
> 
> =head2 Description
> 
> Backtest will run a backtest of a system on the indicated code.
> 
> You can either describe the system using options, give a full system
> name, or you can give a system alias. An alias is defined in the 
> configuration file with entries of the form 
> Aliases::Global::<alias_name> <full_system_name>. 
> 
> The full system name consists of a set of properties, such as trade 
> filters, close strategy, etc., together with their parameters, 
> separated by vertical bars ("|"). Multiple properties of the same 
> type can be defined, e.g., there could be a set of close strategies.
> For example,
>  System:ADX 30 | TradeFilters:Trend 2 5 | MoneyManagement:Normal 
> defines a system based on the "ADX" system, using a trend following trade
> filter "Trend", and the "Normal" money management.
> 
> The following abbreviations are supported:
> Systems = SY
> CloseStrategy = CS
> TradeFilters = TF
> MoneyManagement = MM
> OrderFactory = OF
> Signals = S
> Indicators = I
> Generic = G
> 
> Another example of a full system name is 
>  SY:TFS|CS:SY:TFS|CS:Stop:Fixed 4|MM:VAR.
> 
> =head2 Options
> 
> Backtest provide a set of options, so that you can use a combination
> of MoneyManagement, TradeFilters, OrderFactory an CloseStrategy modules.
> 
> =over 4
> 
> =item --full, --start=<date>, --end=<date>, --nb-item=<nr>
> 
> Determines the time interval over which to perform the backtest. In detail:
> 
> =over
> 
> =item --start=2001-1-10, --end=2002-11-17
> 
> The start and end dates over which to perform the backtest.
> The date needs to be in the
> format configured in ~/.gt/options and must match the timeframe selected. 
> 
> =item --nb-items=100
> 
> The number of periods to use in the analysis.
> 
> =item --full
> 
> Consider all available periods.
> 
> =back
> 
> The periods considered are relative to the selected time frame (i.e., if timeframe
> is "day", these indicate a date; if timeframe is "week", these indicate a week;
> etc.). In GT format, use "YYYY-MM-DD" or "YYYY-MM-DD hh:mm:ss" for days (the
> latter giving intraday data), "YYYY-WW" for weeks, "YYYY/MM" for months, and 
> "YYYY" for years.
> 
> The interval of periods examined is determined as follows:
> 
> =over
> 
> =item 1 if present, use --start and --end (otherwise default to last price)
> 
> =item 1 use --nb-item (from first or last, whichever has been determined), 
> if present
> 
> =item 1 if --full is present, use first or last price, whichever has not yet been determined
> 
> =item 1 otherwise, consider a two year interval.
> 
> =back
> 
> The first period determined following this procedure is chosen. If additional
> options are given, these are ignored (e.g., if --start, --end, --full are given,
> --full is ignored).
> 
> =item --timeframe=1min|5min|10min|15min|30min|hour|3hour|day|week|month|year
> 
> The timeframe can be any of the available modules in GT/DateTime.  
> 
> =item --max-loaded-items
> 
> Determines the number of periods (back from the last period) that are loaded
> for a given market from the data base. Care should be taken to ensure that
> these are consistent with the performed analysis. If not enough data is
> loaded to satisfy dependencies, for example, correct results cannot be obtained.
> This option is effective only for certain data base modules and ignored otherwise.
> 
> =item --template="backtest.mpl"
> 
> Output is generated using the indicated HTML::Mason component.
> For Example, --template="backtest.mpl"
> The template directory is defined as Template::directory in the options file.
> Each template can be predefined by including it into the options file
> For example, Template::backtest backtest.mpl
> 
> =item --html
> 
> Output is generated in html
> 
> =item --graph="filename.png"
> 
> Generate a graph of your portfolio value over the time of the backtest and
> display it in the generated html.
> 
> =item --display-trades
> 
> Display the trades with little symbols on the graph. This works well if
> trades last long enough otherwise your graph will be overwhelmed with
> unsignificant symbols.
> 
> =item --store="portfolio.xml"
> 
> Store the resulting portfolio in the indicated file.
> 
> =item --iv="<investment amount>" or --initial_value
> 
> set the investment amount for the backtest analysis -- default is 10000.
> can also be set using the config option "Backtest::initial_value" which
> will is be loaded from $HOME/.gt/options if present.
> 
> =item --broker="NoCosts"
> 
> Calculate commissions and annual account charge, if applicable, using
> GT::Brokers::<broker_name> as broker.
> 
> =item --system="<system_name>"
> 
> use the GT::Systems::<system_name> as the source of buy/sell orders.  
> 
> =item --money-management="<money_management_name>" 
> 
> use the GT::MoneyManagement::<money_management_name> as money management system.
> 
> =item --trade-filter="<filter_name>"
> 
> use the GT::TradeFilters::<filter_name> as a trade filter.  
> 
> =item --order-factory="<order_factory_name>" 
> 
> use GT::OrderFactory::<order_factory_name> as an order factory.  
> 
> =item --close-strategy="<close_strategy_name>" 
> 
> use GT::CloseStrategy::<close_strategy_name> as a close strategy.
> 
> =item --set=SETNAME
> 
> Stores the backtest results in the "backtests" directory (refer to your options file for the location of this directory) using the set name SETNAME. Use the --set option of analyze_backtest.pl to differentiate between the different backtest results in your directory.
> 
> =item --output-directory=DIRNAME
> 
> Override the "backtests" directory in the options file.
> 
> =item --verbose
> 
> for a more noisy run or debug, multiples increase the verbosity level.
> 
> =item --options=<key>=<value>
> 
> A configuration option (typically given in the options file) in the
> form of a key=value pair. For example,
> --option=DB::Text::format=0
> sets the format used to parse markets via the DB::Text module to 0.
> 
> =back
> 
> =head2 Examples
> 
> =over 4
> 
> =item
> 
> ./backtest.pl TFS 13000
> 
> =item
> 
> ./backtest.pl --full TFS 13000
> 
> =item
> 
> ./backtest.pl --close-strategy="Systems::TFS" --close-strategy="Stop::Fixed 6" --money-management="VAR" --money-management="OrderSizeLimit" --system="TFS" --broker="SelfTrade Intégral" 13000
> 
> =item
> 
> ./backtest.pl --broker="SelfTrade Int™égral" "SY:TFS|CS:SY:TFS|CS:Stop:Fixed 6|MM:VAR|MM:OrderSizeLimit" 13000
> 
> =back
> 
> =cut
> 
> 
> ( our $prog_name = $0 ) =~ s@^.*/@@;    # lets identify ourself
> 
> GT::Conf::load();
> GT::Conf::default("BT::Graphic::Grid::Color", "[220,220,220]");
> my $outputdir = GT::Conf::get("BackTest::Directory") || '';
> my $set = '';
> 
> # Manage options
> our $verbose = 0;
> our $debug = 0;
> 
> my ($full, $html, $display_trades) =
>   (0,     0,     0);
> my ($template, $graph_file, $ofname, $broker, $system, $timeframe) =
>   ('',        '',          '',      '',      '',      'day');
> my ($start, $end, $store_file) =
>   ('',     '',   '');
> my (@mmname, @tfname, @csname) =
>  ( (),      (),      () );
> 
> my ($nb_item, $max_loaded_items) = (0, -1);
> 
> my $initial_value = 0;    # alter using command line option --initial_value
>                          # or the config option Backtest::initial_value
> my @options;
> my $man = 0;
> 
> GetOptions(
>            'full!'                 => \$full,
>            'verbose+'              => \$verbose,
>            'html!'                 => \$html,
>            'template=s'            => \$template,
>            "store=s"               => \$store_file,
>            'sy|system=s'           => \$system,
>            'mm|moneymanagement|money-management=s'  => \@mmname,
>            'tf|tradefilter|trade-filter=s'          => \@tfname,
>            'of|orderfactory|order-factory=s'        => \$ofname,
>            'cs|closestrategy|close-strategy=s'      => \@csname,
>            'dt|displaytrades|display-trades!'       => \$display_trades,
>            'iv|initial_value|initial-value=s'       => \$initial_value,
>            'broker=s'              => \$broker,
>            "timeframe=s"           => \$timeframe,
>            'start=s'               => \$start,
>            'end=s'                 => \$end,
>            'graph=s'               => \$graph_file,
>            "max-loaded-items=i"    => \$max_loaded_items,
>            "option=s"              => \@options,
>            "help|man|?!"           => \$man,
>            'od|output-directory=s' => \$outputdir,
>            'set=s'                 => \$set,
>            'nb-item=i'             => \$nb_item,
>            # 'tight!' => \$tight,
>           );
> 
> if ( $#ARGV <= -1 ) {
>  usage();
>  exit 0;
> }
> 
> if ( $verbose >= 3 ) {
>    $verbose = 0;
>    ++$debug;
> }
> 
> 
> #warn "verbose is $verbose\n";
> #warn "debug is $debug\n";
> 
> 
> foreach (@options) {
>    my ($key, $value) = split (/=/, $_);
>    GT::Conf::set($key, $value);
> }
> 
> #pod2usage( -verbose => 1 ) if $man;
> pod2usage( -verbose => 2 ) if $man;
> 
> if ( $outputdir && ! -d $outputdir ) {
>  die "The directory '$outputdir' doesn't exist !\n";
> } else {
>  print "using . for output files\n" if $verbose || $debug;
> }
> 
> 
> # Create the entire framework
> my $db = create_db_object();
> my $pf_manager = GT::PortfolioManager->new;
> my $sys_manager = GT::SystemManager->new;
> my $sysname;
> if ( $initial_value == 0 ) {
>    $initial_value = GT::Conf::get('Backtest::initial_value')
>     ? GT::Conf::get('Backtest::initial_value')
>     : 10000;
> }
> 
> 
> unless ( $system || $#ARGV == 1 ) {
>    warn "You must give a system or system alias argument!\n";
>    usage();
>    exit 0;
> }
> 
> if ( $system ) {
>    $sys_manager->set_system(
>     create_standard_object(split (/\s+/, "Systems::$system")));
> } else {
> 
> #GT::Conf::conf_dump;
> 
>    my $alias = shift;
>    if (! defined($alias)) {
>        if ( $#ARGV < 0 ) {
>          usage();
>          exit 0;
>        }
>        my $msg = "You must either specify a system via the --system parameter\n"
>         . " or list an alias name on the command line.\n";
>        die "$msg";
>    }
> 
>    if ($alias !~ /\|/) {
>        $sysname = resolve_alias($alias);
>        die "Alias unknown '$alias'" if (! $sysname);
>        $sys_manager->set_alias_name($alias);
>    } else {
>        $sysname = $alias;
>    }
> 
>    if (defined($sysname) && $sysname) {
>        $sys_manager->setup_from_name($sysname);
>        $pf_manager->setup_from_name($sysname);
>    } else {
>        die "You must give either a --system argument or an alias name.";
>    }
> }
> 
> foreach (@mmname)
> {
>    $pf_manager->add_money_management_rule(
> 	create_standard_object(split (/\s+/, "MoneyManagement::$_")));
> }
> 
> if ($ofname)
> {
>    $sys_manager->set_order_factory(
> 	create_standard_object(split (/\s+/, "OrderFactory::$ofname")));
> }
> 
> foreach (@tfname)
> {
>    $sys_manager->add_trade_filter(
> 	create_standard_object(split (/\s+/, "TradeFilters::$_")));
> }
> foreach (@csname)
> {
>    $sys_manager->add_position_manager(
> 	create_standard_object(split (/\s+/, "CloseStrategy::$_")));
> }
> 
> # Prepare data
> my $code = shift;
> if (! $code) {
>    die "You must give a symbol for the simulation.\n";
> }
> 
> if ( $timeframe ) {
>    $timeframe = GT::DateTime::name_to_timeframe($timeframe);
> }
> 
> # Verify dates and adjust to timeframe, comment out if not desired
> check_dates($timeframe, $start, $end);
> 
> my ($calc, $first, $last) = find_calculator($db, $code, $timeframe, $full, $start, $end, $nb_item, $max_loaded_items);
> 
> $pf_manager->finalize;
> warn "post finalize pf-mngr == \"", $pf_manager->get_name(), "\"\n"
> if $verbose || $debug;
> 
> my $posmngt = $pf_manager->get_name();
> 
> if ( ! ( grep { m/MM::*/io } $posmngt ) && ! ( grep { m/MM::*Basic/io } $posmngt ) ) {
>    my $msg = join "", "$prog_name: notice: ",
>     "without explicit MoneyManagemener component the\n",
>     "implicit default is MM::Basic.\n",
>     "More than one MM component can be added to a trading system.\n",
>     "\n";
>    warn "$msg" if $verbose || $debug;
>    $pf_manager->default_money_management_rule(
>     create_standard_object("MoneyManagement::Basic"));
>    $pf_manager->finalize;
> }
> 
> 
> 
> $sys_manager->finalize;
> warn "post finalize sys-mngr == \"", $sys_manager->get_name(), "\"\n"
> if $verbose || $debug;
> 
> my $system = $sys_manager->get_name();
> if ( ! grep { m/OF:+/io } $system ) {
>    my $msg = join "", "$prog_name: notice: ",
>     "without explicit OrderFactory component the\n",
>     "the implicit default is OF::MarketPrice.\n",
>     "Order will be filled at open price point of next time-period.\n",
>     "\n";
>    warn "$msg" if $verbose || $debug;
> }
> 
> 
> unless ( $system =~ m/CS:/ ) {
>    my $msg = join "", "$prog_name: notice: ",
>     "without explicit CloseStrategy component the\n",
>     "the implicit default is CS::NeverClose.\n",
>     "This means once a position is taken it will not be altered during the analysis.\n";
>     "\n";
>    warn "$msg" if $verbose || $debug;
>    $sys_manager->default_position_manager(
>     create_standard_object(split (/\s+/, "CS:NeverClose")));
>    $sys_manager->finalize;
> }
> 
> 
> 
> 
> # The real work happens here
> my $analysis = backtest_single(
> $pf_manager, $sys_manager, $broker, $calc, $first, $last, $initial_value);
> 
> if ($store_file) {
>    $analysis->{'portfolio'}->store($store_file);
> #} else {
> #    $analysis->{'portfolio'}->store("./bt_portfolio.xml");
> }
> 
> 
> if ($graph_file) {
>    # create graph for backtested portfolio
>    my $graph_ds = GT::Graphics::DataSource::PortfolioEvaluation->new($calc, $analysis->{'portfolio'});
>    $graph_ds->set_selected_range($first, $last);
> 
>    # create graph for buy and hold portfolio
>    my $pf_manager2 = GT::PortfolioManager->new;
>    my $sys_manager2 = GT::SystemManager->new;
>    $pf_manager2->setup_from_name("SY:AlwaysInTheMarket | TF:LongOnly | TF:OneTrade | CS:NeverClose");
>    $sys_manager2->setup_from_name("SY:AlwaysInTheMarket | TF:LongOnly | TF:OneTrade | CS:NeverClose");
>    my $def_rule = create_standard_object("MoneyManagement::Basic");
>    $pf_manager2->default_money_management_rule($def_rule);
>    $pf_manager2->finalize;
>    $sys_manager2->finalize;
>    my $analysis2 = backtest_single(
>     $pf_manager2, $sys_manager2, $broker, $calc, $first, $last, $initial_value);
>    my $graph_ds2 = GT::Graphics::DataSource::PortfolioEvaluation->new($calc, $analysis2->{'portfolio'});
>    $graph_ds2->set_selected_range($first, $last);
> 
>    # set up graphic objects
>    my $zone = GT::Graphics::Zone->new(700, 300, 80, 40, 0, 40);
>    my $scale = GT::Graphics::Scale->new();
>    $scale->set_horizontal_linear_mapping($first, $last + 1, 0, $zone->width);
>    $scale->set_vertical_linear_mapping(union_range($graph_ds->get_value_range, $graph_ds2->get_value_range), 0, $zone->height);
>    $zone->set_default_scale($scale);
>    my $axis_h = GT::Graphics::Axis->new($scale);
>    my $axis_v = GT::Graphics::Axis->new($scale);
>    $axis_h->set_custom_big_ticks(build_axis_for_interval(union_range($graph_ds->get_value_range, $graph_ds2->get_value_range), 0, 1));
>    $axis_v->set_custom_big_ticks(build_axis_for_timeframe($calc->prices(), $YEAR, 1, 1), 1);
>    $zone->set_axis_left($axis_h);
>    $zone->set_axis_bottom($axis_v);
>    my $graphic = GT::Graphics::Graphic->new($zone);
>    my $graph = GT::Graphics::Object::Curve->new($graph_ds, $zone);
>    $graph->set_foreground_color([0, 190, 0]);
>    $graph->set_antialiased(0);
>    my $graph2 = GT::Graphics::Object::Curve->new($graph_ds2, $zone);
>    $graph2->set_foreground_color([190, 0, 0]);
>    $graph2->set_antialiased(0);
>    $graphic->add_object($graph2);
>    $graphic->add_object($graph);
>    if ($display_trades) {
> 	my $trades = GT::Graphics::Object::Trades->new($calc, $zone,
> 							$analysis->{'portfolio'},
> 							$first, $last);
> 	$graphic->add_object($trades);
>    }
> 
>    my $driver = create_standard_object("Graphics::Driver::GD");
>    my $picture = $driver->create_picture($zone);
>    $graphic->display($driver, $picture);
>    $driver->save_to($picture, $graph_file);
> }
> 
> # Display the results
> $template = GT::Conf::get('Template::backtest') if ($template eq '');
> if (defined($template) && $template ne '') {
>  my $output;
> 
>  my $use = 'use HTML::Mason;use File::Spec;use Cwd;';
>  eval $use;
>  die($@) if($@);
> 
>  my $root = GT::Conf::get('Template::directory');
>  $root = File::Spec->rel2abs( cwd() ) if (!defined($root));
>  my $interp = HTML::Mason::Interp->new( comp_root => $root,
> 					 out_method => \$output
> 				       );
>  $template='/' . $template unless ($template =~ /\\|\//);
>  $interp->exec($template, analysis => $analysis, 
> 	  sys_manager => $sys_manager, pf_manager => $pf_manager, 
> 	  verbose => $verbose);
>  print $output;
> }
> elsif ($html)
> {
>    print "Analysis of " . $sys_manager->get_name . "|" .
> 	$pf_manager->get_name;
>    if ($graph_file) {
>        print "<h2>Value of portfolio (green) vs. buy and hold (red)</h2>\n";
>    	print "<img src=\"$graph_file\" alt=\"Portfolio evaluation\" \\>\n";
>    }
>    print "<table border='1' bgcolor='#EEEEEE'><tr><td>";
>    print "<pre>";
>    print "## Global analysis (full portfolio always invested)\n";
>    GT::Report::PortfolioAnalysis($analysis->{'real'}, $verbose);
>    print "</pre></td></tr></table>";
>    GT::Report::PortfolioHTML($analysis->{'portfolio'}, $verbose);
> }
> else
> {
>    print "## Analysis of " . $sys_manager->get_name . "|" .
> 	$pf_manager->get_name . "\n";
>    GT::Report::Portfolio($analysis->{'portfolio'}, $verbose);
>    print "## Global analysis (full portfolio always invested)\n";
>    GT::Report::PortfolioAnalysis($analysis->{'real'}, $verbose);
>    print "\n";
> }
> 
> $db->disconnect;
> 
> if ( $set ) {
>  # Store intermediate result
>  my $bkt_spool = GT::BackTest::Spool->new($outputdir);
>  my $stats = [ $analysis->{'real'}{'std_performance'},
> 		$analysis->{'real'}{'performance'},
> 		$analysis->{'real'}{'max_draw_down'},
> 		$analysis->{'real'}{'std_buyandhold'},
> 		$analysis->{'real'}{'buyandhold'}
> 	      ];
> 
>  $bkt_spool->update_index();
>  $bkt_spool->add_alias_name($sys_manager->get_name, $sys_manager->alias_name);
>  $bkt_spool->add_results($sys_manager->get_name, $code, $stats,
> 			  $analysis->{'portfolio'}, $set);
>  $bkt_spool->sync();
> }
> 
> 
> sub usage {
> print <<"eof";
> ./backtest.pl [ options ] <code>
> ./backtest.pl [ options ] <system_alias> <code>
> ./backtest.pl [ options ] "<full_system_name>" <code>
>  options: -full -verbose -html -dt
>  opt w arg: -te fn, -mm s, -tf s, -of s, -cs s, -b s, -sy s, -sto fn, -gr fn,
>             -mli n, -od dn, -set s, -iv n
>  typical opts: -start date, -end date, -timeframe tfname
> eof
> 
> }
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Re: [GT] detail findings on GT/BackTest.pm was Re: Problems With Rev 714 GT/BackTest.pm - Fixing The UNIX Epoch Problem

[Shinaberry D. <gen...@sh...>]

Hi Robert,

I did some spelunking through the -devel list archives and the svn repository to see what I could discover about your description of the intended design goal of GT/BackTest.pm not depending on GT/Tools.pm, script apps needing to invoke find_calculator.  I think four of the six issues I raised involve changes made to move GT/BackTest.pm and Scripts/backtest_multi.pl towards that goal.

This is more than a bit long winded.  I hope you'll forgive the length, but there was a lot of ground for me to cover.  There are some interspersed comments and then much longer commentary towards the end.

Cheers,
Derek

On Aug 22, 2012, at 21:29 , Robert A. Schmied wrote:

> Shinaberry Derek wrote:
>> None of the code changes seem to have anything to do with the UNIX
>> epoch problem.  I don't know whether the comment was a copy and paste
>> error or whether the wrong version of GT/BackTest.pm was committed.
> 
> entire message being looked into in detail. i'll post findings in due course.
> 
> as far as the GT/BackTest.pm commit, it does appear was a near complete drop-in
> from my toolkit, and it was most likely committed to support the epoch date
> fix for backtest.pl primarily. backtest_multi.pl and backtest_many.pl are
> secondary or tertiary apps, but it would be good to keep them operational
> to the extent that they are operational. GT/BackTest.pm itself should not
> involve GT/Tools.pm directly.
> 
> 
> GT/Tools.pm declares/defines method find_calculator. there is very likely
> a discussion of this on the -devel archive years back. method find_calculator
> would only ever be invoked in a script app like backtest.pl or graphic.pl.
> 
> as i evaluate find_calculator, it fails to return the prices queue
> typically $q for which there are many methods that are useful, etc.

True it doesn't directly return the prices queue, but they are indirectly and easily accessible via the calculator object $calc->prices->whatever.

So instead of something like

($calc, $first, $last, $prices) = calculator(...),

you could just do something like

($calc, $first, $last) = find_calculator(...),
$prices = $calc->prices

Doing something like that would make for less worries about differences between find_calculator and calculator unless there are more differences between your custom version calculator and the standard find_calculator.

> so i have a version of it (that is of 'find_calculator') named 'calculator'
> that does return the prices queue. there may (or may not) be a
> discussion of this on the -devel archive, but the trunk-head version of
> GT/Tools.pm should *not* (yet) include method 'calculator' and it should
> *not be used* in any of the trunk-head script apps (at least that was the intent).
> 
> find_calculator is never-the-less useful and trunk-head script apps should
> make use of it where prudent and proper. the effort to complete this upgrade
> is apparently incomplete and bug ridden ...
> 
> it is good that there is another set of detail oriented eyes on this ...
> 
> 
> aloha
> 
> ras
> 
> 
> < complete snip >
> 


On Aug 23, 2012, at 01:42 , Robert A. Schmied wrote:

> Shinaberry Derek wrote:
>> None of the code changes seem to have anything to do with the UNIX epoch problem.  I don't know whether the comment was a copy and paste error or whether the wrong version of GT/BackTest.pm was committed.
>> 
>> Assuming that the intended version of BackTest.pm was, in fact, the version committed, I see the following potential issues:
>> 
>>        A significant number of changes, mostly to the backtest_mutil subroutine, appear to have come from some hacking that was perhaps aimed at eliminating
>>        the need to call find_calculator for all the codes passed into the routine.  It looks like these changes were only half baked and their inclusion has
>>        broken backtest_multi.  I've included a unified diff output where the FROM file is a theorhetically unbroken form of the r714 revision of my own
>> 	creation and the TO file is the actual r714 revision in order to highlight the problems.  I've provided commentary for each of the change blocks to
>> 	clarify what I think is a problem.
>> 
>>        @@ -152,7 +151,9 @@
>>                This is the one change not related to backtest_multi that I think is a problem.  The removal of the " - 1" changes the result from a
>>                percentage to a ratio, e.g. from 0.05 to 1.05.  It is unclear to me how this might ripple through the reporting code since this value is
>>                loaded into the analysis hash under the 'real' key and the analysis hash is returned from the backtest subroutine.
>>     # Buy & hold
>>     my $buyhold = 0;
>>     $buyhold = $calc->prices->at($last)->[$LAST] /
>> -      $calc->prices->at($first)->[$LAST] - 1 if ( $calc->prices->at($first)->[$LAST] != 0);
>> +#      $calc->prices->at($first)->[$LAST] - 1
>> +      $calc->prices->at($first)->[$LAST]
>> +       if ( $calc->prices->at($first)->[$LAST] != 0);
> 
> oops -- good catch -- i must have thought $calc->prices->at($first)->[$LAST] - 1
> yielded a time frame index.
> 
>> 
>>        @@ -442,10 +457,6 @@
>>                It's not clear to me why this code was removed as I believe that the condition that the warning note refers to still applies.
>> @@ -442,10 +457,6 @@
>> #      $re->{'buyandhold_max_draw_down'} = $buyandhold_maxdd;
>> #      $th->{'buyandhold_max_draw_down'} = $buyandhold_maxdd;
>> #    }
>> -    ### NOTE: Need to assign something to these to avoid undef value in output
>> -    $re->{'buyandhold_max_draw_down'} = 0;
>> -    $th->{'buyandhold_max_draw_down'} = 0;
>> -
>> 
>>     # Hack to remove code reference in portfolio
>>     delete $p->{'date2int'};
> 
> dunno will look at in more detail ...
> 
> i'm not getting undef in output is there a command line test case that can be
> used to fix a baseline? i typically fixate on sample_data/13000.txt as the market
> file since that one is 'generally' available.

Running Scripts/backtest_multi.pl in the debugger against the 709 revision snapshot (because that was convenient for me) twice, once with the two hash keys set to zero and once with them taken out as in r714, yielded the following results.  The error about a use of an uninitialized value at BackTest.pm:347 is related to the fact that there is no theoretical analysis done in backtest_multi.  I hacked in setting a value to zero in my own code to avoid the error showing up all the time, which is essentially what the removed code was doing to.

svn-709/Scripts>cat ~/.gt/options
DB::module Text
DB::text::directory svn-709/Scripts/t/data/
DB::text::header_lines 0
DB::text::fields::open     0
DB::text::fields::high     1
DB::text::fields::low      2
DB::text::fields::close    3
DB::text::fields::volume   4
DB::text::fields::datetime 5

Analysis::ReferenceTimeFrame year

svn-709/Scripts>perl -d backtest_multi.pl <(echo alcatel) <(echo "SY:TFS 50 10|CS:SY:TFS")

LONG-CODE:  alcatel
LONG-FIRST: 1921
LONG-LAST:  2421
LONG-FIRST: 2000-10-06 00:00:00
LONG-LAST:  2002-10-01 00:00:00
Use of uninitialized value in addition (+) at ../GT/BackTest.pm line 347.
 at ../GT/BackTest.pm line 347
	GT::BackTest::backtest_multi('GT::DB::Text=HASH(0xd833bc)', 'GT::PortfolioManager=HASH(0x865c4c)', 'ARRAY(0xce4334)', 'ARRAY(0xce4364)', 'ARRAY(0xce46dc)', 70, 0, '', '', ...) called at backtest_multi.pl line 248
History of the portfolio :
--------------------------
.
.
.
## Global analysis (each position is 10000, value of portfolio)
Analysis of the portfolio (2000-10-06 00:00:00 / 2002-10-01 00:00:00) :
-----------------------------------------------------
Performance :  22.8% ( 10.8%)	  Buy & Hold :   0.0% (  0.0%) () => by year
MaxDrawDown :       25.1%  B&H MaxDrawDown :     0.0%
.
.
.
Debugged program terminated.  Use q to quit or R to restart,

svn-709/Scripts>perl -d backtest_multi.pl <(echo alcatel) <(echo "SY:TFS 50 10|CS:SY:TFS")                         

LONG-CODE:  alcatel
LONG-FIRST: 1921
LONG-LAST:  2421
LONG-FIRST: 2000-10-06 00:00:00
LONG-LAST:  2002-10-01 00:00:00
Use of uninitialized value in addition (+) at ../GT/BackTest.pm line 347.
 at ../GT/BackTest.pm line 347
	GT::BackTest::backtest_multi('GT::DB::Text=HASH(0xd83374)', 'GT::PortfolioManager=HASH(0x865c4c)', 'ARRAY(0xce42ec)', 'ARRAY(0xce431c)', 'ARRAY(0xce4694)', 70, 0, '', '', ...) called at backtest_multi.pl line 248
History of the portfolio :
--------------------------
.
.
.
## Global analysis (each position is 10000, value of portfolio)
Analysis of the portfolio (2000-10-06 00:00:00 / 2002-10-01 00:00:00) :
-----------------------------------------------------
Use of uninitialized value in pattern match (m//) at ../GT/Report.pm line 278.
 at ../GT/Report.pm line 278
	GT::Report::safe_sprintf('%.1f%%', undef) called at ../GT/Report.pm line 312
	GT::Report::PortfolioAnalysis('HASH(0xeb3cfc)', 1) called at backtest_multi.pl line 253
Use of uninitialized value in pattern match (m//) at ../GT/Report.pm line 278.
 at ../GT/Report.pm line 278
	GT::Report::safe_sprintf('%.1f%%', undef) called at ../GT/Report.pm line 312
	GT::Report::PortfolioAnalysis('HASH(0xeb3cfc)', 1) called at backtest_multi.pl line 253
Use of uninitialized value in sprintf at ../GT/Report.pm line 281.
 at ../GT/Report.pm line 281
	GT::Report::safe_sprintf('%.1f%%', undef) called at ../GT/Report.pm line 312
	GT::Report::PortfolioAnalysis('HASH(0xeb3cfc)', 1) called at backtest_multi.pl line 253
Performance :  22.8% ( 10.8%)	  Buy & Hold :   0.0% (  0.0%) () => by year
MaxDrawDown :       25.1%  B&H MaxDrawDown :     0.0%
.
.
.
Debugged program terminated.  Use q to quit or R to restart,

> 
>> 
>>        @@ -12,7 +12,8 @@
>>        @@ -491,9 +502,47 @@
>>                The subroutine setup_backtest is unused at this point and appears to me that perhaps the changes were headed towards its use, but it doesn't
>>                seem ready yet and should probably be removed or, at a minimum, heavily documented as to its experimental status.
>> @@ -491,9 +502,47 @@
>>     return $managers;
>> }
>> 
>> -=pod
>> +=item C<< setup_backtest($code, $timeframe, $full, $start, $end) >>
>> 
>> -=back
>> +Setup the system backtest: Returns $calc (the calculator), as well as
>> +$first and $last (indices used by the calculator).
> 
> i believe setup_backtest is an artifact from the thomas weigert effort
> from backtest_opt.pl (likely my name for it) also maybe called backtest_optimal.pl
> where in an effort was made to vary parameters of the trading system in
> an attempt to determine optimum values.
> 
> the method was installed into gt::backtest.pm so that it was generally
> available if and when it might be useful.
> 
> refer -devel archive "[GT] System optimization" dated: 08/19/05 20:30
> 
It doesn't appear that this effort ever made it into svn before now.  There appears to be nothing in the svn history about a script named backtest_opt.  I can't see the attachments in the archive so I don't know what the script and other patch files contained.  At any rate, this setup_backtest looks remarkably like find_calculator.  Too much alike to be a coincidence in my mind.
>> 
>>        @@ -21,8 +22,6 @@
>>                This use declaration is where visibility to GT::Tools::find_calculator comes from.  Assuming that I am correct about the need to restore
>>                the other code which uses find_calculator, then this should be restored as well.
>> @@ -21,8 +22,6 @@
>> use GT::DateTime;
>> ###l4p  use Log::Log4perl qw(:easy);
>> use GT::Indicators::MaxDrawDown;
>> -use GT::Tools qw(:timeframe);
>> -
> 
> is there an actual problem that you can demonstrate or are we conducting a code review?
> 
It's only a problem if find_calculator remains but the use statement doesn't.
> 
> it seems to me GT/BackTest.pm is only ever used in connection with a gt script app, which
> will have already established a calculator object (e.g. $calc). i don't think there is
> any [good] reason for another calculator object being created by this module directly ...
> 
I think backtest_multi.pl may be the bastard child here.  As it existed when I started using GT, I think it was the only script app to NOT directly call find_calculator, but relied upon BackTest.pm to invoke it.  So, in the case of backtest_multi, it never obtains a calculator itself, so maybe that is a [good] reason to allow BackTest.pm to do it?  It used to obtain the list of calculator objects up until a revision that "Optimize[d] BackTest.pm to leverage find_calculator in a more expedient manner [than backtest_multi.pl could]."
>> 
>>        @@ -217,10 +218,10 @@
>>                The parameter list for backtest_multi has completely changed and this doesn't match the code in backtest_multi.pl that calls it anymore.
>>                In addition to a number of parameters being dropped, $code_ref is changed to $calc_ref and the corresponding list $codes is changed to @calcs.
>>                This is what drove my assumption that most of the changes are targeted at elimiinating the need to use find_calculator to get a calc
>>                object corresponding to a code symbol.
>> @@ -217,10 +218,10 @@
>> 
>> 
>> sub backtest_multi {
>> -    my ($db, $pf_manager, $sys_manager_ref, $broker_ref, $code_ref, $timeframe, $full, $start, $end, $nb_item, $max_loaded_items, $init) = @_;
>> +    my ($pf_manager, $sys_manager_ref, $broker_ref, $calc_ref, $start, $end, $full, $init) = @_;
>>     my @sysmanager = @{$sys_manager_ref};
>>     my @brokers = @{$broker_ref};
>> -    my @codes = @{$code_ref};
>> +    my @calc = @{$calc_ref};
>>     $init = 10000 unless ( defined($init) );
>> 
>>     # Create an empty portfolio object and make the manager use it
> 
> this may be a screwup on my part ...
> 
> with
>   market_file === AAPL
>   system_file === crossing_ema
>       with .gt/options file containing:
>           Aliases::Global::crossing_ema           SY:G \
>             { S:G:CrossOverUp   {I:EMA 30} {I:EMA 100} } \
>             { S:G:CrossOverDown {I:EMA 30} {I:EMA 100} } \
>           | CS:OppositeSignal
> 
> 
>   ras [ 484 ] %     ( pushd /usr/local/src/genius_trader/svn_repo/Scripts; \
>     ./backtest_multi.pl --start=2010-08-01  --end=today  \
>       --output-directory /var/tmp \
>       --broker=NoCosts \
>       ../../Scripts/market_file  \
>       ../../Scripts/js_sys_bt_mlt ; \
>     popd ; )
> 
>   Not an ARRAY reference at ../GT/BackTest.pm line 222, <> line 164.
> 
> consider ./backtest_multi.pl 'not ready for prime time'
> 
> 
> but for whatitisworth (not much frankly) with same conditions
>    ras [ 487 ] %       ( pushd /usr/local/src/genius_trader/svn_repo/Scripts; \
>     ./backtest_many.pl --start=2010-08-01  --end=today  \
>       --output-directory /var/tmp \
>       --broker=NoCosts \
>       ../../Scripts/market_file  \
>       ../../Scripts/js_sys_bt_mlt ; \
>    popd ; )
>    ##
>    ## Analysis of code AAPL using System Spec
> 
>    SY:G { S:G:CrossOverUp {I:EMA 30} {I:EMA 100} }
>     { S:G:CrossOverDown {I:EMA 30} {I:EMA 100} } | CS:OppositeSignal
> 
>    ## Global analysis (full portfolio invested)
>    Analysis of the portfolio (2010-08-02 / 2012-08-22)
>    -----------------------------------------------------
>    Performance :  71.5% ( 30.9%)     Buy & Hold : 255.4% ( 88.3%) () => by year
>    MaxDrawDown :        7.3%  B&H MaxDrawDown :    81.8%
>    Best performance :  71.5%  Worst performance :  -7.3%
>    Net gain :        7150.25  Gross gain :       7150.25
> 
>    Trades statistics
>    Number of trades :      2  Trades/Year :         0.97
>    Number of gains :       1  Number of losses :       1  Win. ratio :   50.0%
>    Max consec. win :       1  Max consec. loss :       1  Expectancy :    0.41
>    Average gain :     89.20%  Average loss :      -7.43%  Avg. perf  :  30.96%
>    Biggest gain :     89.20%  Biggest loss :      -7.43%  Profit fac :   12.01
>    Sum of gains :    7883.75  Sum of losses :    -733.50  Risk of ruin : 32.4%
> 
> does still appear functional
> 
> 
>> 
>>        @@ -245,42 +246,56 @@
>>                There are a lot of small sub-blocks here, but I believe the overarching theme is getting rid of the call to find_calculator.  I'm not sure
>>                where the intended destination for the responsibility of getting calculators is, but it seems to be upstream from backtest_multi, so it is
>>                probably in backtest_multi.pl.  The trouble with that (and this) is that a chunk of code that is already in find_calculator is duplicated
>>                here (and again in the, as yet unused, subroutine setup_backtest) in order to figure out the ranges for each code in order to determine
>>                the largest range, which is then the one used for the overall backtesting.  At any rate, since backtest_multi doesn't provide a list of calc
>>                objects and doesn't provide an argument list when calling backtest_multi that matches this vesion of the code, it simply can't work
>>                as intended.
> 
> and this too (may be a screwup)
> 
> frankly i don't use the backtest*.pl apps much, and backtest_multi.pl least of all.
> 
> it just seemed 'reasonable' to move the (potentially common) stuff into GT/BackTest.pm
> that really belongs there and remove it from the script apps to in an effort to
> eliminate duplication and the need for duplicate maintenance ...
> 
> i may have not completed the task, botched it completely or just made a muddle of it.
> 
> probably a bit (well more like a double long) of all three
> 
> 
> 
I appreciate that if you aren't an active user of really any of the backtest script apps least of all the backtest_multi.pl script app, then you really don't have a lot of motivation towards working on them except to, as you said, "keep them operational to the extent that they are operational."  In fact, you mentioned that you weren't sure that backtest_multi.pl and GT::BackTest::backtest_multi were ever really ready for prime time.

Well, let me assure you that from the r709 snapshot that I started from, both worked remarkably well right out of the box.  Virtually all of the changes I've made to either file have only been to add some additional functionality that was useful to me.  And as far as backtest_multi.pl's usefulness, it was virtually the only script app I used until I started pondering how manage_portfolio.pl might work for my reporting desires.  So, different strokes for different folks, but rest assured that these have enormous value for me.  Thus my concern about their stability in the trunk head.

My search of -devel didn't turn up anything directly on point, but poking around to see what revisions modified GT::BackTest::backtest_multi with regards to it calling find_calculator and then looking on the -devel archive for messages related to those revisions turned up a few hits.  I still didn't see anything directly on point with find_calculator, but it is entirely possible, neigh likely, that it is there somewhere and I just didn't find it.  I noticed a lot of messages I clicked on resulted in a "message not found" page.  Usually one of the messages in the thread worked and I could view the whole thread from there.  Not sure what's up with that; a known problem from importing from a previous list manager or potential data corruption of the list archive?  I'm sure I don't know.

At any rate, the discussion that I did find seemed to center on whether it was better to force the Scripts level script apps to deal with opening, connecting to, and ultimately disconnecting from the database.  It seemed like there was a point of view that it caused confusion for users and that it might be better to bury it deeper in the toolkit where users wouldn't have to worry about it.

I looked through the history of GT/Tools.pm, GT/BackTest.pm, and Scripts/backtest_multi.pl with regard to how they used find_calculator.  Here is a summary of my discoveries:

@ The initial versions of backtest_multi.pl and BackTest.pm interfaced using a list of calculators.  The db object creation and disconnection was handled by backtest_multi.pl.

@The sub find_calculator was introduced to Tools.pm in revision 594.  In its initial incarnation it handled the db object creation and disconnection.

@In revision 599 Scripts/backtest_multi.pl adopted the use of find_calculator, but the interface to BackTest.pm remained a list of calculators.  By adopting find_calculator, it gave up the responsibility of creating the db object and disconnecting from it.  The commit log only mentions making options for periods consistent across all the script apps.

@Revisions 613/614 modified backtest_multi.pl and BackTest.pm so that they interfaced using a list of symbol codes instead of calculators.  The management of the db remained with find_calculator.  The commit log comment mentions "Optimiz[ing] BackTest.pm to leverage find_calculator in a more expedient manner."

@Revisions 629/630 changed all three files so that the db object was managed once again by backtest_multi.pl.  This changed the interface between backtest_multi.pl and BackTest.pm to passing the db object and the list of symbol codes.  The the interface between BackTest.pm and find_calculator changed to pass in the db object.  The commit log comment refers to a list discussion and summarizes the usage now required for db access.
> 
> so the bottom lines are:
> 
>  @ reinstate the original $buyhold change percentage calculation
> 
>  @ do not use script app ./backtest_multi.pl *it is* 'not ready for prime time'.
>    probably never really was, but that's another story.
> 
>  @ GT/BackTest.pm sub backtest_multi *is* 'not ready for prime time'.
> 
>  @ without more specifics i'm unable to duplicate undefs that might result
>    from $*->{'buyandhold_max_draw_down'} objects no longer being initialized.
> 
>    but if you want to evaluate reinstating them the method (line 469) must
>    be changed to
>       $indicator_maxdd->find_calculator
>    not the commented out $indicator_maxdd->calculator, which is not available
>    in trunk-head.
> 
>    and the calls vars $calc $last will need to be valid within this scope.
> 
>  @ don't see there's a perl compile/runtime problem caused by the removal of
>        use GT::Tools qw(:timeframe);
>    unless it is part of the failure of ./backtest_multi.pl ...
> 
>  @ finally there may be a good reason to call method find_calculator from
>    GT/BackTest.pm, but i just haven't figured that out yet.
> 
> 
So, my conclusions...

I'm not sure whether demanding that BackTest.pm NOT use Tools.pm is a worthy design goal or not.  I can say that backtest_multi.pl is a bit of the odd one out and that oddness is what drives BackTest.pm to use Tools.pm.

I'd like to restore the trunk head to a functioning backtest_multi.pl, BackTest.pm combination fairly quickly as I think it has value.

I think I see a fairly simple way to move the call to find_calculator back to the script app and out of BackTest.pm.

If it's alright with you, I'll tab a stab at clearing up the issues and post proposed versions for Scripts/backtest_multi.pl and GT/BackTest.pm to the list.
> 
> aloha
> 
> ras
> 
> 
> < snip of the rest >
> 

[GT] Followup: Re: detail findings on GT/BackTest.pm was Re: Problems With Rev 714 GT/BackTest.pm - Fixing The UNIX Epoch Problem

[Robert A. S. <ra...@ac...>]

gt'ers

followup on GT/BackTest.pm issues -- these comments pertain to the
sub backtest_multi() function in GT/BackTest.pm. this function is
for support of app script *backtest_multi.pl* which is, at least on
my system non-operational. [that is both the one in my trunk-head
repo as well as the one in my production-develop partition of gt.]

the non-functionality appears to be from different causes and i'm
investigating but on a fairly low priority basis, the scripts
backtest_m*.pl are not high priority use items for my gt use model.

if anyone undertakes the challenge to fix sub GT::BackTest::backtest_multi()
and you find fixes are also needed to *backtest_multi.pl* you might
also consider replacing the perl pre-requisite XML::Simple in
*backtest_multi.pl*  with XML::LibXML if at all possible. according
to my notes XML::Simple is only used by *backtest_multi.pl*, where
xml is used elsewhere it is supported via XML::LibXML.


aloha

ras



Robert A. Schmied wrote:
> Shinaberry Derek wrote:
> 
>>None of the code changes seem to have anything to do with the UNIX epoch problem.  I don't know whether the comment was a copy and paste error or whether the wrong version of GT/BackTest.pm was committed.
>>
>>Assuming that the intended version of BackTest.pm was, in fact, the version committed, I see the following potential issues:
>>
>>        A significant number of changes, mostly to the backtest_mutil subroutine, appear to have come from some hacking that was perhaps aimed at eliminating
>>        the need to call find_calculator for all the codes passed into the routine.  It looks like these changes were only half baked and their inclusion has
>>        broken backtest_multi.  I've included a unified diff output where the FROM file is a theorhetically unbroken form of the r714 revision of my own
>>	creation and the TO file is the actual r714 revision in order to highlight the problems.  I've provided commentary for each of the change blocks to
>>	clarify what I think is a problem.
>>
>>        @@ -152,7 +151,9 @@
>>                This is the one change not related to backtest_multi that I think is a problem.  The removal of the " - 1" changes the result from a
>>                percentage to a ratio, e.g. from 0.05 to 1.05.  It is unclear to me how this might ripple through the reporting code since this value is
>>                loaded into the analysis hash under the 'real' key and the analysis hash is returned from the backtest subroutine.
>>     # Buy & hold
>>     my $buyhold = 0;
>>     $buyhold = $calc->prices->at($last)->[$LAST] /
>>-      $calc->prices->at($first)->[$LAST] - 1 if ( $calc->prices->at($first)->[$LAST] != 0);
>>+#      $calc->prices->at($first)->[$LAST] - 1
>>+      $calc->prices->at($first)->[$LAST]
>>+       if ( $calc->prices->at($first)->[$LAST] != 0);
> 
> 
> oops -- good catch -- i must have thought $calc->prices->at($first)->[$LAST] - 1
> yielded a time frame index.
> 
> 
>>        @@ -442,10 +457,6 @@
>>                It's not clear to me why this code was removed as I believe that the condition that the warning note refers to still applies.
>>@@ -442,10 +457,6 @@
>> #      $re->{'buyandhold_max_draw_down'} = $buyandhold_maxdd;
>> #      $th->{'buyandhold_max_draw_down'} = $buyandhold_maxdd;
>> #    }
>>-    ### NOTE: Need to assign something to these to avoid undef value in output
>>-    $re->{'buyandhold_max_draw_down'} = 0;
>>-    $th->{'buyandhold_max_draw_down'} = 0;
>>-
>>
>>     # Hack to remove code reference in portfolio
>>     delete $p->{'date2int'};
> 
> 
> dunno will look at in more detail ...
> 
> i'm not getting undef in output is there a command line test case that can be
> used to fix a baseline? i typically fixate on sample_data/13000.txt as the market
> file since that one is 'generally' available.
> 
> 
>>        @@ -12,7 +12,8 @@
>>        @@ -491,9 +502,47 @@
>>                The subroutine setup_backtest is unused at this point and appears to me that perhaps the changes were headed towards its use, but it doesn't
>>                seem ready yet and should probably be removed or, at a minimum, heavily documented as to its experimental status.
>>@@ -491,9 +502,47 @@
>>     return $managers;
>> }
>>
>>-=pod
>>+=item C<< setup_backtest($code, $timeframe, $full, $start, $end) >>
>>
>>-=back
>>+Setup the system backtest: Returns $calc (the calculator), as well as
>>+$first and $last (indices used by the calculator).
> 
> 
> i believe setup_backtest is an artifact from the thomas weigert effort
> from backtest_opt.pl (likely my name for it) also maybe called backtest_optimal.pl
> where in an effort was made to vary parameters of the trading system in
> an attempt to determine optimum values.
> 
> the method was installed into gt::backtest.pm so that it was generally
> available if and when it might be useful.
> 
> refer -devel archive "[GT] System optimization" dated: 08/19/05 20:30
> 
> 
>>        @@ -21,8 +22,6 @@
>>                This use declaration is where visibility to GT::Tools::find_calculator comes from.  Assuming that I am correct about the need to restore
>>                the other code which uses find_calculator, then this should be restored as well.
>>@@ -21,8 +22,6 @@
>> use GT::DateTime;
>> ###l4p  use Log::Log4perl qw(:easy);
>> use GT::Indicators::MaxDrawDown;
>>-use GT::Tools qw(:timeframe);
>>-
> 
> 
> is there an actual problem that you can demonstrate or are we conducting a code review?
> 
> it seems to me GT/BackTest.pm is only ever used in connection with a gt script app, which
> will have already established a calculator object (e.g. $calc). i don't think there is
> any [good] reason for another calculator object being created by this module directly ...
> 
> 
>>        @@ -217,10 +218,10 @@
>>                The parameter list for backtest_multi has completely changed and this doesn't match the code in backtest_multi.pl that calls it anymore.
>>                In addition to a number of parameters being dropped, $code_ref is changed to $calc_ref and the corresponding list $codes is changed to @calcs.
>>                This is what drove my assumption that most of the changes are targeted at elimiinating the need to use find_calculator to get a calc
>>                object corresponding to a code symbol.
>>@@ -217,10 +218,10 @@
>>
>>
>> sub backtest_multi {
>>-    my ($db, $pf_manager, $sys_manager_ref, $broker_ref, $code_ref, $timeframe, $full, $start, $end, $nb_item, $max_loaded_items, $init) = @_;
>>+    my ($pf_manager, $sys_manager_ref, $broker_ref, $calc_ref, $start, $end, $full, $init) = @_;
>>     my @sysmanager = @{$sys_manager_ref};
>>     my @brokers = @{$broker_ref};
>>-    my @codes = @{$code_ref};
>>+    my @calc = @{$calc_ref};
>>     $init = 10000 unless ( defined($init) );
>>
>>     # Create an empty portfolio object and make the manager use it
> 
> 
> this may be a screwup on my part ...
> 
> with
>    market_file === AAPL
>    system_file === crossing_ema
>        with .gt/options file containing:
>            Aliases::Global::crossing_ema           SY:G \
>              { S:G:CrossOverUp   {I:EMA 30} {I:EMA 100} } \
>              { S:G:CrossOverDown {I:EMA 30} {I:EMA 100} } \
>            | CS:OppositeSignal
> 
> 
>    ras [ 484 ] %     ( pushd /usr/local/src/genius_trader/svn_repo/Scripts; \
>      ./backtest_multi.pl --start=2010-08-01  --end=today  \
>        --output-directory /var/tmp \
>        --broker=NoCosts \
>        ../../Scripts/market_file  \
>        ../../Scripts/js_sys_bt_mlt ; \
>      popd ; )
> 
>    Not an ARRAY reference at ../GT/BackTest.pm line 222, <> line 164.
> 
> consider ./backtest_multi.pl 'not ready for prime time'
> 
> 
> but for whatitisworth (not much frankly) with same conditions
>     ras [ 487 ] %       ( pushd /usr/local/src/genius_trader/svn_repo/Scripts; \
>      ./backtest_many.pl --start=2010-08-01  --end=today  \
>        --output-directory /var/tmp \
>        --broker=NoCosts \
>        ../../Scripts/market_file  \
>        ../../Scripts/js_sys_bt_mlt ; \
>     popd ; )
>     ##
>     ## Analysis of code AAPL using System Spec
> 
>     SY:G { S:G:CrossOverUp {I:EMA 30} {I:EMA 100} }
>      { S:G:CrossOverDown {I:EMA 30} {I:EMA 100} } | CS:OppositeSignal
> 
>     ## Global analysis (full portfolio invested)
>     Analysis of the portfolio (2010-08-02 / 2012-08-22)
>     -----------------------------------------------------
>     Performance :  71.5% ( 30.9%)     Buy & Hold : 255.4% ( 88.3%) () => by year
>     MaxDrawDown :        7.3%  B&H MaxDrawDown :    81.8%
>     Best performance :  71.5%  Worst performance :  -7.3%
>     Net gain :        7150.25  Gross gain :       7150.25
> 
>     Trades statistics
>     Number of trades :      2  Trades/Year :         0.97
>     Number of gains :       1  Number of losses :       1  Win. ratio :   50.0%
>     Max consec. win :       1  Max consec. loss :       1  Expectancy :    0.41
>     Average gain :     89.20%  Average loss :      -7.43%  Avg. perf  :  30.96%
>     Biggest gain :     89.20%  Biggest loss :      -7.43%  Profit fac :   12.01
>     Sum of gains :    7883.75  Sum of losses :    -733.50  Risk of ruin : 32.4%
> 
> does still appear functional
> 
>  
> 
>>        @@ -245,42 +246,56 @@
>>                There are a lot of small sub-blocks here, but I believe the overarching theme is getting rid of the call to find_calculator.  I'm not sure
>>                where the intended destination for the responsibility of getting calculators is, but it seems to be upstream from backtest_multi, so it is
>>                probably in backtest_multi.pl.  The trouble with that (and this) is that a chunk of code that is already in find_calculator is duplicated
>>                here (and again in the, as yet unused, subroutine setup_backtest) in order to figure out the ranges for each code in order to determine
>>                the largest range, which is then the one used for the overall backtesting.  At any rate, since backtest_multi doesn't provide a list of calc
>>                objects and doesn't provide an argument list when calling backtest_multi that matches this vesion of the code, it simply can't work
>>                as intended.
> 
> 
> and this too (may be a screwup)
> 
> frankly i don't use the backtest*.pl apps much, and backtest_multi.pl least of all.
> 
> it just seemed 'reasonable' to move the (potentially common) stuff into GT/BackTest.pm
> that really belongs there and remove it from the script apps to in an effort to
> eliminate duplication and the need for duplicate maintenance ...
> 
> i may have not completed the task, botched it completely or just made a muddle of it.
> 
> probably a bit (well more like a double long) of all three
> 
> 
> 
> 
> so the bottom lines are:
> 
>   @ reinstate the original $buyhold change percentage calculation
> 
>   @ do not use script app ./backtest_multi.pl *it is* 'not ready for prime time'.
>     probably never really was, but that's another story.
> 
>   @ GT/BackTest.pm sub backtest_multi *is* 'not ready for prime time'.
> 
>   @ without more specifics i'm unable to duplicate undefs that might result
>     from $*->{'buyandhold_max_draw_down'} objects no longer being initialized.
> 
>     but if you want to evaluate reinstating them the method (line 469) must
>     be changed to
>        $indicator_maxdd->find_calculator
>     not the commented out $indicator_maxdd->calculator, which is not available
>     in trunk-head.
> 
>     and the calls vars $calc $last will need to be valid within this scope.
> 
>   @ don't see there's a perl compile/runtime problem caused by the removal of
>         use GT::Tools qw(:timeframe);
>     unless it is part of the failure of ./backtest_multi.pl ...
> 
>   @ finally there may be a good reason to call method find_calculator from
>     GT/BackTest.pm, but i just haven't figured that out yet.
> 
> 
> 
> aloha
> 
> ras
> 
> 
> < snip of the rest >
> 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 

Re: [GT] revised backtest.pl for evaluation and comment

[Robert A. S. <ra...@ac...>]

gt'ers


attached is an alternate to the flawed backtest.pl as derek has
noted in this thread.

i'll commit it in a few days unless there's more commentary ...

note: i have not looked at backtest_m*.pl to see if there is any
common code in those files that also be changed, so commentary
regarding them is also welcome.

aloha

ras



Robert A. Schmied wrote:
> Shinaberry Derek wrote:
> 
>>On Aug 22, 2012, at 17:47 , Robert A. Schmied wrote:
>>
>>
>>
>>>Shinaberry Derek wrote:
>>>
>>>
>>>>There are, I think, four potential problems with this revision.
>>>>Each issue is broken out below with its own snippet of code for
>>>>documentation.  Thoughts or oversights on my part? Derek The
>>>>following code snippet was removed and while maybe not a major
>>>>deal, I'm not sure it was wise to remove it. Since there is no
>>>>default Close Strategy installed, it probably is wise to require
>>>>a close strategy when the system name is supplied from the
>>>>command line as there is no other source for a close strategy in
>>>>this case. r709 Scripts/backtest.pl:263-265
>>>
>>>aloha derek
>>>
>>>i'm not following either your (svn) version numbers or (some of)
>>>your line number references at least for the csname issue.
>>>
>>>my research shows svn r709 to be related to the old website move to
>>>sourceforge. recent changes directly related to Scripts/backtest.pl
>>>are: 716  	40d 20h  	ras2010  	/trunk/Scripts/backtest.pl  	set
>>>default timeframe to 'day', include terse cryptic sub usage that
>>>was missing
>>>
>>>715 	135d 14h 	ras2010 	/trunk/Scripts/backtest.pl 	incorporate
>>>command line argument --initial_value to set value of initial
>>>amount of money for investment this value can also be set from
>>>configuration key "backtest::initial_value".
>>>
>>>658 	1469d 21h 	joao 	/trunk/Scripts/backtest.pl 	Fix the
>>>max-loaded-items option in scripts
>>>
>>>
>>
>>Hi Robert,
>>
>>By revision 709, I meant to refer to any file under the trunk as it
>>appeared immediately following the commit for revision 709.  Revision
>>658 was the last commit to modify backtest.pl prior to revision 709,
>>so revisions 658 through 714 of backtest.pl are all identical.  The
>>tarball I originally downloaded from the website happens to
>>correspond to revision 709, so that's why that particular revision
>>number has popped up here.  The way I conceptualize svn is that every
>>revision is a snapshot of the entire repository regardless of when
>>the last commit actually modified any given file.  I hope that clears
>>things up; sorry for the confusion.  Anyway it looks like you managed
>>to find the code in question in spite of my attempts to obfuscate.
>>;-)
>>
>>Cheers, Derek
>>
>>
>>>ras
>>>
>>>
>>>
>>>i do see the deathnell message requiring at least one closestrat be
>>>part of a trading system has been removed. this was because by
>>>default a missing closestrat will always result in the 'do nothing'
>>>default closestrat CS:NeverClose. (iirc this is enforced by
>>>add_position_manager).
>>>
>>>this isn't obvious based on the data emitted by the app unless you
>>>study the output results including the trade history chart.
>>>
>>
>>I suppose we could have a philosophical debate on the usefulness of
>>backtesting a buy and hold system, i.e. CS:NeverClose.  Especially
>>when the backtest report will provide a comparison with buy and hold.
>>;-)
>>
>>What you describe is true enough though.  It looks like although the
>>code provides a routine to install a default CS, backtest.pl never
>>avails itself of that.  The result being that with no CS installed,
>>no CS code is ever called, which results in identical behavior to 
>>explicitly installing CS:NeverClose.  You say po-TA-toe and I say
>>po-TAH-toe; let's call the whole thing off. :-)
>>
> 
> 
> you are correct.
> 
> 
>>>here's a working minimal command line: ras [ 266 ] %
>>>./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
>>>
>>>n.b. i've a non-standard tweaked ./backtest.pl that does a better
>>>job describing the actual trading system being used, but it
>>>(probably) requires, in a perl sense, other non-standard ras hacks
>>>to the gt toolkit. contact me directly for more info.
>>>
>>>here's that same example with my hack ... ras [ 270 ] %
>>>./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG 
>>>backtest.pl: warning: setting default CloseStrategy to NeverClose .
>>> .
>>>
>>>
>>>
>>>>- if ($system && ! scalar(@csname)) { -       die "You must give
>>>>at least one --close-strategy argument !\n"; - } It looks like
>>>>some redundant code found its way in here, but even if you fix
>>>>the redundancy I'm not sure the code is kosher.  It appears the
>>>>intent is to warn the user if the default Money Management rule
>>>>is installed, but the default_money_management subroutine
>>>>explicitly returns nothing, so testing it yields nothing. Maybe
>>>>it works in an alternative experimental branch? r709
>>>>Scripts/backtest.pl:385-398 foreach (@mmname) { 
>>>>$pf_manager->add_money_management_rule( 
>>>>create_standard_object(split (/\s+/, "MoneyManagement::$_"))); } 
>>>>+ my $mmbasic_added = $pf_manager->default_money_management_rule(
>>>> +  create_standard_object("MoneyManagement::Basic")); + warn
>>>>"$prog_name: warning: added mm::basic to system\n" +  if
>>>>$mmbasic_added && ( $verbose || $debug ); + 
>>>>$pf_manager->default_money_management_rule( 
>>>>create_standard_object("MoneyManagement::Basic")); + warn
>>>>"$prog_name: warning: added mm::basic to system\n" +  if
>>>>$mmbasic_added && ( $verbose || $debug );
>>>
>>>ok these line refs match with r716 the intent is to install the
>>>minimal moneymanager (basic) in the absence of one. i believe it
>>>does work as intended; refer to attached example output line 1. 
>>>(the -rh.out attachment from ras hacked backtest.pl includes
>>>on-going diagnostics but gives identical results otherwise)
>>>
>>>however, i didn't debug the actual code, just the results, but the
>>>code does look a bit redundant ... added to 'todo list' thinking
>>>out loud without any testing ::: lines 395-398 are redundant
>>
>>
>>Let me clarify.  My nit isn't that MM:Basic won't be installed; I
>>agree with you that it will be installed as your output example
>>demonstrates.  What I think doesn't work as expected is the "warn if
>>$mmbasic_added".
> 
> 
> you are correct.
> 
> 
>>I think $mmbasic_added will always be undef.  The way
>>default_money_management_rule works, you can never know whether the
>>provided MM is installed or not without knowing a priori that no
>>other MMs have been added.  In this particular case, it might be more
>>feasible to test whether @mmname is empty and if so, then install the
>>default and issue the warning.
> 
> 
> yep and yep. i still think this sort of user warning info is appropriate
> for new users ...
> 
> yet another fix-it on the todo list
> 
> with this code fragment in backtest.pl (forgive the line number walking)
>    385	foreach (@mmname)
>    386	{
>    387	    $pf_manager->add_money_management_rule(
>    388		create_standard_object(split (/\s+/, "MoneyManagement::$_")));
>    389	}
>    390	# my $mmbasic_added = $pf_manager->default_money_management_rule(
>    391	#  create_standard_object("MoneyManagement::Basic"));
>    392	# #warn "$prog_name: warning: added mm::basic to system\n"
>    393	# # if $mmbasic_added && ( $verbose || $debug );
>    394	# 
>    395	# warn "$prog_name: mmbasic_added is \"$mmbasic_added\"\n";
>    396	# 
>    397	$pf_manager->default_money_management_rule(
>    398	       create_standard_object("MoneyManagement::Basic"));
>    399	#warn "$prog_name: warning: added mm::basic to system\n"
>    400	# if $mmbasic_added && ( $verbose || $debug );
>    401	# 
>    402	# warn "$prog_name: mmbasic_added is \"$mmbasic_added\"\n";
> 
> i'm getting MM::Basic automatically? with this command line:
>      %    ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
> 
> and no MM::Basic with this command line:
>      %    ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" -mm=Martingale GOOG
> 
> pos backtest.pl will not read a file containing a trading system
> [i think having that feature would be a big improvement]
> so if there's a global alias defined:
>    Aliases::Global::EMA_OPT[]    SY:Generic \
>      { S:Generic:CrossOverUp \
>        { I:EMA #1 } \
>        { I:EMA #2 } \
>      } \
>   | CS:Generic { S:Generic:CrossOverDown { I:EMA #3 } { I:EMA #1 } } \
>   | MM:FixedSum 5000
> 
> the command line:
>       %    ./backtest.pl --iv=100000 --broker="SelfTrade" 'EMA_OPT[ 30, 100, 80 ]' GOOG
> 
> results in no explicit addition of MM::Basic so i'm ready to accept
> the above fragment as a suitable fix for this issue. this means
> the commented out lines 390..396 and 399..402 can be removed.
> 
> if there is some way to parse the $pf_manager object to evaluate
> if MM::Basic alone is there and inform the user of that (unless
> the user requested it via -mm) that would be a good thing i think,
> but certainly not critical ...
> 
> 
> 
>>Does that make any sense?  I hope that makes sense.  Hmm...I'm not
>>sure that makes sense.
>>
>>
>>>>Seems to have a similar problem to the money management code
>>>>above in that set_order_factory has no explicit return value.
>>>>Thus, the implicit return value will be the value of the order
>>>>factory passed in as an argument, which in this case will always
>>>>be undef.  So, unless I'm missing something, this is a quite
>>>>roundabout way of saying "if ( ! defined undef )", which is
>>>>always true by definition. r709 Scripts/backtest.pl:400-414 if
>>>>($ofname) { $sys_manager->set_order_factory( 
>>>>create_standard_object(split (/\s+/, "OrderFactory::$ofname"))); 
>>>>} else { -     if ( $verbose || $debug ) { +     my $of =
>>>>$sys_manager->set_order_factory(); +     if ( ! defined $of && (
>>>>$verbose || $debug ) ) { my $msg = join "", "$prog_name: notice:
>>>>", "without explicit orderfactory the implicit default is\n", 
>>>>"OF::MarketPrice. ", "this equates to market open price day after
>>>>the order.", "\n"; warn "$msg"; } } The else clause was added,
>>>>but is it really a good idea to force the portfolio to be stored?
>>>> The description of the --store flag says nothing about a default
>>>>if the option is unspecified. If it is realy important to store
>>>>the portfolio, maybe it shouldn't be an option.
>>>
>>>valid point ... added to 'todo list'
>>
>>
>>Not sure whether your valid point comment applied to the order
>>factory issue above or the else clause issue below. If it didn't
>>apply to the order factory issue, then I don't think I saw any
>>comments about it.
> 
> 
> applies to the else clause below, without any testing i commented
> out the else part in my svn repo backtest.pl file.
> 
> 
> i must have glazed over on the orderfactory issue, you are correct
> that it is also non-functional code.
> 
> but the intent was to inform the user that the default orderfactory
> is OF::MarketPrice.
> 
> this trading system component is also not 'explicitly' present in the
> listed system even though i do think it is implicit in the tool kit code
> when no orderfactory is spec'ed in the trading system.
>     %     ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
>     ## Analysis of SY:TFS 50 7|MM:Basic
>     History of the portfolio :
>     ...
> 
> 
> 
> bottom lines
> 
> looks like an update of Scripts/backtest.pl is needed to remove the
> cruft that has been accumulating. i've put that on my to-do list,
> but there are other things there that are more important, to me anyway,
> so please be patient. after all this code is, for the most part,
> non-functional, so while it may waste cpu and memory, it does no
> other harm so the update isn't deemed critical. on the other hand
> if anyone wants to hack away and submit an improved version please
> do so.
> 
> additionally, much of this code might also be found in Scripts/backtest_*.pl
> so those files also deserve a good going over as well.
> 
> 
> aloha
> 
> ras
> 
> 
> 
>>
>>>thinking out loud without any testing ::: lines 452 and 353 are not
> 
>                                                        453 ^^^
> 
>>>appropriate but might be conditioned with $debug or something
>>>similar.
>>>
>>>
>>>
>>>>r709 Scripts/backtest.pl:447-451 if ($store_file) { 
>>>>$analysis->{'portfolio'}->store($store_file); + } else { +
>>>>$analysis->{'portfolio'}->store("./bt_portfolio.xml"); }
>>
>><snip>
>>
>>
>>------------------------------------------------------------------------------
>> Live Security Virtual Conference Exclusive live event will cover all
>>the ways today's security and threat landscape has changed and how IT
>>managers can respond. Discussions will include endpoint security,
>>mobile security and the latest in malware threats.
>>http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>
> 
> 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 

Re: [GT] was: Re: Problems With Rev 715 Scripts/backtest.pl - Incorporation Of Initial Value Command Line Option

[Robert A. S. <ra...@ac...>]

Shinaberry Derek wrote:
> On Aug 22, 2012, at 17:47 , Robert A. Schmied wrote:
> 
> 
>> Shinaberry Derek wrote:
>> 
>>> There are, I think, four potential problems with this revision.
>>> Each issue is broken out below with its own snippet of code for
>>> documentation.  Thoughts or oversights on my part? Derek The
>>> following code snippet was removed and while maybe not a major
>>> deal, I'm not sure it was wise to remove it. Since there is no
>>> default Close Strategy installed, it probably is wise to require
>>> a close strategy when the system name is supplied from the
>>> command line as there is no other source for a close strategy in
>>> this case. r709 Scripts/backtest.pl:263-265
>> 
>> aloha derek
>> 
>> i'm not following either your (svn) version numbers or (some of)
>> your line number references at least for the csname issue.
>> 
>> my research shows svn r709 to be related to the old website move to
>> sourceforge. recent changes directly related to Scripts/backtest.pl
>> are: 716  	40d 20h  	ras2010  	/trunk/Scripts/backtest.pl  	set
>> default timeframe to 'day', include terse cryptic sub usage that
>> was missing
>> 
>> 715 	135d 14h 	ras2010 	/trunk/Scripts/backtest.pl 	incorporate
>> command line argument --initial_value to set value of initial
>> amount of money for investment this value can also be set from
>> configuration key "backtest::initial_value".
>> 
>> 658 	1469d 21h 	joao 	/trunk/Scripts/backtest.pl 	Fix the
>> max-loaded-items option in scripts
>> 
>> 
> 
> Hi Robert,
> 
> By revision 709, I meant to refer to any file under the trunk as it
> appeared immediately following the commit for revision 709.  Revision
> 658 was the last commit to modify backtest.pl prior to revision 709,
> so revisions 658 through 714 of backtest.pl are all identical.  The
> tarball I originally downloaded from the website happens to
> correspond to revision 709, so that's why that particular revision
> number has popped up here.  The way I conceptualize svn is that every
> revision is a snapshot of the entire repository regardless of when
> the last commit actually modified any given file.  I hope that clears
> things up; sorry for the confusion.  Anyway it looks like you managed
> to find the code in question in spite of my attempts to obfuscate.
> ;-)
> 
> Cheers, Derek
> 
>> ras
>> 
>> 
>> 
>> i do see the deathnell message requiring at least one closestrat be
>> part of a trading system has been removed. this was because by
>> default a missing closestrat will always result in the 'do nothing'
>> default closestrat CS:NeverClose. (iirc this is enforced by
>> add_position_manager).
>> 
>> this isn't obvious based on the data emitted by the app unless you
>> study the output results including the trade history chart.
>> 
> 
> I suppose we could have a philosophical debate on the usefulness of
> backtesting a buy and hold system, i.e. CS:NeverClose.  Especially
> when the backtest report will provide a comparison with buy and hold.
> ;-)
> 
> What you describe is true enough though.  It looks like although the
> code provides a routine to install a default CS, backtest.pl never
> avails itself of that.  The result being that with no CS installed,
> no CS code is ever called, which results in identical behavior to 
> explicitly installing CS:NeverClose.  You say po-TA-toe and I say
> po-TAH-toe; let's call the whole thing off. :-)
> 

you are correct.

> 
>> here's a working minimal command line: ras [ 266 ] %
>> ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
>> 
>> n.b. i've a non-standard tweaked ./backtest.pl that does a better
>> job describing the actual trading system being used, but it
>> (probably) requires, in a perl sense, other non-standard ras hacks
>> to the gt toolkit. contact me directly for more info.
>> 
>> here's that same example with my hack ... ras [ 270 ] %
>> ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG 
>> backtest.pl: warning: setting default CloseStrategy to NeverClose .
>>  .
>> 
>> 
>>> - if ($system && ! scalar(@csname)) { -       die "You must give
>>> at least one --close-strategy argument !\n"; - } It looks like
>>> some redundant code found its way in here, but even if you fix
>>> the redundancy I'm not sure the code is kosher.  It appears the
>>> intent is to warn the user if the default Money Management rule
>>> is installed, but the default_money_management subroutine
>>> explicitly returns nothing, so testing it yields nothing. Maybe
>>> it works in an alternative experimental branch? r709
>>> Scripts/backtest.pl:385-398 foreach (@mmname) { 
>>> $pf_manager->add_money_management_rule( 
>>> create_standard_object(split (/\s+/, "MoneyManagement::$_"))); } 
>>> + my $mmbasic_added = $pf_manager->default_money_management_rule(
>>>  +  create_standard_object("MoneyManagement::Basic")); + warn
>>> "$prog_name: warning: added mm::basic to system\n" +  if
>>> $mmbasic_added && ( $verbose || $debug ); + 
>>> $pf_manager->default_money_management_rule( 
>>> create_standard_object("MoneyManagement::Basic")); + warn
>>> "$prog_name: warning: added mm::basic to system\n" +  if
>>> $mmbasic_added && ( $verbose || $debug );
>> 
>> ok these line refs match with r716 the intent is to install the
>> minimal moneymanager (basic) in the absence of one. i believe it
>> does work as intended; refer to attached example output line 1. 
>> (the -rh.out attachment from ras hacked backtest.pl includes
>> on-going diagnostics but gives identical results otherwise)
>> 
>> however, i didn't debug the actual code, just the results, but the
>> code does look a bit redundant ... added to 'todo list' thinking
>> out loud without any testing ::: lines 395-398 are redundant
> 
> 
> Let me clarify.  My nit isn't that MM:Basic won't be installed; I
> agree with you that it will be installed as your output example
> demonstrates.  What I think doesn't work as expected is the "warn if
> $mmbasic_added".

you are correct.

> 
> I think $mmbasic_added will always be undef.  The way
> default_money_management_rule works, you can never know whether the
> provided MM is installed or not without knowing a priori that no
> other MMs have been added.  In this particular case, it might be more
> feasible to test whether @mmname is empty and if so, then install the
> default and issue the warning.

yep and yep. i still think this sort of user warning info is appropriate
for new users ...

yet another fix-it on the todo list

with this code fragment in backtest.pl (forgive the line number walking)
   385	foreach (@mmname)
   386	{
   387	    $pf_manager->add_money_management_rule(
   388		create_standard_object(split (/\s+/, "MoneyManagement::$_")));
   389	}
   390	# my $mmbasic_added = $pf_manager->default_money_management_rule(
   391	#  create_standard_object("MoneyManagement::Basic"));
   392	# #warn "$prog_name: warning: added mm::basic to system\n"
   393	# # if $mmbasic_added && ( $verbose || $debug );
   394	# 
   395	# warn "$prog_name: mmbasic_added is \"$mmbasic_added\"\n";
   396	# 
   397	$pf_manager->default_money_management_rule(
   398	       create_standard_object("MoneyManagement::Basic"));
   399	#warn "$prog_name: warning: added mm::basic to system\n"
   400	# if $mmbasic_added && ( $verbose || $debug );
   401	# 
   402	# warn "$prog_name: mmbasic_added is \"$mmbasic_added\"\n";

i'm getting MM::Basic automatically? with this command line:
     %    ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG

and no MM::Basic with this command line:
     %    ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" -mm=Martingale GOOG

pos backtest.pl will not read a file containing a trading system
[i think having that feature would be a big improvement]
so if there's a global alias defined:
   Aliases::Global::EMA_OPT[]    SY:Generic \
     { S:Generic:CrossOverUp \
       { I:EMA #1 } \
       { I:EMA #2 } \
     } \
  | CS:Generic { S:Generic:CrossOverDown { I:EMA #3 } { I:EMA #1 } } \
  | MM:FixedSum 5000

the command line:
      %    ./backtest.pl --iv=100000 --broker="SelfTrade" 'EMA_OPT[ 30, 100, 80 ]' GOOG

results in no explicit addition of MM::Basic so i'm ready to accept
the above fragment as a suitable fix for this issue. this means
the commented out lines 390..396 and 399..402 can be removed.

if there is some way to parse the $pf_manager object to evaluate
if MM::Basic alone is there and inform the user of that (unless
the user requested it via -mm) that would be a good thing i think,
but certainly not critical ...


> 
> Does that make any sense?  I hope that makes sense.  Hmm...I'm not
> sure that makes sense.
> 
>>> Seems to have a similar problem to the money management code
>>> above in that set_order_factory has no explicit return value.
>>> Thus, the implicit return value will be the value of the order
>>> factory passed in as an argument, which in this case will always
>>> be undef.  So, unless I'm missing something, this is a quite
>>> roundabout way of saying "if ( ! defined undef )", which is
>>> always true by definition. r709 Scripts/backtest.pl:400-414 if
>>> ($ofname) { $sys_manager->set_order_factory( 
>>> create_standard_object(split (/\s+/, "OrderFactory::$ofname"))); 
>>> } else { -     if ( $verbose || $debug ) { +     my $of =
>>> $sys_manager->set_order_factory(); +     if ( ! defined $of && (
>>> $verbose || $debug ) ) { my $msg = join "", "$prog_name: notice:
>>> ", "without explicit orderfactory the implicit default is\n", 
>>> "OF::MarketPrice. ", "this equates to market open price day after
>>> the order.", "\n"; warn "$msg"; } } The else clause was added,
>>> but is it really a good idea to force the portfolio to be stored?
>>>  The description of the --store flag says nothing about a default
>>> if the option is unspecified. If it is realy important to store
>>> the portfolio, maybe it shouldn't be an option.
>> 
>> valid point ... added to 'todo list'
> 
> 
> Not sure whether your valid point comment applied to the order
> factory issue above or the else clause issue below. If it didn't
> apply to the order factory issue, then I don't think I saw any
> comments about it.

applies to the else clause below, without any testing i commented
out the else part in my svn repo backtest.pl file.


i must have glazed over on the orderfactory issue, you are correct
that it is also non-functional code.

but the intent was to inform the user that the default orderfactory
is OF::MarketPrice.

this trading system component is also not 'explicitly' present in the
listed system even though i do think it is implicit in the tool kit code
when no orderfactory is spec'ed in the trading system.
    %     ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
    ## Analysis of SY:TFS 50 7|MM:Basic
    History of the portfolio :
    ...



bottom lines

looks like an update of Scripts/backtest.pl is needed to remove the
cruft that has been accumulating. i've put that on my to-do list,
but there are other things there that are more important, to me anyway,
so please be patient. after all this code is, for the most part,
non-functional, so while it may waste cpu and memory, it does no
other harm so the update isn't deemed critical. on the other hand
if anyone wants to hack away and submit an improved version please
do so.

additionally, much of this code might also be found in Scripts/backtest_*.pl
so those files also deserve a good going over as well.


aloha

ras


> 
> 
>> thinking out loud without any testing ::: lines 452 and 353 are not
                                                       453 ^^^
>> appropriate but might be conditioned with $debug or something
>> similar.
>> 
>> 
>>> r709 Scripts/backtest.pl:447-451 if ($store_file) { 
>>> $analysis->{'portfolio'}->store($store_file); + } else { +
>>> $analysis->{'portfolio'}->store("./bt_portfolio.xml"); }
> 
> <snip>
> 
> 
> ------------------------------------------------------------------------------
>  Live Security Virtual Conference Exclusive live event will cover all
> the ways today's security and threat landscape has changed and how IT
> managers can respond. Discussions will include endpoint security,
> mobile security and the latest in malware threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>