geniustrader-devel — participate in geniustrader development, ask for help, suggest improvements, etc

[GT] detail findings on GT/BackTest.pm was Re: Problems With Rev 714 GT/BackTest.pm - Fixing The UNIX Epoch Problem

[Robert A. S. <ra...@ac...>]

Shinaberry Derek wrote:
> None of the code changes seem to have anything to do with the UNIX epoch problem.  I don't know whether the comment was a copy and paste error or whether the wrong version of GT/BackTest.pm was committed.
> 
> Assuming that the intended version of BackTest.pm was, in fact, the version committed, I see the following potential issues:
> 
>         A significant number of changes, mostly to the backtest_mutil subroutine, appear to have come from some hacking that was perhaps aimed at eliminating
>         the need to call find_calculator for all the codes passed into the routine.  It looks like these changes were only half baked and their inclusion has
>         broken backtest_multi.  I've included a unified diff output where the FROM file is a theorhetically unbroken form of the r714 revision of my own
> 	creation and the TO file is the actual r714 revision in order to highlight the problems.  I've provided commentary for each of the change blocks to
> 	clarify what I think is a problem.
> 
>         @@ -152,7 +151,9 @@
>                 This is the one change not related to backtest_multi that I think is a problem.  The removal of the " - 1" changes the result from a
>                 percentage to a ratio, e.g. from 0.05 to 1.05.  It is unclear to me how this might ripple through the reporting code since this value is
>                 loaded into the analysis hash under the 'real' key and the analysis hash is returned from the backtest subroutine.
>      # Buy & hold
>      my $buyhold = 0;
>      $buyhold = $calc->prices->at($last)->[$LAST] /
> -      $calc->prices->at($first)->[$LAST] - 1 if ( $calc->prices->at($first)->[$LAST] != 0);
> +#      $calc->prices->at($first)->[$LAST] - 1
> +      $calc->prices->at($first)->[$LAST]
> +       if ( $calc->prices->at($first)->[$LAST] != 0);

oops -- good catch -- i must have thought $calc->prices->at($first)->[$LAST] - 1
yielded a time frame index.

> 
>         @@ -442,10 +457,6 @@
>                 It's not clear to me why this code was removed as I believe that the condition that the warning note refers to still applies.
> @@ -442,10 +457,6 @@
>  #      $re->{'buyandhold_max_draw_down'} = $buyandhold_maxdd;
>  #      $th->{'buyandhold_max_draw_down'} = $buyandhold_maxdd;
>  #    }
> -    ### NOTE: Need to assign something to these to avoid undef value in output
> -    $re->{'buyandhold_max_draw_down'} = 0;
> -    $th->{'buyandhold_max_draw_down'} = 0;
> -
> 
>      # Hack to remove code reference in portfolio
>      delete $p->{'date2int'};

dunno will look at in more detail ...

i'm not getting undef in output is there a command line test case that can be
used to fix a baseline? i typically fixate on sample_data/13000.txt as the market
file since that one is 'generally' available.

> 
>         @@ -12,7 +12,8 @@
>         @@ -491,9 +502,47 @@
>                 The subroutine setup_backtest is unused at this point and appears to me that perhaps the changes were headed towards its use, but it doesn't
>                 seem ready yet and should probably be removed or, at a minimum, heavily documented as to its experimental status.
> @@ -491,9 +502,47 @@
>      return $managers;
>  }
> 
> -=pod
> +=item C<< setup_backtest($code, $timeframe, $full, $start, $end) >>
> 
> -=back
> +Setup the system backtest: Returns $calc (the calculator), as well as
> +$first and $last (indices used by the calculator).

i believe setup_backtest is an artifact from the thomas weigert effort
from backtest_opt.pl (likely my name for it) also maybe called backtest_optimal.pl
where in an effort was made to vary parameters of the trading system in
an attempt to determine optimum values.

the method was installed into gt::backtest.pm so that it was generally
available if and when it might be useful.

refer -devel archive "[GT] System optimization" dated: 08/19/05 20:30

> 
>         @@ -21,8 +22,6 @@
>                 This use declaration is where visibility to GT::Tools::find_calculator comes from.  Assuming that I am correct about the need to restore
>                 the other code which uses find_calculator, then this should be restored as well.
> @@ -21,8 +22,6 @@
>  use GT::DateTime;
>  ###l4p  use Log::Log4perl qw(:easy);
>  use GT::Indicators::MaxDrawDown;
> -use GT::Tools qw(:timeframe);
> -

is there an actual problem that you can demonstrate or are we conducting a code review?

it seems to me GT/BackTest.pm is only ever used in connection with a gt script app, which
will have already established a calculator object (e.g. $calc). i don't think there is
any [good] reason for another calculator object being created by this module directly ...

> 
>         @@ -217,10 +218,10 @@
>                 The parameter list for backtest_multi has completely changed and this doesn't match the code in backtest_multi.pl that calls it anymore.
>                 In addition to a number of parameters being dropped, $code_ref is changed to $calc_ref and the corresponding list $codes is changed to @calcs.
>                 This is what drove my assumption that most of the changes are targeted at elimiinating the need to use find_calculator to get a calc
>                 object corresponding to a code symbol.
> @@ -217,10 +218,10 @@
> 
> 
>  sub backtest_multi {
> -    my ($db, $pf_manager, $sys_manager_ref, $broker_ref, $code_ref, $timeframe, $full, $start, $end, $nb_item, $max_loaded_items, $init) = @_;
> +    my ($pf_manager, $sys_manager_ref, $broker_ref, $calc_ref, $start, $end, $full, $init) = @_;
>      my @sysmanager = @{$sys_manager_ref};
>      my @brokers = @{$broker_ref};
> -    my @codes = @{$code_ref};
> +    my @calc = @{$calc_ref};
>      $init = 10000 unless ( defined($init) );
> 
>      # Create an empty portfolio object and make the manager use it

this may be a screwup on my part ...

with
   market_file === AAPL
   system_file === crossing_ema
       with .gt/options file containing:
           Aliases::Global::crossing_ema           SY:G \
             { S:G:CrossOverUp   {I:EMA 30} {I:EMA 100} } \
             { S:G:CrossOverDown {I:EMA 30} {I:EMA 100} } \
           | CS:OppositeSignal


   ras [ 484 ] %     ( pushd /usr/local/src/genius_trader/svn_repo/Scripts; \
     ./backtest_multi.pl --start=2010-08-01  --end=today  \
       --output-directory /var/tmp \
       --broker=NoCosts \
       ../../Scripts/market_file  \
       ../../Scripts/js_sys_bt_mlt ; \
     popd ; )

   Not an ARRAY reference at ../GT/BackTest.pm line 222, <> line 164.

consider ./backtest_multi.pl 'not ready for prime time'


but for whatitisworth (not much frankly) with same conditions
    ras [ 487 ] %       ( pushd /usr/local/src/genius_trader/svn_repo/Scripts; \
     ./backtest_many.pl --start=2010-08-01  --end=today  \
       --output-directory /var/tmp \
       --broker=NoCosts \
       ../../Scripts/market_file  \
       ../../Scripts/js_sys_bt_mlt ; \
    popd ; )
    ##
    ## Analysis of code AAPL using System Spec

    SY:G { S:G:CrossOverUp {I:EMA 30} {I:EMA 100} }
     { S:G:CrossOverDown {I:EMA 30} {I:EMA 100} } | CS:OppositeSignal

    ## Global analysis (full portfolio invested)
    Analysis of the portfolio (2010-08-02 / 2012-08-22)
    -----------------------------------------------------
    Performance :  71.5% ( 30.9%)     Buy & Hold : 255.4% ( 88.3%) () => by year
    MaxDrawDown :        7.3%  B&H MaxDrawDown :    81.8%
    Best performance :  71.5%  Worst performance :  -7.3%
    Net gain :        7150.25  Gross gain :       7150.25

    Trades statistics
    Number of trades :      2  Trades/Year :         0.97
    Number of gains :       1  Number of losses :       1  Win. ratio :   50.0%
    Max consec. win :       1  Max consec. loss :       1  Expectancy :    0.41
    Average gain :     89.20%  Average loss :      -7.43%  Avg. perf  :  30.96%
    Biggest gain :     89.20%  Biggest loss :      -7.43%  Profit fac :   12.01
    Sum of gains :    7883.75  Sum of losses :    -733.50  Risk of ruin : 32.4%

does still appear functional

 
> 
>         @@ -245,42 +246,56 @@
>                 There are a lot of small sub-blocks here, but I believe the overarching theme is getting rid of the call to find_calculator.  I'm not sure
>                 where the intended destination for the responsibility of getting calculators is, but it seems to be upstream from backtest_multi, so it is
>                 probably in backtest_multi.pl.  The trouble with that (and this) is that a chunk of code that is already in find_calculator is duplicated
>                 here (and again in the, as yet unused, subroutine setup_backtest) in order to figure out the ranges for each code in order to determine
>                 the largest range, which is then the one used for the overall backtesting.  At any rate, since backtest_multi doesn't provide a list of calc
>                 objects and doesn't provide an argument list when calling backtest_multi that matches this vesion of the code, it simply can't work
>                 as intended.

and this too (may be a screwup)

frankly i don't use the backtest*.pl apps much, and backtest_multi.pl least of all.

it just seemed 'reasonable' to move the (potentially common) stuff into GT/BackTest.pm
that really belongs there and remove it from the script apps to in an effort to
eliminate duplication and the need for duplicate maintenance ...

i may have not completed the task, botched it completely or just made a muddle of it.

probably a bit (well more like a double long) of all three




so the bottom lines are:

  @ reinstate the original $buyhold change percentage calculation

  @ do not use script app ./backtest_multi.pl *it is* 'not ready for prime time'.
    probably never really was, but that's another story.

  @ GT/BackTest.pm sub backtest_multi *is* 'not ready for prime time'.

  @ without more specifics i'm unable to duplicate undefs that might result
    from $*->{'buyandhold_max_draw_down'} objects no longer being initialized.

    but if you want to evaluate reinstating them the method (line 469) must
    be changed to
       $indicator_maxdd->find_calculator
    not the commented out $indicator_maxdd->calculator, which is not available
    in trunk-head.

    and the calls vars $calc $last will need to be valid within this scope.

  @ don't see there's a perl compile/runtime problem caused by the removal of
        use GT::Tools qw(:timeframe);
    unless it is part of the failure of ./backtest_multi.pl ...

  @ finally there may be a good reason to call method find_calculator from
    GT/BackTest.pm, but i just haven't figured that out yet.



aloha

ras


< snip of the rest >

Re: [GT] Problems With Rev 715 Scripts/backtest.pl - Incorporation Of Initial Value Command Line Option

[Shinaberry D. <gen...@sh...>]

On Aug 22, 2012, at 17:47 , Robert A. Schmied wrote:

> Shinaberry Derek wrote:
>> There are, I think, four potential problems with this revision.  Each issue is broken out below with its own snippet of code for documentation.  Thoughts or oversights on my part?
>> Derek
>>        The following code snippet was removed and while maybe not a major deal, I'm not sure it was wise to remove it.
>>        Since there is no default Close Strategy installed, it probably is wise to require a close strategy when the
>>        system name is supplied from the command line as there is no other source for a close strategy in this case.
>>        r709 Scripts/backtest.pl:263-265
> 
> aloha derek
> 
> i'm not following either your (svn) version numbers or (some of) your line number references
> at least for the csname issue.
> 
> my research shows svn r709 to be related to the old website move to sourceforge.
> recent changes directly related to Scripts/backtest.pl are:
> 716  	40d 20h  	ras2010  	/trunk/Scripts/backtest.pl  	set default timeframe to 'day', include terse cryptic sub usage that was missing
> 
> 715 	135d 14h 	ras2010 	/trunk/Scripts/backtest.pl 	incorporate command line argument --initial_value to set value of initial amount of money for investment this value can also be set from configuration key "backtest::initial_value".
> 
> 658 	1469d 21h 	joao 	/trunk/Scripts/backtest.pl 	Fix the max-loaded-items option in scripts
> 
> 
Hi Robert,

By revision 709, I meant to refer to any file under the trunk as it appeared immediately following the commit for revision 709.  Revision 658 was the last commit to modify backtest.pl prior to revision 709, so revisions 658 through 714 of backtest.pl are all identical.  The tarball I originally downloaded from the website happens to correspond to revision 709, so that's why that particular revision number has popped up here.  The way I conceptualize svn is that every revision is a snapshot of the entire repository regardless of when the last commit actually modified any given file.  I hope that clears things up; sorry for the confusion.  Anyway it looks like you managed to find the code in question in spite of my attempts to obfuscate. ;-)

Cheers,
Derek
> 
> ras
> 
> 
> 
> i do see the deathnell message requiring at least one closestrat be part of
> a trading system has been removed. this was because by default a missing
> closestrat will always result in the 'do nothing' default closestrat CS:NeverClose.
> (iirc this is enforced by add_position_manager).
> 
> this isn't obvious based on the data emitted by the app unless you study the
> output results including the trade history chart.
> 
I suppose we could have a philosophical debate on the usefulness of backtesting a buy and hold
system, i.e. CS:NeverClose.  Especially when the backtest report will provide a comparison with
buy and hold. ;-)

What you describe is true enough though.  It looks like although the code provides
a routine to install a default CS, backtest.pl never avails itself of that.  The result being
that with no CS installed, no CS code is ever called, which results in identical behavior to
explicitly installing CS:NeverClose.  You say po-TA-toe and I say po-TAH-toe; let's call the
whole thing off. :-)

> here's a working minimal command line:
>    ras [ 266 ] %      ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
> 
> n.b. i've a non-standard tweaked ./backtest.pl that does a better job describing
> the actual trading system being used, but it (probably) requires, in a perl sense,
> other non-standard ras hacks to the gt toolkit. contact me directly for more info.
> 
> here's that same example with my hack ...
>    ras [ 270 ] %     ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
>    backtest.pl: warning: setting default CloseStrategy to NeverClose
>    .
>    .
> 
>> - if ($system && ! scalar(@csname)) {
>> -       die "You must give at least one --close-strategy argument !\n";
>> - }
>>        It looks like some redundant code found its way in here, but even if you fix the redundancy I'm not sure the
>>        code is kosher.  It appears the intent is to warn the user if the default Money Management rule is installed,
>>        but the default_money_management subroutine explicitly returns nothing, so testing it yields nothing.
>>        Maybe it works in an alternative experimental branch?
>>        r709 Scripts/backtest.pl:385-398
>>  foreach (@mmname)
>>  {
>>        $pf_manager->add_money_management_rule(
>>            create_standard_object(split (/\s+/, "MoneyManagement::$_")));
>>  }
>> + my $mmbasic_added = $pf_manager->default_money_management_rule(
>> +  create_standard_object("MoneyManagement::Basic"));
>> + warn "$prog_name: warning: added mm::basic to system\n"
>> +  if $mmbasic_added && ( $verbose || $debug );
>> +
>>  $pf_manager->default_money_management_rule(
>>          create_standard_object("MoneyManagement::Basic"));
>> + warn "$prog_name: warning: added mm::basic to system\n"
>> +  if $mmbasic_added && ( $verbose || $debug );
> 
> ok these line refs match with r716
> the intent is to install the minimal moneymanager (basic) in the absence of one.
> i believe it does work as intended; refer to attached example output line 1.
> (the -rh.out attachment from ras hacked backtest.pl includes on-going diagnostics
> but gives identical results otherwise)
> 
> however, i didn't debug the actual code, just the results, but the code does
> look a bit redundant ... added to 'todo list'
>     thinking out loud without any testing ::: lines 395-398 are redundant

Let me clarify.  My nit isn't that MM:Basic won't be installed; I agree with you that it will
be installed as your output example demonstrates.  What I think doesn't work as expected is the
"warn if $mmbasic_added".

I think $mmbasic_added will always be undef.  The way default_money_management_rule
works, you can never know whether the provided MM is installed or not without knowing a priori that
no other MMs have been added.  In this particular case, it might be more feasible to test whether @mmname is
empty and if so, then install the default and issue the warning.

Does that make any sense?  I hope that makes sense.  Hmm...I'm not sure that makes sense.
> 
>>        Seems to have a similar problem to the money management code above in that set_order_factory has
>>        no explicit return value.  Thus, the implicit return value will be the value of the order factory
>>        passed in as an argument, which in this case will always be undef.  So, unless I'm missing
>>        something, this is a quite roundabout way of saying "if ( ! defined undef )", which is always
>>        true by definition.
>>        r709 Scripts/backtest.pl:400-414
>>  if ($ofname)
>>  {
>>      $sys_manager->set_order_factory(
>>          create_standard_object(split (/\s+/, "OrderFactory::$ofname")));
>>  } else {
>> -     if ( $verbose || $debug ) {
>> +     my $of = $sys_manager->set_order_factory();
>> +     if ( ! defined $of && ( $verbose || $debug ) ) {
>>        my $msg = join "", "$prog_name: notice: ",
>>         "without explicit orderfactory the implicit default is\n",
>>         "OF::MarketPrice. ",
>>         "this equates to market open price day after the order.",
>>         "\n";
>>        warn "$msg";
>>      }
>>  }
>>        The else clause was added, but is it really a good idea to force the portfolio to be stored?
>>        The description of the --store flag says nothing about a default if the option is unspecified.
>>        If it is realy important to store the portfolio, maybe it shouldn't be an option.
> 
> valid point ... added to 'todo list'

Not sure whether your valid point comment applied to the order factory issue above or the else clause issue below.
If it didn't apply to the order factory issue, then I don't think I saw any comments about it.

>     thinking out loud without any testing ::: lines 452 and 353 are not appropriate
>     but might be conditioned with $debug or something similar.
> 
>>        r709 Scripts/backtest.pl:447-451
>>  if ($store_file) {
>>      $analysis->{'portfolio'}->store($store_file);
>> + } else {
>> +     $analysis->{'portfolio'}->store("./bt_portfolio.xml");
>>  }
<snip>

Re: [GT] Problems With Rev 714 GT/BackTest.pm - Fixing The UNIX Epoch Problem

[Robert A. S. <ra...@ac...>]

Shinaberry Derek wrote:
> None of the code changes seem to have anything to do with the UNIX
> epoch problem.  I don't know whether the comment was a copy and paste
> error or whether the wrong version of GT/BackTest.pm was committed.
> 

entire message being looked into in detail. i'll post findings in due course.

as far as the GT/BackTest.pm commit, it does appear was a near complete drop-in
from my toolkit, and it was most likely committed to support the epoch date
fix for backtest.pl primarily. backtest_multi.pl and backtest_many.pl are
secondary or tertiary apps, but it would be good to keep them operational
to the extent that they are operational. GT/BackTest.pm itself should not
involve GT/Tools.pm directly.


GT/Tools.pm declares/defines method find_calculator. there is very likely
a discussion of this on the -devel archive years back. method find_calculator
would only ever be invoked in a script app like backtest.pl or graphic.pl.

as i evaluate find_calculator, it fails to return the prices queue
typically $q for which there are many methods that are useful, etc.
so i have a version of it (that is of 'find_calculator') named 'calculator'
that does return the prices queue. there may (or may not) be a
discussion of this on the -devel archive, but the trunk-head version of
GT/Tools.pm should *not* (yet) include method 'calculator' and it should
*not be used* in any of the trunk-head script apps (at least that was the intent).

find_calculator is never-the-less useful and trunk-head script apps should
make use of it where prudent and proper. the effort to complete this upgrade
is apparently incomplete and bug ridden ...

it is good that there is another set of detail oriented eyes on this ...


aloha

ras


< complete snip >

Re: [GT] Problems With Rev 715 Scripts/backtest.pl - Incorporation Of Initial Value Command Line Option

[Robert A. S. <ra...@ac...>]

Shinaberry Derek wrote:
> There are, I think, four potential problems with this revision.  Each issue is broken out below with its own snippet of code for documentation.  Thoughts or oversights on my part?
> 
> Derek
> 
>         The following code snippet was removed and while maybe not a major deal, I'm not sure it was wise to remove it.
>         Since there is no default Close Strategy installed, it probably is wise to require a close strategy when the
>         system name is supplied from the command line as there is no other source for a close strategy in this case.
> 
>         r709 Scripts/backtest.pl:263-265

aloha derek

i'm not following either your (svn) version numbers or (some of) your line number references
at least for the csname issue.

my research shows svn r709 to be related to the old website move to sourceforge.
recent changes directly related to Scripts/backtest.pl are:
716  	40d 20h  	ras2010  	/trunk/Scripts/backtest.pl  	set default timeframe to 'day', include terse cryptic sub usage that was missing

715 	135d 14h 	ras2010 	/trunk/Scripts/backtest.pl 	incorporate command line argument --initial_value to set value of initial amount of money for investment this value can also be set from configuration key "backtest::initial_value".

658 	1469d 21h 	joao 	/trunk/Scripts/backtest.pl 	Fix the max-loaded-items option in scripts




ras



i do see the deathnell message requiring at least one closestrat be part of
a trading system has been removed. this was because by default a missing
closestrat will always result in the 'do nothing' default closestrat CS:NeverClose.
(iirc this is enforced by add_position_manager).

this isn't obvious based on the data emitted by the app unless you study the
output results including the trade history chart.

here's a working minimal command line:
     ras [ 266 ] %      ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG

n.b. i've a non-standard tweaked ./backtest.pl that does a better job describing
the actual trading system being used, but it (probably) requires, in a perl sense,
other non-standard ras hacks to the gt toolkit. contact me directly for more info.

here's that same example with my hack ...
     ras [ 270 ] %     ./backtest.pl --iv=100000 --system="TFS" --broker="SelfTrade" GOOG
     backtest.pl: warning: setting default CloseStrategy to NeverClose
     .
     .

> 
> - if ($system && ! scalar(@csname)) {
> -       die "You must give at least one --close-strategy argument !\n";
> - }
> 
> 
>         It looks like some redundant code found its way in here, but even if you fix the redundancy I'm not sure the
>         code is kosher.  It appears the intent is to warn the user if the default Money Management rule is installed,
>         but the default_money_management subroutine explicitly returns nothing, so testing it yields nothing.
>         Maybe it works in an alternative experimental branch?
> 
>         r709 Scripts/backtest.pl:385-398
> 
>   foreach (@mmname)
>   {
>         $pf_manager->add_money_management_rule(
>             create_standard_object(split (/\s+/, "MoneyManagement::$_")));
>   }
> + my $mmbasic_added = $pf_manager->default_money_management_rule(
> +  create_standard_object("MoneyManagement::Basic"));
> + warn "$prog_name: warning: added mm::basic to system\n"
> +  if $mmbasic_added && ( $verbose || $debug );
> +
>   $pf_manager->default_money_management_rule(
>           create_standard_object("MoneyManagement::Basic"));
> + warn "$prog_name: warning: added mm::basic to system\n"
> +  if $mmbasic_added && ( $verbose || $debug );
> 
> 

ok these line refs match with r716
the intent is to install the minimal moneymanager (basic) in the absence of one.
i believe it does work as intended; refer to attached example output line 1.
(the -rh.out attachment from ras hacked backtest.pl includes on-going diagnostics
but gives identical results otherwise)

however, i didn't debug the actual code, just the results, but the code does
look a bit redundant ... added to 'todo list'
      thinking out loud without any testing ::: lines 395-398 are redundant

>         Seems to have a similar problem to the money management code above in that set_order_factory has
>         no explicit return value.  Thus, the implicit return value will be the value of the order factory
>         passed in as an argument, which in this case will always be undef.  So, unless I'm missing
>         something, this is a quite roundabout way of saying "if ( ! defined undef )", which is always
>         true by definition.
> 
>         r709 Scripts/backtest.pl:400-414
> 
>   if ($ofname)
>   {
>       $sys_manager->set_order_factory(
>           create_standard_object(split (/\s+/, "OrderFactory::$ofname")));
>   } else {
> -     if ( $verbose || $debug ) {
> +     my $of = $sys_manager->set_order_factory();
> +     if ( ! defined $of && ( $verbose || $debug ) ) {
>         my $msg = join "", "$prog_name: notice: ",
>          "without explicit orderfactory the implicit default is\n",
>          "OF::MarketPrice. ",
>          "this equates to market open price day after the order.",
>          "\n";
>         warn "$msg";
>       }
>   }
> 
> 
>         The else clause was added, but is it really a good idea to force the portfolio to be stored?
>         The description of the --store flag says nothing about a default if the option is unspecified.
>         If it is realy important to store the portfolio, maybe it shouldn't be an option.

valid point ... added to 'todo list'
      thinking out loud without any testing ::: lines 452 and 353 are not appropriate
      but might be conditioned with $debug or something similar.

> 
>         r709 Scripts/backtest.pl:447-451
> 
>   if ($store_file) {
>       $analysis->{'portfolio'}->store($store_file);
> + } else {
> +     $analysis->{'portfolio'}->store("./bt_portfolio.xml");
>   }
> 
> 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 

[GT] Problems With Rev 715 Scripts/backtest.pl - Incorporation Of Initial Value Command Line Option

[Shinaberry D. <gen...@sh...>]

There are, I think, four potential problems with this revision.  Each issue is broken out below with its own snippet of code for documentation.  Thoughts or oversights on my part?

Derek

        The following code snippet was removed and while maybe not a major deal, I'm not sure it was wise to remove it.
        Since there is no default Close Strategy installed, it probably is wise to require a close strategy when the
        system name is supplied from the command line as there is no other source for a close strategy in this case.

        r709 Scripts/backtest.pl:263-265

- if ($system && ! scalar(@csname)) {
-       die "You must give at least one --close-strategy argument !\n";
- }


        It looks like some redundant code found its way in here, but even if you fix the redundancy I'm not sure the
        code is kosher.  It appears the intent is to warn the user if the default Money Management rule is installed,
        but the default_money_management subroutine explicitly returns nothing, so testing it yields nothing.
        Maybe it works in an alternative experimental branch?

        r709 Scripts/backtest.pl:385-398

  foreach (@mmname)
  {
        $pf_manager->add_money_management_rule(
            create_standard_object(split (/\s+/, "MoneyManagement::$_")));
  }
+ my $mmbasic_added = $pf_manager->default_money_management_rule(
+  create_standard_object("MoneyManagement::Basic"));
+ warn "$prog_name: warning: added mm::basic to system\n"
+  if $mmbasic_added && ( $verbose || $debug );
+
  $pf_manager->default_money_management_rule(
          create_standard_object("MoneyManagement::Basic"));
+ warn "$prog_name: warning: added mm::basic to system\n"
+  if $mmbasic_added && ( $verbose || $debug );


        Seems to have a similar problem to the money management code above in that set_order_factory has
        no explicit return value.  Thus, the implicit return value will be the value of the order factory
        passed in as an argument, which in this case will always be undef.  So, unless I'm missing
        something, this is a quite roundabout way of saying "if ( ! defined undef )", which is always
        true by definition.

        r709 Scripts/backtest.pl:400-414

  if ($ofname)
  {
      $sys_manager->set_order_factory(
          create_standard_object(split (/\s+/, "OrderFactory::$ofname")));
  } else {
-     if ( $verbose || $debug ) {
+     my $of = $sys_manager->set_order_factory();
+     if ( ! defined $of && ( $verbose || $debug ) ) {
        my $msg = join "", "$prog_name: notice: ",
         "without explicit orderfactory the implicit default is\n",
         "OF::MarketPrice. ",
         "this equates to market open price day after the order.",
         "\n";
        warn "$msg";
      }
  }


        The else clause was added, but is it really a good idea to force the portfolio to be stored?
        The description of the --store flag says nothing about a default if the option is unspecified.
        If it is realy important to store the portfolio, maybe it shouldn't be an option.

        r709 Scripts/backtest.pl:447-451

  if ($store_file) {
      $analysis->{'portfolio'}->store($store_file);
+ } else {
+     $analysis->{'portfolio'}->store("./bt_portfolio.xml");
  }

[GT] Problems With Rev 714 GT/BackTest.pm - Fixing The UNIX Epoch Problem

[Shinaberry D. <gen...@sh...>]

None of the code changes seem to have anything to do with the UNIX epoch problem.  I don't know whether the comment was a copy and paste error or whether the wrong version of GT/BackTest.pm was committed.

Assuming that the intended version of BackTest.pm was, in fact, the version committed, I see the following potential issues:

        A significant number of changes, mostly to the backtest_mutil subroutine, appear to have come from some hacking that was perhaps aimed at eliminating
        the need to call find_calculator for all the codes passed into the routine.  It looks like these changes were only half baked and their inclusion has
        broken backtest_multi.  I've included a unified diff output where the FROM file is a theorhetically unbroken form of the r714 revision of my own
	creation and the TO file is the actual r714 revision in order to highlight the problems.  I've provided commentary for each of the change blocks to
	clarify what I think is a problem.

        @@ -152,7 +151,9 @@
                This is the one change not related to backtest_multi that I think is a problem.  The removal of the " - 1" changes the result from a
                percentage to a ratio, e.g. from 0.05 to 1.05.  It is unclear to me how this might ripple through the reporting code since this value is
                loaded into the analysis hash under the 'real' key and the analysis hash is returned from the backtest subroutine.

        @@ -442,10 +457,6 @@
                It's not clear to me why this code was removed as I believe that the condition that the warning note refers to still applies.

        @@ -12,7 +12,8 @@
        @@ -491,9 +502,47 @@
                The subroutine setup_backtest is unused at this point and appears to me that perhaps the changes were headed towards its use, but it doesn't
                seem ready yet and should probably be removed or, at a minimum, heavily documented as to its experimental status.

        @@ -21,8 +22,6 @@
                This use declaration is where visibility to GT::Tools::find_calculator comes from.  Assuming that I am correct about the need to restore
                the other code which uses find_calculator, then this should be restored as well.

        @@ -217,10 +218,10 @@
                The parameter list for backtest_multi has completely changed and this doesn't match the code in backtest_multi.pl that calls it anymore.
                In addition to a number of parameters being dropped, $code_ref is changed to $calc_ref and the corresponding list $codes is changed to @calcs.
                This is what drove my assumption that most of the changes are targeted at elimiinating the need to use find_calculator to get a calc
                object corresponding to a code symbol.

        @@ -245,42 +246,56 @@
                There are a lot of small sub-blocks here, but I believe the overarching theme is getting rid of the call to find_calculator.  I'm not sure
                where the intended destination for the responsibility of getting calculators is, but it seems to be upstream from backtest_multi, so it is
                probably in backtest_multi.pl.  The trouble with that (and this) is that a chunk of code that is already in find_calculator is duplicated
                here (and again in the, as yet unused, subroutine setup_backtest) in order to figure out the ranges for each code in order to determine
                the largest range, which is then the one used for the overall backtesting.  At any rate, since backtest_multi doesn't provide a list of calc
                objects and doesn't provide an argument list when calling backtest_multi that matches this vesion of the code, it simply can't work
                as intended.

--- svn-???/GT/BackTest.pm      2012-08-22 11:37:46.000000000 +0200
+++ svn-714/GT/BackTest.pm      2012-08-21 10:53:31.000000000 +0200
@@ -12,7 +12,8 @@

 require Exporter;
 @ISA = qw(Exporter);
-@EXPORT = qw(backtest_single backtest_all backtest_multi);
+@EXPORT = qw(backtest_single backtest_all backtest_multi setup_backtest);
+#@EXPORT = qw(backtest_single backtest_all backtest_multi);

 use GT::Portfolio;
 use GT::Prices;
@@ -21,8 +22,6 @@
 use GT::DateTime;
 ###l4p  use Log::Log4perl qw(:easy);
 use GT::Indicators::MaxDrawDown;
-use GT::Tools qw(:timeframe);
-

 =head1 NAME

@@ -152,7 +151,9 @@
     # Buy & hold
     my $buyhold = 0;
     $buyhold = $calc->prices->at($last)->[$LAST] /
-      $calc->prices->at($first)->[$LAST] - 1 if ( $calc->prices->at($first)->[$LAST] != 0);
+#      $calc->prices->at($first)->[$LAST] - 1
+      $calc->prices->at($first)->[$LAST]
+       if ( $calc->prices->at($first)->[$LAST] != 0);

 #print STDERR "bt: buyhold = $buyhold\n";
 #print STDERR "    opening price: ", $calc->prices->at($first)->[$LAST], "\n";
@@ -217,10 +218,10 @@


 sub backtest_multi {
-    my ($db, $pf_manager, $sys_manager_ref, $broker_ref, $code_ref, $timeframe, $full, $start, $end, $nb_item, $max_loaded_items, $init) = @_;
+    my ($pf_manager, $sys_manager_ref, $broker_ref, $calc_ref, $start, $end, $full, $init) = @_;
     my @sysmanager = @{$sys_manager_ref};
     my @brokers = @{$broker_ref};
-    my @codes = @{$code_ref};
+    my @calc = @{$calc_ref};
     $init = 10000 unless ( defined($init) );

     # Create an empty portfolio object and make the manager use it
@@ -245,42 +246,56 @@
       $cnt++;
     }

+    # Insert the calc_ojects in the portfolio..
+    foreach my $calc ( @calc ) {
+      $p->{objects}->{calc}->{$calc->code()} = $calc;
+    }
+
     # Precalc the intervals
     my $long_first = 0;
     my $long_last = 0;
     my $long_code = 0;
-    my @calc;
-    foreach my $i ( 0..$#codes ) {
+    foreach my $i ( 0..$#sysmanager ) {

-      my ($calc, $first, $last) = find_calculator($db, $codes[$i], $timeframe, $full, $start, $end, $nb_item, $max_loaded_items);
+      foreach my $j ( 0..$#calc ) {

-      $calc[$i] = $calc;
+        my $c = $calc[$j]->prices->count;
+        my $last = $c - 1;
+        my $first = $c
+         - 2 * GT::DateTime::timeframe_ratio($YEAR, $calc[$j]->current_timeframe);
+        $first = 0 if ($full);
+        $first = 0 if ($first < 0);
+        if ($start) {
+          my $date = $calc[$j]->prices->find_nearest_following_date($start);
+          $first = $calc[$j]->prices->date($date);
+        }
+        if ($end) {
+          my $date = $calc[$j]->prices->find_nearest_preceding_date($end);
+          $last = $calc[$j]->prices->date($date);
+        }

       # Set this code as reference if possible
       if ( ($last - $first) > ($long_last - $long_first) ) {
        $long_last = $last;
        $long_first = $first;
-       $long_code = $i;
+          $long_code = $j;
       }

-#     print STDERR $calc->code() . " --> " . $first . " / " . $last . "\n";
+        $sysmanager[$i]->precalculate_interval($calc[$j], $first, $last);

-      # Insert the calc_ojects in the portfolio..
-      $p->{objects}->{calc}->{$calc->code()} = $calc;
-
-      foreach my $j ( 0..$#sysmanager ) {
-
-       $sysmanager[$j]->precalculate_interval($calc, $first, $last);
+#       print STDERR $calc[$j]->code() . " --> " . $first . " / " . $last . "\n";

       }
     }

-    print STDERR "LONG-CODE:  " . $calc[$long_code]->code() . "\n";
-    print STDERR "LONG-FIRST: " . $long_first . "\n";
-    print STDERR "LONG-LAST:  " . $long_last . "\n";
+#    print STDERR "LONG-CODE:  " . $long_code . "\n";
+#    print STDERR "LONG-FIRST: " . $long_first . "\n";
+#    print STDERR "LONG-LAST:  " . $long_last . "\n";

-    print STDERR "LONG-FIRST: " . $calc[$long_code]->prices->at($long_first)->[$DATE] . "\n";
-    print STDERR "LONG-LAST:  " . $calc[$long_code]->prices->at($long_last)->[$DATE] . "\n";
+#    print STDERR "LONG-FIRST: " . $calc[$long_code]->prices->at(
+#     $long_first)->[$DATE] . "\n";
+#    print STDERR "LONG-LAST:  " . $calc[$long_code]->prices->at(
+#     $long_last)->[$DATE] . "\n";

     # Run the system
     for(my $i = $long_first; $i <= $long_last; $i++)  {
@@ -442,10 +457,6 @@
 #      $re->{'buyandhold_max_draw_down'} = $buyandhold_maxdd;
 #      $th->{'buyandhold_max_draw_down'} = $buyandhold_maxdd;
 #    }
-    ### NOTE: Need to assign something to these to avoid undef value in output
-    $re->{'buyandhold_max_draw_down'} = 0;
-    $th->{'buyandhold_max_draw_down'} = 0;
-

     # Hack to remove code reference in portfolio
     delete $p->{'date2int'};
@@ -491,9 +502,47 @@
     return $managers;
 }

-=pod
+=item C<< setup_backtest($code, $timeframe, $full, $start, $end) >>

-=back
+Setup the system backtest: Returns $calc (the calculator), as well as
+$first and $last (indices used by the calculator).

 =cut
+
+sub setup_backtest {
+  my ($code, $timeframe, $full, $start, $end) = @_;
+
+  my $db = create_db_object();
+  my $q = $db->get_prices($code);
+  #$db->disconnect;
+
+  my $calc = GT::Calculator->new($q);
+  $calc->set_code($code);
+
+  if ($timeframe) {
+    if (! $calc->set_current_timeframe(
+     GT::DateTime::name_to_timeframe($timeframe))) {
+      die "gt:backtest: error: Can't create « $timeframe » timeframe ...\n";
+    }
+  }
+
+  my $c = $calc->prices->count;
+  my $last = $c - 1;
+  my $first = $c
+   - 2 * GT::DateTime::timeframe_ratio($YEAR, $calc->current_timeframe);
+  $first = 0 if ($full);
+  $first = 0 if ($first < 0);
+  if ($start) {
+    my $date = $calc->prices->find_nearest_following_date($start);
+    $first = $calc->prices->date($date);
+  }
+  if ($end) {
+    my $date = $calc->prices->find_nearest_preceding_date($end);
+    $last = $calc->prices->date($date);
+  }
+
+  return ($calc, $first, $last, $q);
+
+}
+
 1;

Thoughts on these?

Derek

[GT] Problems With Rev 713 GT/DateTime.pm - Fixing the UNIX Epoch Problem

[Shinaberry D. <gen...@sh...>]

Problem #1 - The switch from using Time::Local::timelocal to POSIX::mktime introduces a subtle bug because timelocal does not account for timezone/daylight savings time, while mktime does.  So if you take a date during daylight savings time round trip to a time and back, the new code would result in a datetime that is an hour ahead of the original, which is not what most people would expect.

Problem #2 - The new unified map_date_to_time doesn't handle lopping off the time to the nearest boundary period prior to converting for the timeframes of minutes and hours.  So, for example, calling GT::DateTime::map_date_to_time($PERIOD_15MIN, "2012-08-21 12:27:56") results in a time value corresponding to "2012-08-21 12:27:00" rather than the expected "2012-08-21 12:15:00".

Am I missing anything in my analysis, or are these, in fact, real problems.

Cheers,
Derek

[GT] Problems With Relatively Recent (~Six Month Old) Revisions On The Trunk

[Shinaberry D. <gen...@sh...>]

Hi All,

I've been busily merging the current trunk into my workspace in advance of offering up some patches for inclusion to GT and found some changes that I think must be suspect.  I say, I think, because there is a lot I don't understand about GT and it is just as possible that I am wrong about my suspicions.  But I am sure enough about them to not merge them into my workspace until other eyes have a look at them and offer up their own analyses of the purported problems.

I'll separate the issues into multiple posts based on the repository revision that introduced them to make it easier to have discussions about the issues.  Hopefully, I haven't stepped on any toes with my commentary.  It is never my intent to offend nor to criticize other than to be constructive.

Let the discussions begin...

Derek

P.S. To RAS - I've purposely used my non-list email address to send this email in order to test that I have correctly configured my MTA to rewrite my email address on emails send to the list.  If it shows up as being from gen...@sh..., then all is well with my configuration.  If not, then I still have more work to do.

[GT] Users Repo ?

[Delix <del...@t-...>]

What do you think about something like this :
https://gitorious.org/geniustrader-users-repo

?

-- 
Delix <lin...@t-...>

Re: [GT] Experiences with manage_portfolio.pl

[Shinaberry D. <Der...@IE...>]

Hi Robert,

Thanks for the input on manage_portfolio.pl.  I'm actually working from a zip file of GeniusTrader, which I have since discovered is revision 709.  No differences to manage_portfolio.pl between those two revisions.  Although a look at the commit comments shows me that some of the changes since 709 are fixes that I have my own take on in my patched code like specifying an initial sum on the command line for the backtest* scripts and fixing the unix time epoch problem.

I'd seen references to beancounter in the GT code, but haven't had any personal experience with it.  I might have to load it and take a look around to see if it might be more amenable to modifications to suit my purposes than GT.

I completely understand your reservations concerning the law of unintended consequences and making changes to GT.  Especially in light of no tests to help you figure out if you broke something while making changes.  I'd made some substantial changes of my own and haven't looked into merging them with the trunk-head precisely because I really don't know if I broke other parts that I haven't been using.

We'll see where this quest leads me.

Cheers,
Derek

On Aug 10, 2012, at 23:36 , Robert A. Schmied wrote:

> Derek Shinaberry wrote:
>> I'm wondering if anyone out there has any experience with the manage_portfolio.pl script.  I've used the backtesting bits of GT rather extensively, but now am looking for a way to analyze my actual trading and get the same sort of information that GT provides for backtesting, e.g. W/L, PF, expectancy, etc.
>> I've started tinkering with manage_portfolio.pl to see what I could get and it seems like it is not quite ready for prime time.  Right off the bat I see it failing to get an evaluation history because it isn't looking at price history data.  It also lumps all trades for any single symbol into a single position rather than doing something more intelligent like closing a position when the quantity reaches zero and creating a new one if that symbol is encountered again.
>> It also looks like some of the reporting capabilities were never fleshed out.  The performance and analysis subcommands under the report command specifically say that they are not yet implemented.
>> Before I started any hacking in this area, I thought I would put out some feelers to see what experiences other people have had with this area of GeniusTrader.
>> Cheers,
>> Derek
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 
> aloha Derek
> 
> yea manage_portfolio.pl really could use some care and feeding.
> 
> 
> 
> my hacked version differs somewhat from the current trunk-head which i'll presume
> you are using as the app reference.  even it has no 'actual portfolio' performance
> analysis implementation.
> 
>   ras [ 230 ] %      manage_portfolio.pl  bc_pf report per
> 
>   manage_portfolio.pl: sorry performance not yet implemented
> 
> for that i use beancounter, which does have some rudimentary performance analysis:
> 
>   ras [ 221 ] %    beancounter status    .
>   .
>   .
> 
> 
> i think there are two reasons why i didn't pursue trying to improve manage_portfolio.pl
> one was i already had a methodology in place using beancounter along with some glue
> scripts that keep my gt portfolio file in-sync with my beancounter portfolio, and two
> (and maybe three) the beancounter portfolio table was far more hacker friendly than the
> xml form used by gt and almost completely undocumented except by the bits of code that
> write and read it.
> 
> combine this with a significant concern that changes to this and the supporting modules
> (GT::Analyzers) could result in unforeseen effects in backtest*.pl and anashell.pl apps
> it didn't seem like a useful endeavor.
> 
> feel free to improve it or come up with an entirely new app. some of the missing and
> rather hard to implement well aspects of stock and stock portfolio analysis include
> in no particular order:  stock splits and similar corporate actions (spin-outs, spin-ins,
> mergers, etc), dividends: regular and special and similar but different: returns of capital.
> 
> 
> ras
> 
> attach: 1) ras hack version of manage_portfolio.pl without warranty of any kind
>       2) ras file manage_portfolio_weaknesses
> #!/usr/bin/env perl
> #
> #!/usr/local/bin/perl -w
> #
> #!/usr/bin/perl -w
> 
> # Copyright 2004 Raphaël Hertzog
> # This file is distributed under the terms of the General Public License
> # version 2 or (at your option) any later version.
> 
> #
> # ras hack rcs tracking: based on version size 6621 dated May 22 2005
> # merge with version size 9637 dated Jun 03 2005
> #
> # $Id: manage_portfolio.pl,v 1.46 2010/03/08 04:50:30 ras Exp ras $
> #
> 
> use lib '..';
> 
> use strict;
> use vars qw($db);
> 
> use GT::Prices;
> use GT::Calculator;
> use GT::Report;
> use GT::Conf;
> use GT::Eval;
> use GT::DateTime;
> 
> use POSIX qw( stat strftime );
> use Getopt::Long;
> use File::Spec;
> use File::Copy qw( mv );
> use Pod::Usage;
> 
> use Cwd;
> 
> ( our $prog_name = $0 ) =~ s@^.*/@@;    # lets identify ourself
> 
> GT::Conf::load();
> 
> =head1 NAME
> 
> manage_portfolio.pl
> 
> =head1 SYNOPSIS
> 
>  ./manage_portfolio.pl <portfolio> create [<initial-sum>]
>  ./manage_portfolio.pl <portfolio> bought <quantity> <share> \
>    <price> [ <date> <source> ]
>  ./manage_portfolio.pl <portfolio> sold <quantity> <share> \
>    <price> [ <date> <source> ]
>  ./manage_portfolio.pl <portfolio> stop <share> <price>
>  ./manage_portfolio.pl <portfolio> set initial-sum <sum of money>
>  ./manage_portfolio.pl <portfolio> set broker <broker>
>  ./manage_portfolio.pl <portfolio> report { performance | positions \
>   | historic | analysis }
>  ./manage_portfolio.pl <portfolio> file <filename>
>  ./manage_portfolio.pl <portfolio> db
> 
>  ./manage_portfolio.pl <portfolio> { ls | list }
> 
> where
> <portfolio> is filename of portfolio to use. it can be a non-existent
> file, in which case it will be created
> 
> <quantity> <price> are numeric values for <share> which is the
> appropriate stock symbol or cusip or other identifier
> 
> <date> is optional, the date the transaction happened on. if not
> supplied the default value for 'today' will be supplied. preferred GT
> format for dates is 'YYYY-MM-DD'.
> 
> <source> is optional, a text string, it can be used to note the source
> of the stock transaction. typically an internal GT used field. for
> individual transactions <source> can be set via the --source='string'
> option.
> 
> <broker> name of broker module to use. see ../GT/Brokers/. there is
> no error checking. if the supplied broker module fails to exist the
> portfolio will be flawed.
> 
> =head1 DESCRIPTION
> 
> This tool lets you create or otherwise manage a gt portfolio.
> uses include creating virtual portfolios to test how a strategy works
> over time or to track an actual securities portfolio and use GeniusTrader
> to make various performance analyses on it.
> 
> The first parameter is a filename of a portfolio. The file is searched
> for in the portfolio directory (e.g. $HOME/.gt/portfolio) or in the dir
> defined by the value of gt configuration key "GT::Portfolio::Directory".
> If the portfolio file is not found in that dir the local directory will
> be tried. the portfolio file must exist unless the command is create, in
> which case the portfolio file will be created.
> 
> if the portfolio is a full or relative pathname (e.g. /tmp/pf, ./pf)
> the filename is used as is without directory searching.
> 
> the second parameter is the command to execute with respect to the
> portfolio.
> 
> =head1 COMMANDS
> 
> =over 4
> 
> =item create
> 
> creates <portfolio> and optionally set <initial-sum> available funds.
> <initial-sum> is unset if omitted
> 
> =item bought <quantity> <share> <price> [<date>]
> 
> =item sold <quantity> <share> <price> [<date>]
> 
> create a <portfolio> purchase (bought) or sale (sold) transaction
> 
> it doesn't appear to support long and short transactions but ...
> 
> =item stop <share> <price>
> 
> create a stop order for <share> at <price>
> 
> =item report { performance | positions | historic | analysis }
> 
> generate and output the specified report
> 
> =item set { initial-sum | broker } <value>
> 
> create and set the value of variables  initial-sum or broker to <value>
> 
> =item file <filename>
> 
> Specifies the name of a file which contains a list of
> 
> bought <quantity> <share> <price> [ <date> <source> ]
> 
> and
> 
> sold <quantity> <share> <price> [ <date> <source> ]
> 
> commands, one per line. This allows you to submit multiple bought/sold
> commands in a single instance.
> 
> =item db
> 
> reads beancounter database portfolio table and creates <portfolio> from it.
> 
> negative stock quantities are considered sells, <source> is derived from
> the 'type' column. currency is ignored, as GT seems to do.
> 
> since the beancounter portfolio really doesn't have necessary
> functionality to manage closed positions it is probably best to manage
> sells in the GT portfolio using command line or command file features
> provided here. in addition, by not introducing negative stock quantities
> in the beancounter portfolio table you will also avoid tickling bugs
> and messing up the report formats with the unexpected negative quantity
> value.
> 
> note: this feature is a work in process. it is really only suited
> for beancounter on postgresql and uses an external sql file
> extract_beancounter_portfolio.sql. the general approach should be suitable
> for alteration to support gt configuration key-values to control both
> the database accessor program (psql in this instance) as well as the
> sql control file (extract_beancounter_portfolio.sql).
> 
> =item list or ls
> 
> prints a list of current open positions by symbol
> 
> this is useful for example if you want to get a current list of holdings
> in your porfolio. the only option supported is --source
> 
> other options that might be of some use, but are yet to be inplemented include
> 
>  filtering the list by open date, share quantity
> 
>  listing closed positions,
> 
>  ordering the list in various ways,
> 
>  others ...
> 
> this feature might be used as follows to generate a set of charts of your
> portfolio holdings:
> 
>  %   chart.pl -g 6mon.gconf -d graphs -a port \
>    `manage_portfolio.pl pf ls --source=Zebank`
> 
> NB: if you hold a security tagged with multiple sources the security symbol
> will be duplicated for each unique source tag.
> 
> =back
> 
> =head1 OPTIONS
> 
> =over 4
> 
> =item --marged  --nomarged
> 
> Only useful for "bought" and "sold" commands. It explains that the
> corresponding positions are marged, no personal money has been used for
> them, the money has been rented.
> 
> =item --source <source>
> 
> Useful to tag certain orders as the result of a particular strategy. All
> orders passed by following the advice of someone could be tagged with
> his name and later you'll be able to make stats on the performance you
> made with his advices.
> 
> =item --template=<template file>
> 
> Output is generated using the indicated HTML::Mason component.
> For example, when using "report historic" use
> 
>  --template="portfolio_historic.mpl"
> 
> when using "report positions" use
> 
>  --template="portfolio_positions.mpl"
> 
> The template directory is defined as Template::directory in the options file.
> Each template can be predefined by including it into the options file
> For example,
> 
>  Template::manage_portfolio_positions portfolio_positions.mpl
>  Template::manage_portfolio_historic portfolio_historic.mpl
> 
> humm --- if you put Template::manage_portfolio_* in your .gt/options file
> and things don't work as expected you need to look a the perl code to see
> what is really being assigned $template = GT::Conf::get
> 
> =item --timeframe { day | week | ... }
> 
> Tell how to parse the format of the date.
> 
> =item --noconfirm --confirm 
> 
> Do not prompt for confirmation, just apply the request.
> by default changes require confirmation.
> 
> =item --detailed --nodetailed
> 
> Add extra information into the output. On by default.
> Turn off by using --nodetailed
> 
> =item --backup --nobackup
> 
> Make backup of <portfolio> if changes made and applied.
> Turn off by using --nobackup.
> backup portfolio filename will be <portfolio>.<yyyymmddhhmmss>
> where yyyymmddhhmmss is the date-time of the portfolio files'
> last modification date and time.
> 
> =item --since <date>
> 
> =item --until <date>
> 
> Those two options are used to restrict the result of a "report" command to
> a certain timeframe.
> 
> =item --option='<key>=<value>'
> 
> A configuration option (typically given in the options file) in the
> form of a key=value pair. For example,
> --option='DB::Text::format=0'
> sets the format used to parse markets via the DB::Text module to 0.
> 
> the embedded = sign in the 'key=val' pair string must be quoted (as shown)
> to escape the it from the perl options processing module.
> 
> =back
> 
> =head1 BUGS (or maybe just rough edges (in my opinion))
> 
> Needs to do a better job of checking input values for bought/sold operations
> or
> needs to provide a way of completely removing a bad entry
> 
> might also be nice to provide a couple of output modes; say one to generate
> a file that can used as input for the
>  ./manage_portfolio.pl <portfolio> file <filename>
> capability
> and one to generate a textualized version of the portfolio, if that makes sense.
> 
> =cut
> 
> # check for plea for usage
> if ( $#ARGV >= 0 && $ARGV[0] =~ /-h|-\?|-:/ ) {
>    ( defined &usage )
>     ? usage()
>     : pod2usage( -verbose => 2, -output => \*STDERR );
>    exit 0;
> }
> 
> # check for reasonable number of arguments
> if ( $#ARGV <= 0 ) {
>    print "$prog_name: both portfolio_name and command are required\n";
>    ( defined &usage )
>     ? usage()
>     : pod2usage( -verbose => 2, -output => \*STDERR );
>    exit 1;
> }
> 
> # Get all options
> my ($marged, $source, $since, $until, $confirm, $timeframe, $template, $detailed)
> = (0,       '',      '',     '',     1,        'day',      '',        1);
> my ($backup) # make makeup of existing portfolio if changes made and applied
> = (1);
> our ($verbose) # app gets noisy
> = (0);
> my $man = 0;
> my @options = ();
> GetOptions("marged!"    => \$marged,
>           "source=s"   => \$source,
>           "since=s"    => \$since,
>           "until=s"    => \$until,
>           "confirm!"   => \$confirm,
>           "timeframe"  => \$timeframe,
>           "template=s" => \$template,
>           "detailed!"  => \$detailed,
>           "backup!"    => \$backup,
>           "verbose+"   => \$verbose,
>           "option=s"   => \@options,
>           "help!"      => \$man,
>           );
> 
> $timeframe = GT::DateTime::name_to_timeframe($timeframe);
> 
> # allow user to override/set any key/value set via .gt/options on the command line
> foreach (@options) {
>    my ($key, $value) = split (/=/, $_);
>    GT::Conf::set($key, $value);
> }
> 
> pod2usage( -verbose => 2, -output => \*STDERR ) if ($man);
> 
> # Check the portfolio directory
> #GT::Conf::default("GT::Portfolio::Directory", $ENV{'HOME'} . "/.gt/portfolio");
> GT::Conf::default("GT::Portfolio::Directory", GT::Conf::_get_home_path()
> . "/.gt/portfolio");
> my $pf_dir = GT::Conf::get("GT::Portfolio::Directory");
> mkdir $pf_dir if (! -d $pf_dir);
> 
> # Load the portfolio
> my $pfname = shift || pod2usage( verbose => 2, -output => \*STDERR );
> if ($pfname !~ m@^(./|/)@) {
>    if ( -e "$pf_dir/$pfname" ) {
>        $pfname = "$pf_dir/$pfname";
>    } elsif ( -e "./$pfname" ) {
>        $pfname = "./$pfname";
>    }
> }
> 
> my $pf;
> if (-e $pfname) {
>    $pf = GT::Portfolio->create_from_file($pfname);
> } else {
>    $pf = GT::Portfolio->new();
> }
> 
> my $changes = 0;
> 
> #
> # parse and process command
> #
> my $cmd = shift || pod2usage( verbose => 2, -output => \*STDERR );
> 
> # 
> # create
> #
> if ($cmd eq "create") {
> 
>    $changes = 1;
> #    my $cash = shift || pod2usage(verbose => 2);
>    my $cash = shift;
>    if (-e $pfname) {
>        warn "A portfolio with this name already exists."
>         . " The current portfolio will be replaced!\n";
>        my $pf = GT::Portfolio->new();
>    } else {
>        print "Creation of a new portfolio in $pfname...\n";
>    }
>    if (defined($cash)) {
>        print "Setting initial sum to $cash.\n";
>        $pf->set_initial_value($cash);
>    }
> 
> # 
> # bought | buy | sold | sell | file | db
> #
> } elsif ( ($cmd eq "bought") or ($cmd eq "buy")
>       or ($cmd eq "sold") or ($cmd eq "sell")
>       or ($cmd eq "file") or ($cmd eq "db")) {
> 
>    $changes = 1;
>    my @orderlist;
>    my $batchorder;
> 
>    # 
>    # file
>    #
>    if ($cmd eq "file") {
>        my $file = shift || pod2usage(verbose => 2, -output => \*STDERR);
>        if ( ! -e $file ) {
>            my $msg = join "", "$prog_name: error: ",
>             "input file \"$file\" not found",
>             "\n";
>            warn "$msg";
>            exit 1;
>        }
>        else {
>            # there is a batch file, read it in
>            open( FILE, "<", "$file" ) || die ("Cannot open batch file $file");
>            @orderlist = <FILE>;
>            close( FILE );
>        }
> 
>    # 
>    # db
>    #
>    } elsif ($cmd eq "db") {
>      print STDERR "\nhello, this is where i'll read the beancounter"
>       .           " portfolio table database\nand create a GT equivalent."
>       .           "  thing is i'd like to be able to alter the GT"
>       .           "\nportfolio by only updating the new positions"
>       .           " not necessarily creating\na whole new GT portfolio\n\n"
>       .           "yea i thought this was the way it worked -- each subsequent"
>       .           " run adds whatever to the current GT portfolio\n"
>       .           "\n\n";
> 
>      print STDERR "in the alternative we should be able to dump this one"
>       .           " and then re-input it so we could maintain it's history\n"
>       .           "\n";
> 
>      #
>      # i made the sql statement an external file so you can easily craft
>      # one that suits your needs without having to hack this perl script.
>      #
>      # the default sql statement will select everything in your beancounter
>      # portfolio table except tuples that have a type of 'n/a'.
>      # also note on unix at least postgresql (and probably standard sql)
>      # considers case significant, so 'N/A' will not be excluded
>      #
>      my $PSQL_CMD = "/usr/pgsql/8.0.3/bin/psql";
>      my $SQL_QUERY_FILE = "./extract_beancounter_portfolio.sql";
>      #
>      open(BC_PORTFOLIO,
>       "$PSQL_CMD -q -d beancounter -f $SQL_QUERY_FILE |")
>        # the sql yields colon separated values (csv) of
>        # $quantity, $code, $price, $date, $source
>       || die "could not open input pipe from psql $!/n";
>      my @portlist = <BC_PORTFOLIO>;
>      close(BC_PORTFOLIO);
> 
>      # populate @orderlist from stuff read from db portfolio
>      # after a bit of massage and careful kneeding
>      foreach $batchorder (@portlist) {
>         # separate lines into list
>         # prepend "bought" or "sold" depending on qty (postive or negative)
>         # ensure $source present even if ""
>         my $trans;
>         my ( $quantity, $code, $price, $date, $source )
>          = split /:/, $batchorder;
>         $source = "" unless defined $source;
>         ( $quantity > 0 )
>          ? ( $trans = "bought" )
>          : ( $trans = "sold" );
>         push @orderlist, "$trans $quantity $code $price $date $source";
>      }
> 
>    # 
>    # data provided via individual argument lines
>    #
>    # { bought | buy } args
>    # { sell | sold }  args
>    #
>    } else {
>        # no batch file, place the command line parameters into the
>        # start of the data array
>        #my ($quantity, $code, $price, $date) = @ARGV;
>        my ($quantity, $code, $price, $date, $source) = @ARGV;
>        # for this one off we have to insert the $cmd (bought or sold)
>        #$orderlist[0]="$cmd $quantity $code $price $date";
>        $orderlist[0]="$cmd $quantity $code $price $date $source";
>    }
> 
>    # Creation of DB module
>    my $db = create_db_object();
>    foreach $batchorder (@orderlist) {
> 
>        chomp($batchorder);
>        next if ( $batchorder =~ m/^\s*$/ );
>        next if ( $batchorder =~ m/^\s*#/ );
> #        my ($cmd, $quantity, $code, $price, $date) = split(/ /,$batchorder);
>        my ($cmd, $quantity, $code, $price, $date, $source) =
>         split(/\s+/,$batchorder);
> 
> print STDERR "batch line\n"
> . "cmd=$cmd, quantity=$quantity, code=$code, price=$price,"
> . " date=$date, source=$source\n" if ( $verbose > 1 );
> 
>        if (! defined($date)) {
>            $date = sprintf("%04d-%02d-%02d",
>             localtime->year + 1900, localtime->mon + 1, localtime->mday);
>        }
> 
>        if ( $date !~ m@\d{4}-\d{2}-\d{2}@ ) {
>            if ( eval { require Date::Manip } ) {
>                use Date::Manip qw(ParseDate UnixDate);
>                $date = UnixDate(ParseDate($date), "%Y-%m-%d");
>            }
>        }
> #        if ( $date =~ m@/@ ) {
>        if ( $date !~ m@\d{4}-\d{2}-\d{2}@ ) {
>            die "come on -- gt only likes dates formatted yyyy-mm-dd\n"
>            .   "i'm not going accept \"$date\", nor will i convert it, sorry (not).\n";
>        }
> 
>        my $order = GT::Portfolio::Order->new;
> #        if ($cmd eq "bought") {
>        if ( $cmd =~ m/bought|buy/i ) {
>            $order->set_buy_order;
>        }
>        elsif ( $cmd =~ m/sold|sell/i ) {
>            $order->set_sell_order;
>        }
>        else {
>            die "sorry -- i don't recognize $cmd in \"$batchorder\"\n";
>        }
>        $order->set_type_limited;
>        $order->set_price($price);
>        $order->set_submission_date($date);
>        $order->set_quantity($quantity);
>        $order->set_source($source) if ($source);
> 
> #        # Creation of DB module
> #        my $db = create_db_object();
>        my $name = $db->get_name($code);
> 
>        # Look for open positions to complete
>        my $pos = find_position($pf, $code, $source, $marged);
>        if (! defined $pos) {
>            if (defined($name) && $name) {
>                print "Creating a new position ($name - $code, $source).\n";
>            } else {
>                print "Creating a new position ($code, $source).\n";
>            }
> 
>            $pos = $pf->new_position($code, $source, $date);
>            $pos->set_timeframe($timeframe);
>            # here is the only place one can make a distinction between
>            # long and short orders/positions
>            # but the only distinction(s) being considered is(are) $marged
>            if ($marged) {
>                $pos->set_marged();
>            } else {
>                $pos->set_not_marged();
>            }
>        }
>        print "Applying order: $cmd $code $quantity at $price on $date";
>        $pf->apply_order_on_position($pos, $order, $price, $date);
>        print " done.\n";
> 
>        # store the portfolio evaluation for the given date
>        print "eval portfolio for $date";
>        $pf->store_evaluation($date);
>        print " done.\n";
> 
>    }
>    # close the database connection
>    $db->disconnect;
> 
> # 
> # stop
> #
> } elsif ($cmd eq "stop") {
> 
>    unless ( -e $pfname ) {
>        usage() if defined &usage;
>        my $msg = join "", "$prog_name: error: ",
>         "portfolio file \"$pfname\" not found",
>         "\n";
>        warn "\n$msg";
>        exit 1;
>    }
> 
>    my ($code, $price) = @ARGV;
>    # Creation of DB module
>    my $db = create_db_object();
>    my $name = $db->get_name($code);
> 
>    my $pos = find_position($pf, $code, $source, $marged);
>    if (! defined($pos)) {
>        #print "No corresponding open positions found.\n";
>        my $msg = join "", "$prog_name: error: ",
>         "No open position found corresponding to args:\n",
>         "code \"$code\", source \"$source\", and marged \"$marged\".",
>         "\n";
>        warn "$msg";
>    } else {
>        $changes = 1;
>        $pos->force_stop($price);
>        if (defined($name) && $name) {
>            print "Setting the stop level of $name - $code to $price.\n";
>        } else {
>            print "Setting the stop level of $code to $price.\n";
>        }
>    }
>    # close the database connection
>    $db->disconnect;
> 
> # 
> # set
> #
> } elsif ($cmd eq "set") {
> 
>    unless ( -e $pfname ) {
>        usage() if defined &usage;
>        my $msg = join "", "$prog_name: error: ",
>         "portfolio file \"$pfname\" not found",
>         "\n";
>        warn "\n$msg";
>        exit 1;
>    }
> 
>    $changes = 1;
>    my ($var, $value) = @ARGV;
>    if ($var eq "initial-sum") {
>        print "Setting initial sum to $value.\n";
>        $pf->set_initial_value($value);
>    } elsif ($var eq "broker") {
>        print "Setting broker to $value.\n";
>        my $broker = create_standard_object("Brokers::$value");
>        $pf->set_broker($broker);
>    }
> 
> # 
> # split
> #
> } elsif ($cmd eq "split") {
> 
>    #
>    # split required arguments: code and split ratio (post:pre)
>    #
>    my ($code, $ratio) = @ARGV;
>    my $sf;
> 
>    if ( ! $code ) {
>        print "$prog_name: error: symbol required\n";
>        print "\nexample $prog_name bc_pf split POT 3:1";
>        exit 1;
>    }
>    if ( $ratio =~ /\d+\s*:\s*\d+/ ) {
>        my ($post, $pre) = split /:/, $ratio;
>        $sf = $post / $pre;
>    }
>    else {
>        print "$prog_name: error: split ratio required\n";
>        print "\nexample $prog_name bc_pf split POT 3:1";
>        exit 1;
>    }
> 
>    unless ( -e $pfname ) {
>        usage() if defined &usage;
>        my $msg = join "", "$prog_name: error: ",
>         "portfolio file \"$pfname\" not found",
>         "\n";
>        warn "\n$msg";
>        exit 1;
>    }
>    #
>    # for each position involving $code
>    #
>    for my $pos ( $pf->list_open_positions() ) {
>        if ( $pos->{'code'} eq $code ) {
>            my ( $piq, $pq, $pop, $oq, $op );
>            $changes = 1;
> 
>            $piq = $pos->{'initial_quantity'};
>            $pos->{'initial_quantity'} = $piq * $sf;
> 
>            $pq = $pos->{'quantity'};
>            $pos->{'quantity'} = $pq * $sf;
> 
>            $pop = $pos->{'open_price'};
>            $pos->{'open_price'} = sprintf("%.6f", $pop / $sf);
> 
>            if ( $verbose ) {
>                print "splitting $pos->{'code'} position:\n";
>                print "  initial quantity: was $piq, now $pos->{'initial_quantity'}\n";
>                print "  quantity: was $pq, now $pos->{'quantity'}\n";
>                print "  position price: was $pop, now $pos->{'open_price'}\n";
>            }
>            for my $order ( @{$pos->{'details'}} ) {
>                $oq = $order->{'quantity'};
>                $order->{'quantity'} = $oq * $sf;
> 
>                $op = $order->{'price'};
>                $order->{'price'} = sprintf("%.6f", $op / $sf);
> 
>                if ( $verbose ) {
>                    print "splitting $pos->{'code'} order:\n";
>                    print "  quantity: was $oq, now $order->{'quantity'}\n";
>                    print "  position price: was $op, now $order->{'price'}\n";
>                }
>            }
>        }
>    }
> 
> # 
> # report
> #
> } elsif ($cmd eq "report") {
> 
>    my ($what) = @ARGV;
>    unless ( $what ) {
>      print STDERR "need to know what to report\n"
>       . "(e.g. performance | positions | historic | analysis\n";
>      usage() if defined &usage;
>      exit 1;
>    }
> 
>    # code for templating setup
>    my $root = GT::Conf::get('Template::directory');
>    $root = File::Spec->rel2abs( cwd() ) unless defined $root;
> 
>    # reset root
>    $root .= "/Templates" if ( $root =~ m/Scripts$/o );
> 
>    unless ( -e $pfname ) {
>        usage() if defined &usage;
>        my $msg = join "", "$prog_name: error: ",
>         "portfolio file \"$pfname\" not found",
>         "\n";
>        warn "\n$msg";
>        exit 1;
>    }
> 
> # also see more extensive comments regarding templates below where
> # the attempt is made to load HTML::Mason
> 
>    # try and get the template from the config file if it has not been
>    # defined on the command line
>    # the template keyname in the config file is expected to be
>    # manage_portfolio_$what
>    # with $what designating the type of report being selected
>    # provided examples are portfolio_positions.mpl and portfolio_historic.mpl
> 
>    unless ( ! $template gt '' ) {
>        $template = GT::Conf::get("Template::manage_portfolio_$what")
>         if ($template eq '');
>    }
>    else {
>        $template = '';
> 
> #print STDERR "not using template\n";
> 
>    }
> 
>    # Creation of DB module
>    my $db = create_db_object();
> 
>    if ($template eq '') {
>        # no template is defined either on the command line
>        # or in the config file
>        # use Report.pm for standard reporting
>        if ( $what eq "performance" || $what =~ m/pe.*/i ) {
>            print "\n$prog_name: sorry performance not yet implemented\n";
>        }
>        elsif ( $what eq "positions" || $what =~ m/po.*/i ) {
>            GT::Report::OpenPositions($pf, $detailed);
>        }
>        elsif ( $what eq "historic" || $what =~ m/hi.*/i ) {
>            GT::Report::Portfolio($pf, $detailed)
>             || print "\ndidn't find any history in the portfolio\n";
>        }
>        elsif ( $what eq "analysis" || $what =~ m/an.*/i ) {
>            print "\n$prog_name: sorry analysis not yet implemented\n";
>            print "\nhumm, well, not so fast there grasshopper\n"
>             . "-- here we have hacked in a call to\n"
>             . "GT::Report::SimplePortfolioAnalysis(...)\n\n";
>            GT::Report::SimplePortfolioAnalysis($pf->real_global_analysis);
>            print "\n\nbut as you can see it don't do very much\n";
> 
>            print "\nfixme fixme fixme\n";
> 
> #
> # the tirade here (below -- and now commented out) isn't completely correct
> # if you run a backtest (i'll assume it's just a backtest not a backtest_many)
> # the output can be a portfolio of that run, and it will have completely
> # closed position history. but if you then run managage_portfolio and pass that
> # portfolio file the method GT::Report::SimplePortfolioAnalysis doesn't do very much, 
> # but there is a history in that porfolio. furthermore method GT::Report::Portfolio
> # shows the historic performance as expected, so there is something that isn't
> # complete (or isn't done correctly) when running a porfolio analysis here
> #
> # FIXME
> 
> #             print "\ntrial and error poking on this stuff suggests that\n";
> #             print "a portfolio cannot be analyzed without a history, and\n";
> #             print "you ain't got no history unless a position is closed.\n";
> #             print "seems odd to me, but that, methinks, is the way it works.\n";
> # 
> #             my ($sec, $min, $hour, $d, $m, $y, $wd, $yd) = localtime();
> #             my $today = sprintf("%04d-%02d-%02d", $y+1900, $m+1, $d);
> #             if ( $pf->has_historic_evaluation("$today") ) {
> #               GT::Report::SimplePortfolioAnalysis($pf->real_global_analysis);
> #               # following one divides by zero, without a history in portfolio
> #               GT::Report::PortfolioAnalysis($pf->real_global_analysis, $detailed);
> #             } else {
> #               print "\ndon't see no stinkin' history in your portfolio man!\n";
> #               print "you got not stinkin' history, you get no stinkin' analysis!\n";
> #               print "\n\n";
> #               print "on the other hand we could be nice and artifically close\n";
> #               print "everthing today, run the analyses and leave, but that's\n";
> #               print "just too bloody polite, if you ask me!\n";
> #               print "\nif you're interested see the bit near the comment\n";
> #               print "# Close the open positions in sub backtest_single ../GT/BackTest.pm\n";
> #               print "\n\n";
> #               
> #               print "\n\n";
> #             }
> 
> #             # lets try GT::Report::PortfolioAnalysis($pf, $detailed)
> #             print "\nwell, ok -- here we have hacked in a call to\n"
> #              . "GT::Report::PortfolioAnalysis($pf->real_global_analysis, $detailed)\n\n";
> #             GT::Report::PortfolioAnalysis($pf->real_global_analysis, $detailed);
> #             print "\n\nbut as you can see it don't work so good!\n";
> #             print "\n\n---\n\n";
> # 
> #             print "\n\n real_analysis_by_code T \n\n";
> #             GT::Report::PortfolioAnalysis($pf->real_analysis_by_code, "T");
> #             print "\n\n---\n\n";
>        }
>        else {
>            print "\n$prog_name: unknown report type \"$what\"\n";
>        }
>    }
>    else {
>        # using templates
>        #
>        # template reporting using HTML::Mason is being invoked
>        #
>        # template file sets have the extensions .mpl and .mhtml
>        #
>        # portfolio_positions.mpl, it references portfolio_positions.mhtml
>        # portfolio_historic.mpl, it references portfolio_historic.mhtml
>        #
>        # the configuration option 'Template::directory' must be set to the
>        # directory (absolute path) containing the template file(s)
>        #
>        # you cannot set this value any other way -- a defect.
>        #
>        # if 'Template::directory' is not set the default will be the
>        # current working directory, likely that will be Scripts,
>        # and that will probably be wrong.
>        #
>        # you cannot correct the template path or alter the root directory
>        # by setting the template file name relative to current working directory
>        # (e.g. --template ../Templates/portfolio_positions.mpl) or as an absolute path
>        # (e,g. --template /usr/local/genius_trader/Scripts/Templates/portfolio_historic.mpl).
>        # the examples above will not work, it'd be nice if they did, but they don't
>        #
>        # bottom lines:
>        # a) template file(s) must be in current working directory
>        #    unless configuration option 'Template::directory' is set to the
>        #    absolute pathname directory containing the template file(s)
>        #
>        # b) the path of least pain is to set 'Template::directory' to the
>        #    template directory, even if that happens to be Scripts
> 
>        my $output;
>        #my $use = 'use HTML::Mason;use File::Spec;use Cwd;';
> 
>        #eval $use;
>        #die(@!) if(@!);
> 
>        my $err = 0;
>        my $msg = "$prog_name: error: attempt to use html/mason template failed.\n";
> 
>        for my $class ( "HTML::Mason", "File::Spec", "Cwd", ) {
>            eval "require $class";
>            if ( $@ ) {
>                ++$err;
>                $msg .= "cannot load perl add-on package \"$class\".\n"
>            }
>            else {
>                eval "use $class";
>            }
>        }
> 
>        if ( $err > 1 ) {
>            $msg .= "add the perl modules listed above to your system,\n";
>            $msg .= "or don't use template to format the output\n";
> 
>            # close the database connection
>            $db->disconnect;
>            die "$msg";
>        }
>        elsif ( $err == 1 ) {
>            $msg .= "add the perl module listed above to your system,\n";
>            $msg .= "or don't use template to format the output\n";
> 
>            # close the database connection
>            $db->disconnect;
>            die "$msg";
>        }
> 
>        my $interp = HTML::Mason::Interp->new(
>         comp_root => $root,
>         out_method => \$output,
>        );
>        $template='/' . $template unless ($template =~ /\\|\//);
>        $interp->exec($template, detailed => $detailed, p => $pf, db => $db);
>        print $output;
> 
>    }
>    # close the database connection
>    $db->disconnect;
> 
> # 
> # list or ls
> #
> } elsif ($cmd eq "ls" | $cmd =~ m/list/o ) {
> 
>    unless ( -e $pfname ) {
>        usage() if defined &usage;
>        my $msg = join "", "$prog_name: error: ",
>         "portfolio file \"$pfname\" not found",
>         "\n";
>        warn "\n$msg";
>        exit 1;
>    }
> 
>    if ( defined $source && $source ) {
>        for my $pos ( $pf->list_open_positions( $source ) ) {
>            print "$pos->{'code'}\n";
>        }
>    }
>    else {
>        for my $pos ( $pf->list_open_positions() ) {
>            print "$pos->{'code'}\n";
>        }
>    }
> 
> # 
> # command string not recognized
> #
> } else {
> 
>    usage() if defined &usage;
>    print "$prog_name: error: Invalid command \"$cmd\"\n";
>    exit 1;
> }
> 
> #
> # process any changes and rewrite the portfolio file
> #
> if ($changes) {
>    local $| = 1;
>    my $is_confirmed = $confirm ? 0 : 1;
>    if ($confirm) {
>        print "\n-- Do you confirm all the above operations ? [y]n ";
>        my $ans = <STDIN>;
>        chomp($ans);
>        if ($ans =~ /^(y|)$/io) {
>            print "Applying all the planned changes !\n";
>            $is_confirmed = 1;
>        }
>        else {
>            print "Discarding all the planned changes !\n";
>            $is_confirmed = 0;
>        }
>    }
>    if ($is_confirmed) {
>        print "Storing changes to portfolio file $pfname ... ";
>        if ( $backup && -e $pfname ) {
>          print "\ncreating backup ";
>          # create backup portfolio filename
>          my $ext = strftime( "%Y%m%d%H%M%S",
>           localtime( (POSIX::stat("$pfname"))[9] )
>          );
>          if ( ( mv "$pfname", "$pfname.$ext" ) == 1 ) {
>            print "file $pfname.$ext\n";
>          }
>          else {
>            print "\noh no! the rename failed. $!\n";
>          }
>        }
>        $pf->store($pfname);
>        print "done.\n";
>    }
> }
> 
> # return 'good' status condition
> exit 0;
> 
> 
> 
> 
> sub find_position {
>    my ($pf, $code, $source, $marged) = @_;
>    my $pos;
>    foreach ($pf->get_position($code, $source)) {
>        next unless defined $_;
>        if ($marged) {
>           $pos = $_ if ($_->is_marged);
>        }
>        else {
>           $pos = $_ if (! $_->is_marged);
>        }
>    }
>    return $pos;
> }
> 
> 
> sub usage {
>  print "$prog_name <portfolio_name> <cmd> [ options ] [ arguments ]\n";
>  print "\n";
>  print "  where <portfolio_name> is the pathname of a gt portfolio file\n";
>  print "        <cmd> is one of:\n";
>  print "          create [ <initial-sum> ]\n";
>  print "          bought <args> | sold <args> | file <filename> | db\n";
> #  print "          bought <args> | sold <args> | file { <filename> | '-' } | db\n";
>  print "          report { per | pos | his | ana } \n";
>  print "          set { initial-sum | broker } <value>\n";
>  print "          stop <ticker-symbol> <stop-price>\n";
>  print "          split <ticker-symbol> <post : pre>\n";
>  print "          ls or list\n";
>  print "\n";
>  print "   <portfolio_name> has no default value and is a required argument.\n";
>  print "   the default location for portfolio files is \$HOME/.gt/portfolio\n";
>  print "   or as defined by the key-value GT::Portfolio::Directory.\n";
>  print "   if <portfolio_name> isn't found in the GT::Portfolio::Directory location\n";
>  print "   $prog_name will try the current working dir before giving up.\n";
>  print "\n";
>  print "   <cmd> has no default value and is a required argument.\n";
>  print "\n";
>  print "   db command\n";
>  print "   $prog_name attempt to use psql to read the beancounter portfolio\n";
>  print "   table.\n";
>  print "   * transactions found therein will be added to this GT portfolio.\n";
>  print "     - transactions are considered buys if qty > 0 otherwise it will be\n";
>  print "       considered a 'sell' transaction.\n";
>  print "   * this can cause duplicate entries since no processing is done to\n";
>  print "     exclude existing transactions already present in the portfolio.\n";
>  print "\n";
>  print "   NB: this will likely fail unless you are using beancounter with a\n";
>  print "       postgresql based database, or if $prog_name fails to\n";
>  print "       find the psql command.\n";
>  print "\n";
>  print "   ls or list command\n";
>  print "   list symbols of current (open) holdings, the option source is supported.\n";
>  print "   NB: if you hold a security tagged with multiple sources the security symbol\n";
>  print "       will be duplicated for each unique source tag.\n";
>  print "\n";
>  print "   file <filename> command\n";
>  print "   will cause $prog_name to read transaction records from <filename>.\n";
>  print "\n";
>  print "   transaction records are line based whitespace separated records as follows:\n";
>  print "       <'bought'|'sold'> <quantity> <symbol> <price> [<date>] [<source>]\n";
>  print "   notes: no validation or sanity checks are performed on the values\n";  
>  print "          <date> on command line over-ridden if <date> in data file\n";  
>  print "          <source> on command line over-ridden if <source> in data file\n";  
>  print "          comment or header lines are not supported\n";  
>  print "\n";
>  print "arguments include\n";
>  print "    all arguments to describe a transaction (via 'bought'|'sold' cmds)\n";
>  print "    filename of bought|sold records (via file cmd)\n";
>  print "    symbol and price used by stop cmd\n";
>  print "    filename of output reporting template (via report cmd)\n";
>  print "    cash amount to initialize a new portfolio (via create cmd)\n";
>  print "    variable ('initial-sum'|'broker') and value used by set cmd\n";
>  print "\n";
>  print "  report arguments are { performance | positions | historic | analysis }\n";
>  print "    unique abbreviations are fine (e.g. pe == performance, po == positions).\n";
>  print "    as of this writing 'performance' is a no-op,\n";
>  print "    'analysis' doesn't seem to do much, and\n";
>  print "    'historic' requires properly closed position records.\n";
>  print "\n";
>  print "options can appear anywhere on the command line. some options may not be\n";
>  print "applicable to all commands, and are silently ignored.\n";
>  print "\n";
>  print "    option                  default  description\n";
>  print "  --marged --nomarged        (off)   borrowed funds used (not used) for position\n";
>  print "  --confirm --noconfirm      (on)    ask (don't ask) for input confirmation\n";
>  print "  --detailed --nodetailed    (on)    include extra info on output\n";
>  print "  --since <date>             ('')    report since <date>\n";
>  print "  --until <date>             ('')    report until <date>\n";
>  print "  --source <source>          ('')    tag order as result of a GT strategy\n";
>  print "  --template=<template file> ('')    format per HTML::Mason template\n";
>  print "  --timeframe {day|week|...} (day)   date format\n";
>  print "  --backup --nobackup        (on)    make (don't make) backup of portfolio\n";
>  print "  --verbose                  (off)   app gets noisy\n";
>  print "\n";
>  print "  backup option -- by default any changes to the named portfolio will cause the\n";
>  print "  original portfolio to be saved as a backup. the naming scheme is inflexible\n";
>  print "  but is unlikely to clobber a prior precious backup:\n";
>  print "   the creation time stamp (yyyymmddhhmmss) is added to the filename\n";
>  print "\n";
>  print "  confirm option -- by default any changes to the named portfolio will\n";
>  print "  require user concurrance of the changes\n";
>  print "\n";
>  print "  marged option is not well supported except for individually added\n";
>  print "  positions. in addition, it isn't clear if these positions are all\n";
>  print "  long, since there seems to be no distinction in this code.\n";
>  print "\n";
>  print "  <template file> option\n";
>  print "  allows user to customize the report output and format.\n";
>  print "  you will need to have both a Templates directory and templates for\n";
>  print "  each type of report (e.g. performance, positions, ...) you create.\n";
>  print "\n";
>  print "  template files are looked for in the Scripts/Templates dir but that location\n";
>  print "  can be changed via your .gt/options file using these key-value pairs:\n";
>  print "    Template::manage_portfolio_<report_type>, where <report_type> is\n";
>  print "    one of the report arguments\n";
>  print "\n";
>  print "  NB as of now these strings must match report arguments used exactly\n";
>  print "\n";
>  print "  in addition template files can be either mpl or mhtml type files\n";
>  print "  see perldoc HTML::Mason for details\n";
>  print "\n";
>  print "for all the gory details on $prog_name use perldoc:\n";
>  print "perldoc $prog_name\n";
>  print "\n";
> 
> }
> 
> __END__
> 
> #
> # to implement a split feature of a portfolio holding
> #
> # what happens if you've got a short position and the company
> # does a split -- does the position size get split too?
> # you borrowed and sold presplit sized shares so i'd guess you'd need
> # to replace the equivalent sized shares
> #
> # 1) applies only to current orders and positions -- closed positions don't count
> # 2) GT::Portfolio::Position
> #    position initial_quantity
> #    position open_price
> #    position quantity
> #    
> # 3) GT::Portfolio::Order
> #    order price
> #    order quantity
> #    
> #
> 
> # } elsif ($cmd eq "split") {
> #   my $eval = "true";
> #   #
> #   # split requires code and split ratio (post:pre)
> #   #
> #   my ($code, $ratio) = @ARGV;
> #
> #   if ( ! $code ) {
> #     print "$prog_name: error: symbol required\n";
> #     print "\nexample $prog_name bc_pf split POT 3:1";
> #     exit 1;
> #   }
> #   if ( $ratio =~ /\d+\s*:\s*\d+/ ) {
> #     my ($post, $pre) = split /:/ $ratio;
> #     my $sf = $post / $pre;
> #   } else {
> #     print "$prog_name: error: split ratio required\n";
> #     print "\nexample $prog_name bc_pf split POT 3:1";
> #     exit 1;
> #   }
> #   #
> #   # for each position involving $code
> #   #
> #   foreach my $pos ( $p->list_open_positions() ) {
> #     if ( $pos->{'code'} eq $code ) {
> #       if ( ! $eval ) {
> #         $pos->{'initial_quantity'} = $pos->{'initial_quantity'} * $sf;
> #         $pos->{'quantity'}         = $pos->{'quantity'} * $sf;
> #         $pos->{'open_price'}       = sprintf "%.6f", $pos->{'open_price'} / $sf;
> #         foreach my $ord ( @{$pos->{'details'}} ) {
> #           $ord->{'quantity'} = $ord->{'quantity'} * $sf;
> #           $ord->{'price'}    = sprintf "%.6f", $ord->{'price'} / $sf;
> #         }
> #       } else {
> #         print "code = $pos->{'code'}\n";
> #         print "iqty = $pos->{'initial_quantity'} * $sf, ";
> #         print "qty  = $pos->{'quantity'} * $sf, ";
> #         print "oprc = $pos->{'open_price'} / $sf\n";
> #         foreach my $ord ( @{$pos->{'details'}} ) {
> #           print "qty   = $ord->{'quantity'} * $sf, ";
> #           print "price = sprintf "%.6f", $ord->{'price'} / $sf\n";
> #         }
> #         print "\n";
> #       }
> #     }
> #   }
> manage_portfolio.pl  weaknesses
> 
> well the gt portfolio file doesn't appear to be valid xml
> if that is even what it is supposed to be:
> 
>  %     /usr/local/bin/xmlwf bc_pf
>  bc_pf:1:0: not well-formed (invalid token)
> 
> and
> 
>  %    /usr/local/bin/xmllint --noout --valid bc_pf
>  bc_pf:1: validity error : Validation failed: no DTD found !
>  <hashref type="GT::Portfolio"><scalar value="666.66" key="cash"/><array
>                               ^
> 
> i dunno -- there does not seem to be a readily available simple fix
> for this problem
> 
> 
> 
> 
> this is the way to ''look-at'' a gt portfolio file
> /usr/local/bin/xmllint --format --recover 'bt_portfolio' > 'bt_portfolio.xml'
> 
> --recover is needed since the file isn't really 'valid' xml
> as there isn't a corresponding dtd and there isn't an xml version line
> 
> --format indents and newlines the file ...
> 
>    it also probably adds this line at the top of the output
> <?xml version="1.0"?>
> 
> 
> 
> 
> should have a default for the portfolio either/both via
> a configuration key-value or an shell envvar
> 
> should be able to specify a symbol with the report command
> to filter the output to just that code
> 
> then it should allow multiples, then is should allow the
> complement (report all but the one(s) listed)
> 
> should have a mode where one could edit/change/delete an existing transaction
> 
> when input a split there should be at least an optional date
> to specify when the split takes effect
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Re: [GT] Experiences with manage_portfolio.pl

[Robert A. S. <ra...@ac...>]

Derek Shinaberry wrote:
> I'm wondering if anyone out there has any experience with the manage_portfolio.pl script.  I've used the backtesting bits of GT rather extensively, but now am looking for a way to analyze my actual trading and get the same sort of information that GT provides for backtesting, e.g. W/L, PF, expectancy, etc.
> 
> I've started tinkering with manage_portfolio.pl to see what I could get and it seems like it is not quite ready for prime time.  Right off the bat I see it failing to get an evaluation history because it isn't looking at price history data.  It also lumps all trades for any single symbol into a single position rather than doing something more intelligent like closing a position when the quantity reaches zero and creating a new one if that symbol is encountered again.
> 
> It also looks like some of the reporting capabilities were never fleshed out.  The performance and analysis subcommands under the report command specifically say that they are not yet implemented.
> 
> Before I started any hacking in this area, I thought I would put out some feelers to see what experiences other people have had with this area of GeniusTrader.
> 
> Cheers,
> Derek
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 

aloha Derek

yea manage_portfolio.pl really could use some care and feeding.



my hacked version differs somewhat from the current trunk-head which i'll presume
you are using as the app reference.  even it has no 'actual portfolio' performance
analysis implementation.

    ras [ 230 ] %      manage_portfolio.pl  bc_pf report per

    manage_portfolio.pl: sorry performance not yet implemented

for that i use beancounter, which does have some rudimentary performance analysis:

    ras [ 221 ] %    beancounter status 
    .
    .
    .


i think there are two reasons why i didn't pursue trying to improve manage_portfolio.pl
one was i already had a methodology in place using beancounter along with some glue
scripts that keep my gt portfolio file in-sync with my beancounter portfolio, and two
(and maybe three) the beancounter portfolio table was far more hacker friendly than the
xml form used by gt and almost completely undocumented except by the bits of code that
write and read it.

combine this with a significant concern that changes to this and the supporting modules
(GT::Analyzers) could result in unforeseen effects in backtest*.pl and anashell.pl apps
it didn't seem like a useful endeavor.

feel free to improve it or come up with an entirely new app. some of the missing and
rather hard to implement well aspects of stock and stock portfolio analysis include
in no particular order:  stock splits and similar corporate actions (spin-outs, spin-ins,
mergers, etc), dividends: regular and special and similar but different: returns of capital.


ras

attach: 1) ras hack version of manage_portfolio.pl without warranty of any kind
        2) ras file manage_portfolio_weaknesses

[GT] Experiences with manage_portfolio.pl

[Derek S. <gen...@sh...>]

I'm wondering if anyone out there has any experience with the manage_portfolio.pl script.  I've used the backtesting bits of GT rather extensively, but now am looking for a way to analyze my actual trading and get the same sort of information that GT provides for backtesting, e.g. W/L, PF, expectancy, etc.

I've started tinkering with manage_portfolio.pl to see what I could get and it seems like it is not quite ready for prime time.  Right off the bat I see it failing to get an evaluation history because it isn't looking at price history data.  It also lumps all trades for any single symbol into a single position rather than doing something more intelligent like closing a position when the quantity reaches zero and creating a new one if that symbol is encountered again.

It also looks like some of the reporting capabilities were never fleshed out.  The performance and analysis subcommands under the report command specifically say that they are not yet implemented.

Before I started any hacking in this area, I thought I would put out some feelers to see what experiences other people have had with this area of GeniusTrader.

Cheers,
Derek

Re: [GT] CloseStrategy and OrderFactory

[Robert A. S. <ra...@ac...>]

Delix wrote:
> On Fri, 20 Jul 2012 18:45:18 -0700
> "Robert A. Schmied" <ra...@ac...> hatte geschrieben :
> 
> 
>>what is the trading system you are using to evaluate this closestrat?
> 
> 
>>ok -- just to get other readers that may not have the time to untangle
>>these aliases for themselves ( *correct* me if i've confused myself )
>>
>>two signals aliases are defined either in .gt/options or .gt/aliases/signals
>>(see perldoc -t ../GT/Docs/object_aliases.pod for gory details)
>>
>># @S:H-S_Schritt_ja
>>Aliases::Signals::H-S_Schritt_ja	{ S:G:Above \
>>                {I:Prices CLOSE} { I:G:Eval {I:Prices HIGH} - #1 } \
>>                                        }
>>
>># @S:H-S_Schritt_nein
>>Aliases::Signals::H-S_Schritt_nein	{ S:G:Below \
>>                {I:Prices CLOSE} { I:G:Eval {I:Prices HIGH} - #1 } \
>>                                        }
>>
>>one system alias (.gt/options or .gt/aliases/systems)
>># @SY:H-S_pur
>>Aliases::Systems::H-S_pur	{ SY:G \
>>                                     { @S:H-S_Schritt_nein #1 } \
>>                                     { @S:H-S_Schritt_ja   #1 } \
>>                                }
>>
>>and one global alias as the trading system (in .gt/options)
>># the overall trading system
>>Aliases::Global::H-S	@SY:H-S_pur 0.15 0.15    \
>>                         | TF:ShortOnly          \
>>                         | OF:SignalClosingPrice \
>>                         | CS:LimitPeriodInTheMarket 1 {S:G:True} {S:G:True}
>>
>>and a backtest.pl example:
>>
>>     %     ./backtest.pl -graph=/tmp/bt_trades.png -display-trades -html \
>>     H-S 13000 > /tmp/bt_trades.html
>>
>>the html output (sent to stdout) will automagically reference the generated
>>chart and the winning/losing trades will be highlighted ...
> 
> 
> I checked the signal and the system with display_signal/system
> and it seems to be correct. And I get a readable result from backtest
> (althogh it is a 100% looser system  :-)   ).
> However, when I checked the output I saw the trade entry is always on
> the next days OPEN prices. Therefore I tried to figure out what the
> SignalClosingPrice should do -- actually, I still guess it can't use
> CLOSE prices as long as the code is not modified for that.

yes, that's the basic operating mode. flag an opportunity to open a position,
(that's the SYstem part), create an order (OrderFactory), pass the orders thru
one or more TradeFilters and finally the PortfolioManager processes the order.
as far as i know, all orders are filled on the next market period, and
usually at the open price. i suppose some sort of a limit order could be created
at a price other than the open but i've got no specifics on it.
   maybe perldoc -t ../GT/OrderFactory/ClosedToClose.pm

i'd expect such to be in either OrderFactory or in a TradeFilter. but as
far as i know the order cannot be considered in the same time-period it
originates from. makes sense in a non-realtime processing context like gt,
but at the same time better trades might be implemented if, instead of
always opening a new position 'at the open', a one-day limit order be
created to see if a better price can be achieved. looks like there are
methods for *_limited_price in PortfolioManager, but i do not see a
OF or TF that uses them.

> 
> 
>>>>as for specifics, the use of a percentage of the position opening price
>>>>instead of a fixed value is probably intentional, for instance a 1 unit
>>>>price change for a stock that trades below 10 units is a large change,
>>>>conversely for a stock like goog or aapl a 1 unit change is likely to
>>>>occur every market day.
>>>>
>>>
>>>True, but the percentage is not useful to backtest bonds
>>> 
>>
>>i'll take your word for it, since i don't follow bond prices.
>>can you explain why?
> 
> 
> Bonds are usually not traded directly, but via futures or derivates on
> futures. The range over a full year is often below a few percent (let's
> say 5). The daily/weekly changes are usually below 1 percent. 
> So many traders don't use percentages but units as their "mental price
> markers".
> 
> 
>>also take note that there are no provisions in gt to factor
>>dividends in any of the gain/loss calculations. possibly a large
>>error factor when dealing with long term holdings and or bonds.
> 
> 
> true, but doesn't effect the trading with futures or their derivates.
> 
> And don't worry :
> I'm not going to mail every single problem I'm running into. Guess. I
> snapped the basics of GT now.

no problem

> Thanks for all your help.
>  

you are welcome

aloha

ras

Re: [GT] CloseStrategy and OrderFactory

[Delix <del...@t-...>]

On Fri, 20 Jul 2012 18:45:18 -0700
"Robert A. Schmied" <ra...@ac...> hatte geschrieben :

>what is the trading system you are using to evaluate this closestrat?

> ok -- just to get other readers that may not have the time to untangle
> these aliases for themselves ( *correct* me if i've confused myself )
> 
> two signals aliases are defined either in .gt/options or .gt/aliases/signals
> (see perldoc -t ../GT/Docs/object_aliases.pod for gory details)
> 
> # @S:H-S_Schritt_ja
> Aliases::Signals::H-S_Schritt_ja	{ S:G:Above \
>                 {I:Prices CLOSE} { I:G:Eval {I:Prices HIGH} - #1 } \
>                                         }
> 
> # @S:H-S_Schritt_nein
> Aliases::Signals::H-S_Schritt_nein	{ S:G:Below \
>                 {I:Prices CLOSE} { I:G:Eval {I:Prices HIGH} - #1 } \
>                                         }
> 
> one system alias (.gt/options or .gt/aliases/systems)
> # @SY:H-S_pur
> Aliases::Systems::H-S_pur	{ SY:G \
>                                      { @S:H-S_Schritt_nein #1 } \
>                                      { @S:H-S_Schritt_ja   #1 } \
>                                 }
> 
> and one global alias as the trading system (in .gt/options)
> # the overall trading system
> Aliases::Global::H-S	@SY:H-S_pur 0.15 0.15    \
>                          | TF:ShortOnly          \
>                          | OF:SignalClosingPrice \
>                          | CS:LimitPeriodInTheMarket 1 {S:G:True} {S:G:True}
> 
> and a backtest.pl example:
> 
>      %     ./backtest.pl -graph=/tmp/bt_trades.png -display-trades -html \
>      H-S 13000 > /tmp/bt_trades.html
> 
> the html output (sent to stdout) will automagically reference the generated
> chart and the winning/losing trades will be highlighted ...

I checked the signal and the system with display_signal/system
and it seems to be correct. And I get a readable result from backtest
(althogh it is a 100% looser system  :-)   ).
However, when I checked the output I saw the trade entry is always on
the next days OPEN prices. Therefore I tried to figure out what the
SignalClosingPrice should do -- actually, I still guess it can't use
CLOSE prices as long as the code is not modified for that.

> >>as for specifics, the use of a percentage of the position opening price
> >>instead of a fixed value is probably intentional, for instance a 1 unit
> >>price change for a stock that trades below 10 units is a large change,
> >>conversely for a stock like goog or aapl a 1 unit change is likely to
> >>occur every market day.
> >>
> > True, but the percentage is not useful to backtest bonds
> >  
> 
> i'll take your word for it, since i don't follow bond prices.
> can you explain why?

Bonds are usually not traded directly, but via futures or derivates on
futures. The range over a full year is often below a few percent (let's
say 5). The daily/weekly changes are usually below 1 percent. 
So many traders don't use percentages but units as their "mental price
markers".

> also take note that there are no provisions in gt to factor
> dividends in any of the gain/loss calculations. possibly a large
> error factor when dealing with long term holdings and or bonds.

true, but doesn't effect the trading with futures or their derivates.

And don't worry :
I'm not going to mail every single problem I'm running into. Guess. I
snapped the basics of GT now.
Thanks for all your help.
 
-- 
Delix <lin...@t-...>

Re: [GT] CloseStrategy and OrderFactory

[Robert A. S. <ra...@ac...>]

Delix wrote:
> On Fri, 20 Jul 2012 10:30:28 -0700
> "Robert A. Schmied" <ra...@ac...> hatte geschrieben :
> 
> 
>>Delix wrote:
>>

< snip >

>>
>>what is the trading system you are using to evaluate this closestrat?
> 
> 
> The @S:
> H-S_Schritt_ja  { S:G:Above {I:Prices Close} {I:G:Eval {I:Prices
> High}-#1} } H-S_Schritt_nein { S:G:Below {I:Prices Close} {I:G:Eval
> {I:Prices High} - #1} }
> 
> The @SY:
> H-S_pur SY:Generic {@S:H-S_Schritt_nein #1} {@S:H-S_Schritt_ja #1}
---------^

please note this from ../GT/Docs/object_aliases.pod

  some important notes:

    1   it's critical that the leading '{' is present in each object alias
        otherwise the arguments don't get fully substituted

> 
> The trading system:
> Aliases::Global::H-S  @SY:H-S_pur 0.15 0.15 | TF:ShortOnly | \
> OF:SignalClosingPrice | \ CS:LimitPeriodInTheMarket 1 {S:G:True}
> {S:G:True} 
> 

ok -- just to get other readers that may not have the time to untangle
these aliases for themselves ( *correct* me if i've confused myself )

two signals aliases are defined either in .gt/options or .gt/aliases/signals
(see perldoc -t ../GT/Docs/object_aliases.pod for gory details)

# @S:H-S_Schritt_ja
Aliases::Signals::H-S_Schritt_ja	{ S:G:Above \
                {I:Prices CLOSE} { I:G:Eval {I:Prices HIGH} - #1 } \
                                        }

# @S:H-S_Schritt_nein
Aliases::Signals::H-S_Schritt_nein	{ S:G:Below \
                {I:Prices CLOSE} { I:G:Eval {I:Prices HIGH} - #1 } \
                                        }

one system alias (.gt/options or .gt/aliases/systems)
# @SY:H-S_pur
Aliases::Systems::H-S_pur	{ SY:G \
                                     { @S:H-S_Schritt_nein #1 } \
                                     { @S:H-S_Schritt_ja   #1 } \
                                }

and one global alias as the trading system (in .gt/options)
# the overall trading system
Aliases::Global::H-S	@SY:H-S_pur 0.15 0.15    \
                         | TF:ShortOnly          \
                         | OF:SignalClosingPrice \
                         | CS:LimitPeriodInTheMarket 1 {S:G:True} {S:G:True}

and a backtest.pl example:

     %     ./backtest.pl -graph=/tmp/bt_trades.png -display-trades -html \
     H-S 13000 > /tmp/bt_trades.html

the html output (sent to stdout) will automagically reference the generated
chart and the winning/losing trades will be highlighted ...


<snip>

>>as for specifics, the use of a percentage of the position opening price
>>instead of a fixed value is probably intentional, for instance a 1 unit
>>price change for a stock that trades below 10 units is a large change,
>>conversely for a stock like goog or aapl a 1 unit change is likely to
>>occur every market day.
>>
> 
> 
> True, but the percentage is not useful to backtest bonds
>  

i'll take your word for it, since i don't follow bond prices.
can you explain why?

also take note that there are no provisions in gt to factor
dividends in any of the gain/loss calculations. possibly a large
error factor when dealing with long term holdings and or bonds.



aloha

ras


<snip>

Re: [GT] CloseStrategy and OrderFactory

[Delix <del...@t-...>]

On Fri, 20 Jul 2012 10:30:28 -0700
"Robert A. Schmied" <ra...@ac...> hatte geschrieben :

> Delix wrote:
> > Hi,
> 
> aloha delix
> 
> glad to hear you've got a working system. that's great news.
> 
> on the assumption that you've got fairly recent versions of a lot
> of key parts, such as perl, were there any particular problems
> you encountered that forced you to install older versions of something?
> 

I started with Perl v5.10.1.
Most of the perl packages I had to build anyway as only some basic
moduls are included in the ConnochaetOS repo.
https://gitorious.org/cur/i586/trees/master
includes the working packages with their version numbers.
Params-Validate had to be build with Module::Build v0.36 , but it didn't
build with the perl-5.10.1 version (included in the core package) 
as it was v0.34.... That was the challange for me......

> > after the successful installation,I'm playing around with the backtest
> > facilities to get familiar with GT.
> > 
> > My first attempt was the modification of the CloseGain.pm to
> > CloseFixGain.pm :
> 
> i dunno, most users tend to monkey around with trading systems
> definitions and fancy signals before starting to hack on the
> code but whatever floats your boat ...

Some very simple signals and systems I created already, so I wanted to
take the next step with the trading system. The first attempts with @SY
|@TF | @CS were successful and so I went further to create @OF and @MM aliases.

> 
> > 
> 
> <snip>
> 
> > I think, it should be okay this way.
> > 
> > The second attempt was the usage of the SignalClosingPrice.pm .
> > Maybe I didn't understand it right, but I guess the
> >                           virtual_buy/sell_at_signal
> > should be replaced by
> >                           virtual_buy/sell_at_close
> > 
> > if it is intended for closing prices.
> 
> what is the trading system you are using to evaluate this closestrat?

The @S:
H-S_Schritt_ja  { S:G:Above {I:Prices Close} {I:G:Eval {I:Prices
High}-#1} } H-S_Schritt_nein { S:G:Below {I:Prices Close} {I:G:Eval
{I:Prices High} - #1} }

The @SY:
H-S_pur SY:Generic {@S:H-S_Schritt_nein #1} {@S:H-S_Schritt_ja #1}

The trading system:
Aliases::Global::H-S  @SY:H-S_pur 0.15 0.15 | TF:ShortOnly | \
OF:SignalClosingPrice | \ CS:LimitPeriodInTheMarket 1 {S:G:True}
{S:G:True} 

> 
> 
> best guidance i can give is for you to review the functional diagrams
> (http://geniustrader.org/development.html) carefully. note that only
> one (1) orderfactory is allowed (verify but that's my understanding).
> 
> further, note that orderfactory only is involved when a position
> is initiated, thereafter it is *NO* longer involved (again my understanding
> validation encouraged) for orders that relate to altering an existing
> positions' holding qty. that's what closestrategy(ies) is(are) for.
> you can have many closestrategy and they are applied first .. last
> in the order they are defined in the 'trading system'.
> 
> as for specifics, the use of a percentage of the position opening price
> instead of a fixed value is probably intentional, for instance a 1 unit
> price change for a stock that trades below 10 units is a large change,
> conversely for a stock like goog or aapl a 1 unit change is likely to
> occur every market day.
>

True, but the percentage is not useful to backtest bonds
 
> you need to correct (possibly i did not run any test cases) a few things
> here are hackouts from my diff listing so remove the leading char (-+).
> 
> -@NAMES = ("CloseGain[#1]");
> +@NAMES = ("CloseFixGain[#1]");
> 
> -=head1 GT::CloseStrategy::CloseGain
> +=head1 GT::CloseStrategy::CloseFixGain
> 
> in pod i would include:
> 
> +nb: the target reference price used is opening price on the day the position
> +is entered. there is no way to change this price reference.
> +
> +refer to $position->open_price in methods *_position_opened
> 
> no need for the return statements in sub *_position_opened methods. perl
> will automatically return at end if sub the value is the last expression evaluated.
> 
> no need for the sub manage_*_position methods. they do nothing and the
> GT::CloseStrategy class provides generic ones.
> 
> need to terminate the package with something that evaluates to perl true like
> +1;
> 

Great - thanks for the hints !

> 
> ras
> 
> 
> 
> 
> > Please, can any body check this and explain these code line to me ?
> > Thanks, delix 
> > 
> 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/


-- 
Delix <lin...@t-...>

Re: [GT] CloseStrategy and OrderFactory

[Robert A. S. <ra...@ac...>]

Delix wrote:
> Hi,

aloha delix

glad to hear you've got a working system. that's great news.

on the assumption that you've got fairly recent versions of a lot
of key parts, such as perl, were there any particular problems
you encountered that forced you to install older versions of something?

> after the successful installation,I'm playing around with the backtest
> facilities to get familiar with GT.
> 
> My first attempt was the modification of the CloseGain.pm to
> CloseFixGain.pm :

i dunno, most users tend to monkey around with trading systems
definitions and fancy signals before starting to hack on the
code but whatever floats your boat ...

> 

<snip>

> I think, it should be okay this way.
> 
> The second attempt was the usage of the SignalClosingPrice.pm .
> Maybe I didn't understand it right, but I guess the
>                           virtual_buy/sell_at_signal
> should be replaced by
>                           virtual_buy/sell_at_close
> 
> if it is intended for closing prices.

what is the trading system you are using to evaluate this closestrat?


best guidance i can give is for you to review the functional diagrams
(http://geniustrader.org/development.html) carefully. note that only
one (1) orderfactory is allowed (verify but that's my understanding).

further, note that orderfactory only is involved when a position
is initiated, thereafter it is *NO* longer involved (again my understanding
validation encouraged) for orders that relate to altering an existing
positions' holding qty. that's what closestrategy(ies) is(are) for.
you can have many closestrategy and they are applied first .. last
in the order they are defined in the 'trading system'.

as for specifics, the use of a percentage of the position opening price
instead of a fixed value is probably intentional, for instance a 1 unit
price change for a stock that trades below 10 units is a large change,
conversely for a stock like goog or aapl a 1 unit change is likely to
occur every market day.

you need to correct (possibly i did not run any test cases) a few things
here are hackouts from my diff listing so remove the leading char (-+).

-@NAMES = ("CloseGain[#1]");
+@NAMES = ("CloseFixGain[#1]");

-=head1 GT::CloseStrategy::CloseGain
+=head1 GT::CloseStrategy::CloseFixGain

in pod i would include:

+nb: the target reference price used is opening price on the day the position
+is entered. there is no way to change this price reference.
+
+refer to $position->open_price in methods *_position_opened

no need for the return statements in sub *_position_opened methods. perl
will automatically return at end if sub the value is the last expression evaluated.

no need for the sub manage_*_position methods. they do nothing and the
GT::CloseStrategy class provides generic ones.

need to terminate the package with something that evaluates to perl true like
+1;


ras




> Please, can any body check this and explain these code line to me ?
> Thanks, delix 
> 

[GT] CloseStrategy and OrderFactory

[Delix <del...@t-...>]

Hi,
after the successful installation,I'm playing around with the backtest
facilities to get familiar with GT.

My first attempt was the modification of the CloseGain.pm to
CloseFixGain.pm :

[code]
package GT::CloseStrategy::CloseFixGain;

# Copyright 2000-2002 Raphaël Hertzog, Fabien Fulhaber
# Modified for fix gains by Delix 2012
# This file is distributed under the terms of the General Public License
# version 2 or (at your option) any later version.

use strict;
use vars qw(@ISA @NAMES);

use GT::CloseStrategy;

@ISA = qw(GT::CloseStrategy);
@NAMES = ("CloseGain[#1]");

=head1 GT::CloseStrategy::CloseGain

This strategy closes the position once the prices have crossed a
limit called target. This target is defined as a fixed gain from the
initial price. By default, it's defined as +1.  If you use it together
with PartialGain, this strategy will only close the remaining shares to
be sold/bought. Take care however to place the CloseGain strategy after
the PartialGain strategy.

=cut
sub new {
    my $type = shift;
    my $class = ref($type) || $type;
    my $args = shift;

    my $self = { "args" => defined($args) ? $args : [ 1 ] };

    $self->{'args'}->[0] = 1 if (! defined($self->{'args'}->[0]));

    return manage_object(\@NAMES, $self, $class, $self->{'args'}, "");
}

sub initialize {
    my ($self) = @_;
    $self->{'long_factor'} = $self->{'args'}[0];
    $self->{'short_factor'} = 0 - $self->{'args'}[0];
}

sub long_position_opened {
    my ($self, $calc, $i, $position, $pf_manager, $sys_manager) = @_;

    # Check for other "target" orders
    my $quantity = $position->quantity;
    foreach ($position->list_pending_orders)
    {
	if ((! $_->discardable) && $_->is_sell_order &&
$_->is_type_limited) {
	    $quantity -= $_->quantity;
	}
    }

    my $order = $pf_manager->sell_limited_price($calc,
$position->source, $position->open_price + $self->{'long_factor'});
    $order->set_not_discardable;
    $order->set_quantity($quantity);
    $pf_manager->submit_order_in_position($position, $order, $i, $calc);
    $position->set_no_intent_to_close;

    return;
}

sub short_position_opened {
    my ($self, $calc, $i, $position, $pf_manager, $sys_manager) = @_;
    
    # Check for other "target" orders
    my $quantity = $position->quantity;
    foreach ($position->list_pending_orders)
    {
	if ((! $_->discardable) && $_->is_buy_order &&
$_->is_type_limited) {
	    $quantity -= $_->quantity;
	}
    }
    
    my $order = $pf_manager->buy_limited_price($calc, $position->source,
			$position->open_price + $self->
{'short_factor'}); $order->set_not_discardable;
    $order->set_quantity($quantity);
    $pf_manager->submit_order_in_position($position, $order, $i, $calc);
    $position->set_no_intent_to_close;

    return;
}

sub manage_long_position {
    my ($self, $calc, $i, $position, $pf_manager, $sys_manager) = @_;
    
    return;
}

sub manage_short_position {
    my ($self, $calc, $i, $position, $pf_manager, $sys_manager) = @_;
   
    return;
}

[/code]

I think, it should be okay this way.

The second attempt was the usage of the SignalClosingPrice.pm .
Maybe I didn't understand it right, but I guess the
                          virtual_buy/sell_at_signal
should be replaced by
                          virtual_buy/sell_at_close

if it is intended for closing prices.
Please, can any body check this and explain these code line to me ?
Thanks, delix 

-- 
Delix <lin...@t-...>

Re: [GT] patch for backtest.pl

[Delix <del...@t-...>]

On Thu, 12 Jul 2012 09:25:39 -0700
"Robert A. Schmied" <ra...@ac...> hatte geschrieben :

> >>give me a shout if this also affects the "The directory ' ' doesn't exist"
> >>problem in any way.
> >>
> >>i'm still perplexed about this error. that directory isn't even used
> >>unless --set is set, the trade history file is always written to ./bt_portfolio.xml
> >>unless --store is defined and then it's still always likely cwd
> >>(but there could be a key elsewhere that alters that dir).

Over the weekend I tried to solve the problem on the Debian machine by
reinstalling all the lib and perl packages again order by (my guess of
the) dependencies and occurrence. However, with no success. 

> beancounters' Finance::YahooQuote probably represents the most significant
> dependency addition over what GT requires. yell if there is anything you
> run in to that you cannot resolve. as far as i know, if your system has
> both gcc and perl you should be able to build any perl module from it's
> source package and not need to rely on pre-packaged packages. the risk
> is knowing where to install stuff ...

On the small machine I compiled and installed all the
dependencies, beancounter and GT -- with success !
It passed all the tests I performed so far.

If somebody wants to fetch the PKGBUILD files (there is one for
Beancounter now, too) for ArchLinux-based distros :
they canbe found at the CUR project on the Gitorious.org server.

-- 
Delix <lin...@t-...>

Re: [GT] patch for backtest.pl

[Robert A. S. <ra...@ac...>]

Delix wrote:
> On Wed, 11 Jul 2012 21:00:35 -0700
> "Robert A. Schmied" <ra...@ac...> hatte geschrieben :
> 
> 
>>aloha delix
>>
>>append the following terse usage subroutine to the backtest.pl file.
>>
>>i don't think it's going to resolve the   "The directory ' ' doesn't exist"
>>error, but it might just.
>>
>>you can do a before and after eval using this command:
>>
>>     $  ./backtest.pl --man
>>
>>before the fix it should error out something like:
>>     $  ./backtest.pl --man
>>Undefined subroutine &main::usage called at ./backtest.pl line 304.
>>
>>after the fix
>>
>>     $  ./backtest.pl --man
>>     ./backtest.pl [ options ] <code>
>>     ./backtest.pl [ options ] <system_alias> <code>
>>     ./backtest.pl [ options ] "<full_system_name>" <code>
>>     options: -full -verbose -html -dt
>>     opt w arg: -te fn, -mm s, -tf s, -of s, -cs s, -b s, -sy s, -sto fn, -gr fn,
>>                -mli n, -od dn, -set s, -iv n
>>     typical opts: -start date, -end date, -timeframe tfname
>>
>>give me a shout if this also affects the "The directory ' ' doesn't exist"
>>problem in any way.
>>
>>i'm still perplexed about this error. that directory isn't even used
>>unless --set is set, the trade history file is always written to ./bt_portfolio.xml
>>unless --store is defined and then it's still always likely cwd
>>(but there could be a key elsewhere that alters that dir).
> 
> 
> It will take some time, 'til I can do some corrections in the code. As
> I wrote I'm a novice not only to GT but also to Perl. As this is a good
> way to get more familiar with Perl, I'll try to understand the code
> before I'll insert the modifications. 
> 
> 
> 
>>
>>
>>
>>and if you want the pending fix for the *no timeframe default* you
>>can change line around 263
> 
> 
> 
> Yesterday I played around with the GT installation on the small machine.
> Actually, I guess the problems don't arise from GT or Beancounter,
> but from the Perl modules they depend on.
> I guess, for some modules the backward compatibility is broken and
> several packages of the AUR (ArchLinux User Repo) are outdated and
> have incomplete dependency lists. It's a me....
> I tried to solve this by updating several packages, but then it
> even get's worse.  Probably that's the reason why the is no package of
> Beancounter for ArchLinux available.
>  With all these problem and having your warning regarding the RAM usage
> in mind, I think it's better to quit this attempt and to concentrate on
> the running Debian version.

beancounters' Finance::YahooQuote probably represents the most significant
dependency addition over what GT requires. yell if there is anything you
run in to that you cannot resolve. as far as i know, if your system has
both gcc and perl you should be able to build any perl module from it's
source package and not need to rely on pre-packaged packages. the risk
is knowing where to install stuff ...

> 
> But please be aware : due to all the new things I have to handle I'll be
> much slower with progress than you fix the errors   :-)))

i understand -- i went through much the same a long long long time ago.
and once i got the few gt sample examples working using the flatfile
sample database (e.g. 13000, etc) i felt it was an application that
would work for me. then i began to concentrate on getting current
price data into my beancounter database.

> 
> Thanks again for the help !
> Cheers,Delix
> 
> P.S.: I saw my posting with the other e-mail address (fm.bergm...) made
> its way to the mailing list although this address isn't registered.
> Can you please remove this message from the archive ? I'm not happy
> with having this address visible on a public website, but I could heal
> this fault by myself -- Thanks for that.
> 

oops --- sure i'll delete it. i saw it was about a current topic and
accepted it before realizing it was probably you -- so is there a story
behind the meaning/importance of delix? -- ras

Re: [GT] patch for backtest.pl

[Delix <del...@t-...>]

On Wed, 11 Jul 2012 21:00:35 -0700
"Robert A. Schmied" <ra...@ac...> hatte geschrieben :

> aloha delix
> 
> append the following terse usage subroutine to the backtest.pl file.
> 
> i don't think it's going to resolve the   "The directory ' ' doesn't exist"
> error, but it might just.
> 
> you can do a before and after eval using this command:
> 
>      $  ./backtest.pl --man
> 
> before the fix it should error out something like:
>      $  ./backtest.pl --man
> Undefined subroutine &main::usage called at ./backtest.pl line 304.
> 
> after the fix
> 
>      $  ./backtest.pl --man
>      ./backtest.pl [ options ] <code>
>      ./backtest.pl [ options ] <system_alias> <code>
>      ./backtest.pl [ options ] "<full_system_name>" <code>
>      options: -full -verbose -html -dt
>      opt w arg: -te fn, -mm s, -tf s, -of s, -cs s, -b s, -sy s, -sto fn, -gr fn,
>                 -mli n, -od dn, -set s, -iv n
>      typical opts: -start date, -end date, -timeframe tfname
> 
> give me a shout if this also affects the "The directory ' ' doesn't exist"
> problem in any way.
> 
> i'm still perplexed about this error. that directory isn't even used
> unless --set is set, the trade history file is always written to ./bt_portfolio.xml
> unless --store is defined and then it's still always likely cwd
> (but there could be a key elsewhere that alters that dir).

It will take some time, 'til I can do some corrections in the code. As
I wrote I'm a novice not only to GT but also to Perl. As this is a good
way to get more familiar with Perl, I'll try to understand the code
before I'll insert the modifications. 


> 
> 
> 
> 
> and if you want the pending fix for the *no timeframe default* you
> can change line around 263


Yesterday I played around with the GT installation on the small machine.
Actually, I guess the problems don't arise from GT or Beancounter,
but from the Perl modules they depend on.
I guess, for some modules the backward compatibility is broken and
several packages of the AUR (ArchLinux User Repo) are outdated and
have incomplete dependency lists. It's a me....
I tried to solve this by updating several packages, but then it
even get's worse.  Probably that's the reason why the is no package of
Beancounter for ArchLinux available.
 With all these problem and having your warning regarding the RAM usage
in mind, I think it's better to quit this attempt and to concentrate on
the running Debian version.

But please be aware : due to all the new things I have to handle I'll be
much slower with progress than you fix the errors   :-)))

Thanks again for the help !
Cheers,Delix

P.S.: I saw my posting with the other e-mail address (fm.bergm...) made
its way to the mailing list although this address isn't registered.
Can you please remove this message from the archive ? I'm not happy
with having this address visible on a public website, but I could heal
this fault by myself -- Thanks for that.

-- 
Delix <lin...@t-...>

Re: [GT] Date problem

[Delix <del...@t-...>]

> 
> i'm not following -- can you please submit your complete command line.
> 
> the message suggests you are trying to write command output to a particular
> directory which might not exist or is not writable. the exact, complete
> comamnd line and message would help diagnose. and because ones gt config file
> ($HOME/.gt/options) can also affect the location backtest uses for output
> the *complete content of that file* is also helpful.
> 
> 

On both machines I followed exactely  the instruction I found on
http://www.geniustrader.org/first_use.html

I copied the suggested options file to my GT directory and modified
ONLY the paths for the data and fonts directory and let everything
else unchanged.
Then I run your build.pl script (thanks for that -- this really helps
a lot), found no problems for the Debian machine and the missing
beancounter and its dependencies (MASON, LWP,....) on the smallmachine.

Then I run the "display_indicator.pl" test successfully.
When I starteted the command
                          ./backtest .pl TFS 13000
on the small machine I got the error message I sent in the first
posting (this was the exact and complete text). Here the problem
was solved by adding the --timeframe 'day' option to the command.

When I run the command
                    ./backtest.pl TFS 13000
on my Debian system, I got the error message :
                         "The directory ' ' doesn't exist"
(exact text; then the program exits without any further message)

I checked the code of the backtest.pl file and saw the item
description for the output-directory.
So I run the command
./backtest.pl --output-directory=/home/user/geniustrader TFS 13000

and got the correct output of the example ON THE SCREEN. No other
file was created with this command.

      
-- 
Delix <lin...@t-...>

Re: [GT] Date problem

[Robert A. S. <ra...@ac...>]

Delix wrote:
> On Sat, 07 Jul 2012 10:35:20 -0700
> "Robert A. Schmied" <ra...@ac...> hatte geschrieben :
> 
> 
>>Delix wrote:
>>
>>>Hi again !
>>>Sorry for being back so soon...
>>>I went on with the installation and started the tests as written in the wiki.
>>>the display-indicator.pl test was successful without any problems
>>>
>>>However, the backtest.pl tests failed.
>>
>>ok assuming your backtest command is
>>
>>    backtest.pl TFS 13000
>>
>>and your gt options file contains
>>Aliases::Global::TFS    SY:TFS 50 10 | CS:SY:TFS
>>
>>and using the text based database i'm getting
>>
>>localhost:(Scripts)-ras [ 327 ] %    backtest.pl TFS 13000
>>the requested timeframe "" is not in the list of available timeframes
>> ""
>>
>>so something is truly amiss someplace -- looks like backtest.pl requires an explicit timeframe argument
>>this might be due to the nature of file based prices data ...
>>
> 
> 
> Hi again,
> in the meantime I've finished the installation on my Debian system
> successfully.
> However, I had an issue with the suggested backtest example again.
> This time the --timeframe option didn't help, as I got an error message
> about a missing ' ' directory. Adding a "--output-directory= /home/..."

i'm not following -- can you please submit your complete command line.

the message suggests you are trying to write command output to a particular
directory which might not exist or is not writable. the exact, complete
comamnd line and message would help diagnose. and because ones gt config file
($HOME/.gt/options) can also affect the location backtest uses for output
the *complete content of that file* is also helpful.


ras



> instead the --timeframe option solved this problem this time.
> I used the same GT version on both machines, so to me this seems to be a
> (serious ?)  memory allocation problem or something similar.
> 
> Probably the changes in the wiki should be removed again ....
> 
> Hope this info may help to find the origin of the testing problems.

Re: [GT] Date problem

[Delix <del...@t-...>]

On Sat, 07 Jul 2012 10:35:20 -0700
"Robert A. Schmied" <ra...@ac...> hatte geschrieben :

> Delix wrote:
> > Hi again !
> > Sorry for being back so soon...
> > I went on with the installation and started the tests as written in the wiki.
> > the display-indicator.pl test was successful without any problems
> > 
> > However, the backtest.pl tests failed.
> 
> ok assuming your backtest command is
> 
>     backtest.pl TFS 13000
> 
> and your gt options file contains
> Aliases::Global::TFS    SY:TFS 50 10 | CS:SY:TFS
> 
> and using the text based database i'm getting
> 
> localhost:(Scripts)-ras [ 327 ] %    backtest.pl TFS 13000
> the requested timeframe "" is not in the list of available timeframes
>  ""
> 
> so something is truly amiss someplace -- looks like backtest.pl requires an explicit timeframe argument
> this might be due to the nature of file based prices data ...
> 

Hi again,
in the meantime I've finished the installation on my Debian system
successfully.
However, I had an issue with the suggested backtest example again.
This time the --timeframe option didn't help, as I got an error message
about a missing ' ' directory. Adding a "--output-directory= /home/..."
instead the --timeframe option solved this problem this time.
I used the same GT version on both machines, so to me this seems to be a
(serious ?)  memory allocation problem or something similar.

Probably the changes in the wiki should be removed again ....

Hope this info may help to find the origin of the testing problems.
-- 
Delix <lin...@t-...>

[GT] revised Build.PL hack

[Robert A. S. <ra...@ac...>]

gt'ers

there's an updated Build.PL available -- see "for the bold ..." para on
http://geniustrader.org/first_use.html.

it has the correct geniustrader url -- thanks jae

plus a bunch of extra operating environment checks, things like
ensuring cwd is Scripts, and Scripts is writable and the toolkit
dir is ../GT Scripts relative.

for brand new users it can be used to help figure out what perl
modules are missing ...

for current users or users having trouble it will look over your
options file checking the database module settings and it will
validate every font path for both existence and file type (if
the *nix 'file' util is found. this latter might be non-portable,
in terms of the text string defined for a truetype file. if you
run into issues where Build.PL claims a fontpath file isn't truetype,
but that fontface works fine with graphic.pl please contact me, thanks.

enjoy


aloha


ras