geniustrader-devel — participate in geniustrader development, ask for help, suggest improvements, etc

[GT] Re: Graphing under Ubuntu

[Robert A. S. <ra...@ac...>]

jec...@te... wrote:
> After finally getting the font issue resolved, I tried some more test and got
> stuck real well on the very next step.
> 
> I tried to make a graph with:
> user@ubuntu:~/geniustrader/Scripts$ perl ./backtest.pl --graph=¨test.png¨ TFS
> 13000
> Can't locate GD.pm in @INC (@INC contains: .. /etc/perl
> /usr/local/lib/perl/5.10.0 /usr/local/share/perl/5.10.0 /usr/lib/perl5
> /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10
> /usr/local/lib/site_perl .) at ../GT/Graphics/Driver/GD.pm line 9.
> BEGIN failed--compilation aborted at ../GT/Graphics/Driver/GD.pm line 9.
> Compilation failed in require at (eval 38) line 1.
> BEGIN failed--compilation aborted at (eval 38) line 1.
> user@ubuntu:~/geniustrader/Scripts$
> 
> Iǘe tried various variations on using -I on the command line, but no luck at
> all thus far.
> 
> Any help will be most appreciated.



jecxz112

to see the trading activity you need to add -display-trades

%  ./backtest.pl --graph=test.png --display-trades TFS 13000
##
## Analysis of code 13000 using System Spec

 SY:TFS 50 10 | CS:SY:TFS 50 | MM:Basic

...

[GT] Re: Where is the on-line archive?

[Robert A. S. <ra...@ac...>]

jecxz112 wrote:
> Bhaskar S. Manda wrote:
> 
>> On Tue, Nov 3, 2009 at 7:23 PM, jecxz112 <jec...@te...> wrote:
>>
>>  
>>
>>> are the archives searchable in any way?
>>>     
>>
>>
>> There's no search capability there. You can always do a Google search as
>> follows.
>>    site:geniustrader.org <search terms>

Bhaskar -- thanks for that. it should be worked into the web site someplace ...

>>   
> 
> Thank you for your help;  I will use that from now on.
> 
>>> C:\Program Files\GeniusTrader\Scripts>display_indicator.pl I:RSI 13000
>>> Can't locate XML/LibXML.pm in @INC (@INC contains: .. C:\Program
>>>     
>>
>> Files\GeniusTra
>> der C:/Perl/site/lib C:/Perl/lib .) at ../GT/Serializable.pm line 10.
>>  
>>
>>> BEGIN failed--compilation aborted at ../GT/Serializable.pm line 10.
>>>     
>>
>>
>> You need to install the Perl modules for XML, Serializable, etc. For your
>> Ubuntu machine, see the section "Install the Dependencies" in the
>> first_use.html link I mentioned in my earlier reply. 
> 
> All of those packages were installed on Ubuntu - and I finally by trial 
> and error found some fonts
> that seemed to do the trick.
> For the record:
> /usr/share/fonts/truetype/freefonts/ FreeSans.ttf - FreeMono.ttf and 
> FreeSerif.ttf.

so for Ubuntu linux you needed lines something like the following
in your $HOME/.gt/options file?

Path::Font::Arial	/usr/share/fonts/truetype/freefonts/FreeSans.ttf
Path::Font::Courier	/usr/share/fonts/truetype/freefonts/FreeMono.ttf
Path::Font::Times	/usr/share/fonts/truetype/freefonts/FreeSerif.ttf

note the gt wiki page 'The user config file' (gt webpage box upper right)
discusses the $HOME/.gt/options file in some detail and specifically
addresses Ubuntu linux fonts. if these notes are mis-leading or wrong
you can correct them yourself provided you register and login.

> 
> Also, FWIW, the page first_use.html  shows
> 
> $ ./backtest.pl 'TFS[30,7,7]' 13000 | less
> 
> As a test. On my machine, I needed to omit the ticks.

the ticks are only *needed* if your shell considers something
in the string being quoted as a shell meta-character.
the culprit in this case are the square brackets (e.g. []),
in other cases it may be the curly brackets (e.g. {}). if you
has embedded spaces they also would need to be escaped ...
of course you can quote (in other words escape) the shell
meta-characters with a leading backslash (e.g. \)

  %  ./backtest.pl TFS\[30,7,7\] 13000

the csh is one shell that needs both squares and curlys escaped.
here, bash, sh, and ksh don't care about the squares. i didn't
look at the curlys.

the perplexing bit is that you had to remove them for ./backtest.pl
to work. i'm not seeing that here with any of the tested shells. maybe
there's an argument cleanup that removes them internally that is still
missing from the gt trunk version?

you do have the TFS[] alias defined in your $HOME/.gt/options file
something like this?

Aliases::Global::TFS[]	SY:TFS #1 #2|CS:SY:TFS #1|CS:Stop:Fixed #3

and you are using the correct quote character (e.g. ')?

ras

> 
> Now that I have it going on one system, I may or may not try to make the 
> Windows system work as well,
> but ....
> 
>> For your Windows
>> machine, I don't remember how to install dependencies for the Activestate
>> distribution. However,
>>    perl -MCPAN -e shell
>> should work at a DOS prompt, and the help in that tool will guide you
>> through installing the dependent modules listed in the first_use.html. 
>> Note
>> that I don't know if this is the approved way of doing it for Windows.
>>   
> 
> I had tried to install the same Perl modules, but IIRC, I could not find 
> the XML module equivalent in the CPAN list.
> As well, there seems to be a XML-LibXML.ppd in the modules subdirectory, 
> but the error message refers to
> XML/LibXML.pm - using the slash rather than the dash as well as the pm 
> rather than ppd extension.
> I'm not that much of a Perl guru to be able to resolve that issue 
> without a fair bit of digging.
> 
> 

[GT] Re: Re: Graphing under Ubuntu

[jecxz112 <jec...@te...>]

Bhaskar S. Manda wrote:
> And you have GD installed, I assume? It is in the package libgd-gd2-perl in
> Debian.
>   
I have now and it did produce a graph. :-)

Thank you ever so much. Hope this is the last one ;-)

Again, thank you

-- 
Fight Spam - report it with wxSR
http://www.columbinehoney.net/wxSR.shtml

[GT] Re: Graphing under Ubuntu

[Bhaskar S. M. <bha...@gm...>]

And you have GD installed, I assume? It is in the package libgd-gd2-perl in
Debian.

On Wed, Nov 4, 2009 at 2:19 PM, <jec...@te...> wrote:

>
> Can't locate GD.pm in @INC (@INC contains: .. /etc/perl
>

-- 
Bhaskar

[GT] Graphing under Ubuntu

[<jec...@te...>]

After finally getting the font issue resolved, I tried some more test and got
stuck real well on the very next step.

I tried to make a graph with:
user@ubuntu:~/geniustrader/Scripts$ perl ./backtest.pl --graph=¨test.png¨ TFS
13000
Can't locate GD.pm in @INC (@INC contains: .. /etc/perl
/usr/local/lib/perl/5.10.0 /usr/local/share/perl/5.10.0 /usr/lib/perl5
/usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10
/usr/local/lib/site_perl .) at ../GT/Graphics/Driver/GD.pm line 9.
BEGIN failed--compilation aborted at ../GT/Graphics/Driver/GD.pm line 9.
Compilation failed in require at (eval 38) line 1.
BEGIN failed--compilation aborted at (eval 38) line 1.
user@ubuntu:~/geniustrader/Scripts$

Iǘe tried various variations on using -I on the command line, but no luck at
all thus far.

Any help will be most appreciated.

[GT] Re: Re: Re: Re: Where is the on-line archive?

[jecxz112 <jec...@te...>]

Bhaskar S. Manda wrote:
> On Tue, Nov 3, 2009 at 7:23 PM, jecxz112 <jec...@te...> wrote:
>
>   
>> are the archives searchable in any way?
>>     
>
> There's no search capability there. You can always do a Google search as
> follows.
>    site:geniustrader.org <search terms>
>   
Thank you for your help;  I will use that from now on.
>> C:\Program Files\GeniusTrader\Scripts>display_indicator.pl I:RSI 13000
>> Can't locate XML/LibXML.pm in @INC (@INC contains: .. C:\Program
>>     
> Files\GeniusTra
> der C:/Perl/site/lib C:/Perl/lib .) at ../GT/Serializable.pm line 10.
>   
>> BEGIN failed--compilation aborted at ../GT/Serializable.pm line 10.
>>     
>
> You need to install the Perl modules for XML, Serializable, etc. For your
> Ubuntu machine, see the section "Install the Dependencies" in the
> first_use.html link I mentioned in my earlier reply. 
All of those packages were installed on Ubuntu - and I finally by trial 
and error found some fonts
that seemed to do the trick.
For the record:
/usr/share/fonts/truetype/freefonts/ FreeSans.ttf - FreeMono.ttf and 
FreeSerif.ttf.

Also, FWIW, the page first_use.html  shows

$ ./backtest.pl 'TFS[30,7,7]' 13000 | less

As a test. On my machine, I needed to omit the ticks.

Now that I have it going on one system, I may or may not try to make the 
Windows system work as well,
but ....

> For your Windows
> machine, I don't remember how to install dependencies for the Activestate
> distribution. However,
>    perl -MCPAN -e shell
> should work at a DOS prompt, and the help in that tool will guide you
> through installing the dependent modules listed in the first_use.html. Note
> that I don't know if this is the approved way of doing it for Windows.
>   
I had tried to install the same Perl modules, but IIRC, I could not find 
the XML module equivalent in the CPAN list.
As well, there seems to be a XML-LibXML.ppd in the modules subdirectory, 
but the error message refers to
XML/LibXML.pm - using the slash rather than the dash as well as the pm 
rather than ppd extension.
I'm not that much of a Perl guru to be able to resolve that issue 
without a fair bit of digging.


-- 
Fight Spam - report it with wxSR
http://www.columbinehoney.net/wxSR.shtml

[GT] Re: Re: perl floating point great-equal bug?

[Thomas W. <we...@ms...>]

I think this warrants a broader examination.

I think that prices, if they need to be matched precisely, should not be
implemented as floating point numbers, or if we do so, they should be
compared using an epsilon region.

But it might be best to treat prices as fixed point numbers, to ensure
that we can make exact comparisons?

In either case, a systematic review of the use of prices is warranted....

Th.

Robert A. Schmied wrote:
> fr....@ti... wrote:
>> Hello,
>>     i'm new to this  mailing list, i started using GT at the
>> beginning of this year but i didnt realize the project was going on,
>> so i used the older version and i modified it a little bit.
>> Now that i discoved the svn version im switching to it.
>>
>> One of the things that i changed on the library that seems not
>> corrected in the svn version is a little bug using the >= and <=
>> operators in floating context.
>>
>> The problem is that if you create the order using for the price
>> ($order->price) a division like ($var/100)*100, the $order->price
>> will not be exactly $var but $var ± 0.000....01 (this is a known perl
>> problem)
>> and the is_executed function could return false if the $order->price
>> is equal to the HIGH or the LOW of the bar.
>>
>> In particular i created 2 functions in Tools.pm and changed all the
>> <= and >= with these functions in
>>
>> Portfolio/Order.pm
>> is_executed function
>> lines 372 373 389 394 402 403 408 409
>>
>> I dont know if this problem is present in other part of GT but i
>> fixed only in Order.pm
>>
>> If someone is interested in the problem i can send  the
>> Portfolio/Order.pm and Tools.pm modified.
>>
>> I hope this is usefull.
>>
>> PS: im using the http://www.geniustrader.org/wws/compose_mail/devel
>> to write this post but i cant attach anything here.Is possible to
>> send a post
>> from thunderbird?
>>
>> thx
>
>
> aloha fr.gamberale
>
> your observation could very well be the cause of many of the odd
> results one
> sees from time to time.
>
> as there are three (at least) gt branches on the svn at
> http://geniustrader.org/
> you need to identify the one you're using so we can get on the same
> page (or at
> least identify it as a reference).
>
> as far as sharing your fixes you can either post differences or the
> complete files.
> differences might be the preferred method, but it probably doesn't
> matter much.
> see man page diff(1) and/or info diff for details on how to create
> difference files.
> for reference diff -u orig_file changed_file. there are probably diff
> equivalents
> built-in in svn, but i don't know about them, so you are on your own
> there.
>
> if you have subscribed to the gt devel mailing list you should be able
> to post
> messages with attachments to the list with any email client. see the
> gt website
> (http://geniustrader.org/) main page 'Mailing lists' section. for
> reference you
> post to de...@ge....
>
> ciao
>
> ras
>

[GT] Re: perl floating point great-equal bug?

[Robert A. S. <ra...@ac...>]

fr....@ti... wrote:
> Hello,
>     i'm new to this  mailing list, i started using GT at the beginning 
> of this year but i didnt realize the project was going on, so i used the 
> older version and i modified it a little bit.
> Now that i discoved the svn version im switching to it.
> 
> One of the things that i changed on the library that seems not corrected 
> in the svn version is a little bug using the >= and <= operators in 
> floating context.
> 
> The problem is that if you create the order using for the price 
> ($order->price) a division like ($var/100)*100, the $order->price will 
> not be exactly $var but $var ± 0.000....01 (this is a known perl problem)
> and the is_executed function could return false if the $order->price is 
> equal to the HIGH or the LOW of the bar.
> 
> In particular i created 2 functions in Tools.pm and changed all the <= 
> and >= with these functions in
> 
> Portfolio/Order.pm
> is_executed function
> lines 372 373 389 394 402 403 408 409
> 
> I dont know if this problem is present in other part of GT but i fixed 
> only in Order.pm
> 
> If someone is interested in the problem i can send  the 
> Portfolio/Order.pm and Tools.pm modified.
> 
> I hope this is usefull.
> 
> PS: im using the http://www.geniustrader.org/wws/compose_mail/devel 
> to write this post but i cant attach anything here.Is possible to send a post
> from thunderbird?
> 
> thx


aloha fr.gamberale

your observation could very well be the cause of many of the odd results one
sees from time to time.

as there are three (at least) gt branches on the svn at http://geniustrader.org/
you need to identify the one you're using so we can get on the same page (or at
least identify it as a reference).

as far as sharing your fixes you can either post differences or the complete files.
differences might be the preferred method, but it probably doesn't matter much.
see man page diff(1) and/or info diff for details on how to create difference files.
for reference diff -u orig_file changed_file. there are probably diff equivalents
built-in in svn, but i don't know about them, so you are on your own there.

if you have subscribed to the gt devel mailing list you should be able to post
messages with attachments to the list with any email client. see the gt website
(http://geniustrader.org/) main page 'Mailing lists' section. for reference you
post to de...@ge....

ciao

ras

[GT] Re: Re: Re: Where is the on-line archive?

[Bhaskar S. M. <bha...@gm...>]

On Tue, Nov 3, 2009 at 7:23 PM, jecxz112 <jec...@te...> wrote:

> are the archives searchable in any way?
>

There's no search capability there. You can always do a Google search as
follows.
   site:geniustrader.org <search terms>


> C:\Program Files\GeniusTrader\Scripts>display_indicator.pl I:RSI 13000
> Can't locate XML/LibXML.pm in @INC (@INC contains: .. C:\Program
Files\GeniusTra
der C:/Perl/site/lib C:/Perl/lib .) at ../GT/Serializable.pm line 10.
> BEGIN failed--compilation aborted at ../GT/Serializable.pm line 10.

You need to install the Perl modules for XML, Serializable, etc. For your
Ubuntu machine, see the section "Install the Dependencies" in the
first_use.html link I mentioned in my earlier reply. For your Windows
machine, I don't remember how to install dependencies for the Activestate
distribution. However,
   perl -MCPAN -e shell
should work at a DOS prompt, and the help in that tool will guide you
through installing the dependent modules listed in the first_use.html. Note
that I don't know if this is the approved way of doing it for Windows.
-- 
Bhaskar

[GT] perl floating point great-equal bug?

[<fr....@ti...>]

Hello,
    i'm new to this  mailing list, i started using GT at the beginning 
of this year but i didnt realize the project was going on, so i used the 
older version and i modified it a little bit.
Now that i discoved the svn version im switching to it.

One of the things that i changed on the library that seems not corrected 
in the svn version is a little bug using the >= and <= operators in 
floating context.

The problem is that if you create the order using for the price 
($order->price) a division like ($var/100)*100, the $order->price will 
not be exactly $var but $var ± 0.000....01 (this is a known perl problem)
and the is_executed function could return false if the $order->price is 
equal to the HIGH or the LOW of the bar.

In particular i created 2 functions in Tools.pm and changed all the <= 
and >= with these functions in

Portfolio/Order.pm
is_executed function
lines 372 373 389 394 402 403 408 409

I dont know if this problem is present in other part of GT but i fixed 
only in Order.pm

If someone is interested in the problem i can send  the 
Portfolio/Order.pm and Tools.pm modified.

I hope this is usefull.

PS: im using the http://www.geniustrader.org/wws/compose_mail/devel 
to write this post but i cant attach anything here.Is possible to send a post
from thunderbird?

thx

[GT] Re: Re: Where is the on-line archive?

[jecxz112 <jec...@te...>]

Bhaskar S. Manda wrote:
> On Tue, Nov 3, 2009 at 5:45 PM, <jec...@te...> wrote:
>
>   
>> I'm signing on via Firefox 3.53, but I can't get to the mail archive - that
>> link is not active -What do I need to do to make it work?
>>     
>
> Please use the links to the mailing list on the GeniusTrader.org page. They
> are as follows.
> Devel List:   http://www.geniustrader.org/lists/devel/
> System_Traders List: http://www.geniustrader.org/lists/system-traders/
>   
Thank you for the links;

are the archives searchable in any way?
>> I have been unable to use either the Windows installer - under Vista 64,
>> nor
>> the manual install via the tar ball as described here under Ubuntu Jaunty
>> Jack:
>> http://www.elitetrader.com/vb/printthread.php?threadid=121252
>>     
>
> Use this approach.
>    http://www.geniustrader.org/first_use.html
>
>   
Initially I started from
http://www.geniustrader.org/first_use.html
with the supposedly latest Windows installer; when that failed I found 
the page you mention, but had no better luck trying to install it on 
Ubuntu 9.04 in a VMWare Player 3.0.0, just different problems.

I have now tried to install the Windows version on an XP machine after 
installing ActiveState Perl 5.10.1 and I get the same errors as I did 
under Windows:

C:\Program Files\GeniusTrader\Scripts>display_indicator.pl I:RSI 13000
Can't locate XML/LibXML.pm in @INC (@INC contains: .. C:\Program 
Files\GeniusTra
der C:/Perl/site/lib C:/Perl/lib .) at ../GT/Serializable.pm line 10.
BEGIN failed--compilation aborted at ../GT/Serializable.pm line 10.
Compilation failed in require at ../GT/Prices.pm line 13.
BEGIN failed--compilation aborted at ../GT/Prices.pm line 13.
Compilation failed in require at C:\Program 
Files\GeniusTrader\Scripts\display_i
ndicator.pl line 12.
BEGIN failed--compilation aborted at C:\Program 
Files\GeniusTrader\Scripts\displ
ay_indicator.pl line 12.

C:\Program Files\GeniusTrader\Scripts>

I have used Perl off and on in the past, but I am by no means a guru and 
certainly don't want to dig into the code to make it run.

>   
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com 
> Version: 8.5.424 / Virus Database: 270.14.47/2478 - Release Date: 11/03/09 07:36:00
>
>   


-- 
Fight Spam - report it with wxSR
http://www.columbinehoney.net/wxSR.shtml

[GT] Re: Where is the on-line archive?

[Bhaskar S. M. <bha...@gm...>]

On Tue, Nov 3, 2009 at 5:45 PM, <jec...@te...> wrote:

> I'm signing on via Firefox 3.53, but I can't get to the mail archive - that
> link is not active -What do I need to do to make it work?
>

Please use the links to the mailing list on the GeniusTrader.org page. They
are as follows.
Devel List:   http://www.geniustrader.org/lists/devel/
System_Traders List: http://www.geniustrader.org/lists/system-traders/


> I have been unable to use either the Windows installer - under Vista 64,
> nor
> the manual install via the tar ball as described here under Ubuntu Jaunty
> Jack:
> http://www.elitetrader.com/vb/printthread.php?threadid=121252
>

Use this approach.
   http://www.geniustrader.org/first_use.html


-- 
Bhaskar

[GT] Where is the on-line archive?

[<jec...@te...>]

I'm a newb to this forum and just subscribed after finding GT just these last
few days.

I'm signing on via Firefox 3.53, but I can't get to the mail archive - that
link is not active -What do I need to do to make it work?

I have been unable to use either the Windows installer - under Vista 64, nor
the manual install via the tar ball as described here under Ubuntu Jaunty Jack:
http://www.elitetrader.com/vb/printthread.php?threadid=121252

Before I start asking questions, I wanted to search the forum, but have had no
luck yet.

TIA

[GT] Re: [PATCH] Use a managed list for pending order in portfolio, instead of an array indexed by order_id, as this causes a long running backtest to be working with a very large sparse array with lots of undef at an overall perforamnce penality of 20%.

[Chia-liang K. <cl...@cl...>]

The original change caused some regression (hooray for finally having
some tests!), so here's the updated version.

The change makes running large (ie 8 years of 5min bars) more linear
in time, rather than growing slower along with order id increment over
time.

[GT] [PATCH] short circuit I::Prices in ArgsTree to avoid is_available calls.

[Chia-liang K. <cl...@cl...>]

---
 lib/Finance/GeniusTrader/ArgsTree.pm |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/lib/Finance/GeniusTrader/ArgsTree.pm b/lib/Finance/GeniusTrader/ArgsTree.pm
index f8c3d30..c2827b5 100644
--- a/lib/Finance/GeniusTrader/ArgsTree.pm
+++ b/lib/Finance/GeniusTrader/ArgsTree.pm
@@ -141,7 +141,10 @@ sub get_arg_values {
 	    my $object = $self->[$n][0]{"object"};
 	    my $number = $self->[$n][0]{"number"};
 	    my $name = $object->get_name($number);
-	    if (ref($object) =~ /Finance::GeniusTrader::Indicators/) {
+	    if (ref($object) =~ /Finance::GeniusTrader::Indicators::Prices/ && $object->{'use_std_prices'}) {
+                return if $day<0 || $day >= $calc->prices->count;
+                return $calc->prices->at($day)->[$object->{'data_ind'}];
+            } elsif (ref($object) =~ /Finance::GeniusTrader::Indicators/) {
 		$object->calculate($calc, $day) 
 		  unless ($indic->is_available($name, $day));
 		if ($indic->is_available($name, $day)) {
-- 
1.6.4.2.253.g0b1fac

[GT] [PATCH] Use a managed list for pending order in portfolio, instead of an array indexed by order_id, as this causes a long running backtest to be working with a very large sparse array with lots of undef at an overall perforamnce penality of 20%.

[Chia-liang K. <cl...@cl...>]

---
 lib/Finance/GeniusTrader/Portfolio.pm |   15 +++++++--------
 1 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/lib/Finance/GeniusTrader/Portfolio.pm b/lib/Finance/GeniusTrader/Portfolio.pm
index 47cd55a..2da8226 100644
--- a/lib/Finance/GeniusTrader/Portfolio.pm
+++ b/lib/Finance/GeniusTrader/Portfolio.pm
@@ -71,7 +71,7 @@ sub add_order {
     $order->set_id($self->{'order_id'});
     
     $self->{'order_id'}++;
-    $self->{'pending_orders'}[$order->id] = $order;
+    push @{$self->{'pending_orders'}}, $order;
     return $order->id;
 }
 
@@ -87,9 +87,8 @@ sub discard_order {
     #WAR#  WARN  "valid type for order" if ( ref($order) =~ /Portfolio::Order/);
     #WAR#  WARN  "order id is positive" if ( $order->id >= 0);
     
-    push @{$self->{'discarded_orders'}},
-	 $self->{'pending_orders'}[$order->id];
-    $self->{'pending_orders'}[$order->id] = undef;
+    push @{$self->{'discarded_orders'}}, $order;
+    $self->delete_order($order);
     
     return;
 }
@@ -99,8 +98,9 @@ sub delete_order {
 
     #WAR#  WARN  "valid type for order" if ( ref($order) =~ /Portfolio::Order/);
     #WAR#  WARN  "order id is positive" if ( $order->id >= 0);
-    
-    $self->{'pending_orders'}[$order->id] = undef;
+    my $order_id = $order->id;
+    @{$self->{'pending_orders'}} = grep { $_->id != $order_id } @{$self->{'pending_orders'}};
+
     return;
 }
 
@@ -219,7 +219,6 @@ sub apply_pending_orders {
     }
 
     foreach (@{$self->{'pending_orders'}}) {
-	next if (! defined($_));
 	next if ($_->code ne $calc->code);
 	next if ($_->source ne $source);
 
@@ -443,7 +442,7 @@ sub list_pending_orders {
 	return grep { defined($_) && ($_->source eq $source) }
 		    @{$self->{'pending_orders'}};
     } else {
-	return grep { defined($_) } @{$self->{'pending_orders'}};
+	return @{$self->{'pending_orders'}};
     }
 }
 
-- 
1.6.4.2.253.g0b1fac

[GT] [PATCH] aggregated stop orders

[Chia-liang K. <cl...@cl...>]

Hi all,

the failing test is now available at
http://github.com/clkao/finance-geniustrader/blob/master/t/cs/stop-fixed.t

The test requires some of the helper functions i committed to the CPAN
branch of thomas' mirror.  Raphael, can I please get a commit bit for
the current svn repository while we are still figuring out the project
ownership issues to avoid divergence?

The fix is a continuation of my previous position fixes branch. note
that the first 3 patches were to be applied to the old GT tree, so
please do a mental substitution.

2009/10/24 Chia-liang Kao <cl...@cl...>:
> The attached 5min chart should illustrate the problem better.
>
> the system enters a short at 8885, with a 1% CS:Stop:Fixed, which
> should S/L at 8973, which is at the obvious huge breakout with the
> blue arrow at volumes.  However the system has another
> CS:ChannelBreakout stop, for the highest high over a certain period.
> the value is said to be 8990 (comes from some bar before this chart).
> so the system basically has instructions to stop at both 8973 and
> 8990.  in reality with streamy price, whichever reached first would
> simply close the position.  whereas in current GT code, it depends on
> the order of the CS (and whether the CS is using set_stop or
> submitting an order into the opened position), and that makes the
> backtest far from being accurate.
>
> 2009/10/22 Robert A. Schmied <ra...@ac...>:
>> but back to this particular closing strategy observation of yours:
>> i've found that the order closing strategies appear in a gt trading system
>> description will/can affect the results with the same input. i'd expect
>> that if you reverse the order of your two closing strategy descriptions
>> in the trading system description you would see a different outcome.
>
> yes and no.  if the two CS are both using orders submitted to a opened
> position, the order would affect the results.  but with one of the CS
> being CS:Stop:Fixed, the code is always trying the stop value after
> all other stop orders have been tried.
>
>> if not, you have probably found the right solution, but if the answer is
>> yes then i would argue (suggest/wonder) that the problem may be larger
>> than just stop orders versus close orders. because multiple closing
>> strategies
>> are allowed it may be that all (CS orders) should be fully evaluated
>> before any are applied. then they could be ranked in terms of the magnitude
>> of any gain/loss and applied in the most advantageous way.
>
> Actually my original solution didn't solve all the problems.  Those CS
> that always have monotonously increasing stop (toward the profitable
> direction, of course) should use the set_stop api.  for others, the
> submitted stop orders' value should be extracted and only the nearest
> should be tried in apply_pending_order. this solves the issue
> described above, but for the kind of CS with stop value not always
> increasing.  for example, a CS:ChannelBreakout with I:BOL.
>
> I should come up with a patch this weekend.
>
> Cheers,
> CLK
>

[GT] Re: Re: aggregated stop orders

[Chia-liang K. <cl...@cl...>]

The attached 5min chart should illustrate the problem better.

the system enters a short at 8885, with a 1% CS:Stop:Fixed, which
should S/L at 8973, which is at the obvious huge breakout with the
blue arrow at volumes.  However the system has another
CS:ChannelBreakout stop, for the highest high over a certain period.
the value is said to be 8990 (comes from some bar before this chart).
so the system basically has instructions to stop at both 8973 and
8990.  in reality with streamy price, whichever reached first would
simply close the position.  whereas in current GT code, it depends on
the order of the CS (and whether the CS is using set_stop or
submitting an order into the opened position), and that makes the
backtest far from being accurate.

2009/10/22 Robert A. Schmied <ra...@ac...>:
> but back to this particular closing strategy observation of yours:
> i've found that the order closing strategies appear in a gt trading system
> description will/can affect the results with the same input. i'd expect
> that if you reverse the order of your two closing strategy descriptions
> in the trading system description you would see a different outcome.

yes and no.  if the two CS are both using orders submitted to a opened
position, the order would affect the results.  but with one of the CS
being CS:Stop:Fixed, the code is always trying the stop value after
all other stop orders have been tried.

> if not, you have probably found the right solution, but if the answer is
> yes then i would argue (suggest/wonder) that the problem may be larger
> than just stop orders versus close orders. because multiple closing
> strategies
> are allowed it may be that all (CS orders) should be fully evaluated
> before any are applied. then they could be ranked in terms of the magnitude
> of any gain/loss and applied in the most advantageous way.

Actually my original solution didn't solve all the problems.  Those CS
that always have monotonously increasing stop (toward the profitable
direction, of course) should use the set_stop api.  for others, the
submitted stop orders' value should be extracted and only the nearest
should be tried in apply_pending_order. this solves the issue
described above, but for the kind of CS with stop value not always
increasing.  for example, a CS:ChannelBreakout with I:BOL.

I should come up with a patch this weekend.

Cheers,
CLK

[GT] Re: aggregated stop orders

[Robert A. S. <ra...@ac...>]

Chia-liang Kao wrote:
> Hi all,
> 
> I noticed something interesting from history of a backtest. a few
> trades suffer extremely large loss. larger than the CS:Stop:Fixed 1% i
> set. and this is not due to price splits since they are daytrades.
> 
> I dug further a few days ago, and found out why. i have another
> CS:ChannelBreakout, along with my CS:Stop:Fixed.  Position's
> apply_pending_orders method is first trying the "pending closing
> orders that are of type: stop", then the stop price associated with
> the position.  This causes the price executed to be some what
> unrealistic, as the position can be closed by a stop order that is not
> the closest one.
> 
> I propose all the closing strategy should use the set_stop method that
> already keeps track of the closest stop for us, and have all other
> pending order in positions executed after stop price is tried.
> 
> the affected CS are: ChannelBreakout, PartialStop.
> 
> objections? tests? patches?
> 
> Cheers,
> CLK
> 

aloha clk

as you are one of the worlds foremost experts on the inner workings
of a gt trading system you are probably in the best position to both
demonstrate the problem and propose the solution.

reviewing the code while trying to follow your narrative it seems you
are proposing to change GT::Portfolio::Position::apply_pending_orders
to use the GT::Portfolio::Position::set_stop method (which it currently
does not appear to use) in order to ensure all stop orders are processed
before processing any other orders that may also be pending.

because you didn't disclose the trading system description, the code
and the time span you used to observe this condition we are unable to
duplicate your observation which is not in question, i've no doubt that
you are observing this and have pinpointed where/how the problem occurs.
but without a known baseline it is difficult to be certain the effects
being observed with differing data are caused by the exact same mechanism.

as i've written before gt trading system backtesting isn't something
i've found too useful. when the results being returned seem intuitively
incorrect it is difficult to determine how the system is handling an
order to arrive at that result. i think an order auditing mechanism
would be a helpful addition. with it one could track any/all orders
that get generated and follow them as the are processed through each
method that interacts with them. unfortunately i don't have anythin
to contribute in that area, as i have no idea how to begin to implement
such a thing.

but back to this particular closing strategy observation of yours:
i've found that the order closing strategies appear in a gt trading system
description will/can affect the results with the same input. i'd expect
that if you reverse the order of your two closing strategy descriptions
in the trading system description you would see a different outcome.

if not, you have probably found the right solution, but if the answer is
yes then i would argue (suggest/wonder) that the problem may be larger
than just stop orders versus close orders. because multiple closing strategies
are allowed it may be that all (CS orders) should be fully evaluated
before any are applied. then they could be ranked in terms of the magnitude
of any gain/loss and applied in the most advantageous way.



ras

[GT] aggregated stop orders

[Chia-liang K. <cl...@cl...>]

Hi all,

I noticed something interesting from history of a backtest. a few
trades suffer extremely large loss. larger than the CS:Stop:Fixed 1% i
set. and this is not due to price splits since they are daytrades.

I dug further a few days ago, and found out why. i have another
CS:ChannelBreakout, along with my CS:Stop:Fixed.  Position's
apply_pending_orders method is first trying the "pending closing
orders that are of type: stop", then the stop price associated with
the position.  This causes the price executed to be some what
unrealistic, as the position can be closed by a stop order that is not
the closest one.

I propose all the closing strategy should use the set_stop method that
already keeps track of the closest stop for us, and have all other
pending order in positions executed after stop price is tried.

the affected CS are: ChannelBreakout, PartialStop.

objections? tests? patches?

Cheers,
CLK

[GT] Re: Future of GeniusTrader

[Robert A. S. <ra...@ac...>]

Raphael Hertzog wrote:
> (Please keep me in CC on replies as I don't really read the list)
> 
> Hello,
> 
> Chia-liang Kao mailed me recently to be granted commit access to
> GeniusTrader. I'm glad that the project seems to take off after
> having languished so many years.
> 
> However, even if I initiated the project, I'm not really leading it any
> more. Except me 4 persons have commit access nowadays:
> - joao: João Costa <joa...@zo...> (last seen in 2008)
> - olf: Oliver Bossert <ol...@ol...> (last seen in 2006)
> - ras: "Robert A. Schmied" <ra...@ac...> (active)
> - thomas: Thomas Weigert <we...@ms...> (active)
> 
> I don't want to be a blocker for the project so I would suggest to move
> the project on a forge so that I'm no longer the sole project admin that
> can grant commit access.
> 
> As part of my Debian work, I happen to be administrator of the
> http://alioth.debian.org forge so if that forge could suit you, I'd be
> glad to move the project there. It supports all modern VCS that could
> be of interest for the project (SVN, Git, bzr, hg, ...).
> 
> What do you think ? (Input from the new contributors who have no commit
> access is welcome too)
> 
> Thomas and Robert, if you think that I should immediately grant commit
> access to some new contributors, please tell me and I'll create them
> an SVN account.
> 
> Cheers,

Raphael and all gt'ers:

i'm of two minds (at least) on the latest craze of interest with GeniusTrader
(gt) but have yet to see any (much) of the sound and fury yield much (any) in
the way of *beneficial* change or functional improvement to gt itself, but as
the effort has only just begun it may be too early to tell just yet.

i've been silent on the gt cpanization because i see lots of seemingly
unnecessary change with little (no) gain. however, my negativity shouldn't
impede something that might be good for gt in the long run, hence my silence.
now, however, could be a turning point that dictates the direction gt goes
in the future and i'd like to encourage the retention of gts' original
operational philosophy yet allow the adding of new capabilities that fit
with its technical analysis and investment management roots.

first and foremost, i consider gt as an indispensable non-interactive
command line based tool that i use constantly. as such it should be
(or have a) reliable and fairly stable release that retains its current
operational characteristics.


current state -- from my point of view how we got where we are

unfortunately the legacy gt development scheme never had a process
to implement and maintain multiple branches. instead it relied
on users that took the time to report bugs and contributors that
investigated, developed, and submitted proposed patches and/or evaluated
and reported on proposed patches. positive reports on patches, or the
lack of negative reports usually, but not always, resulted in a change
commit. but without any user feedback a patch without an exceptional
set of documentation might languish until there is (was) time to evaluate
it. past experience shows very few proposed patches include sufficient
documentation to even formulate a test strategy, let alone be considered
exceptional.

this process worked because it had to, but it resulted in gt (the
developer community) never having to develop a set of regression test
processes and methods to automate change validation. the complexity and
magnitude of the gt toolkit (packages in GT hier) make the implementation
of this a fairly large seeming unproductive undertaking. i agree with
those that think this is a critical bit of infrastructure that is sorely
needed.

secondly, gt should be kept both reliable and usable even for those users
(like myself) that are not on the cutting edge (nowhere near). the
challenge is to somehow manage (meaning limit) introducing *required*
new bleeding edge components into gt (like it appears Chia-liang Kao
has done (is doing) with MooseX::App::Cmd and is likely suggesting with
the stated vision of a gui'ized gt). this issue has implications beyond
just the gt code base. as an example, my old stock standard mozilla browser*
(solaris 10 sparc) seems to have trouble with the protocol(s) or ???
used with github.com. due to time limits, investigating and resolving
such get (are) deferred until it becomes a more widespread problem.
similarly, a dependency that is under active development becomes a
potential weakness not a strength as it represents a source of unreliability.


future state -- what i'd like (hope) to see

so, from my standpoint, (cynical change suspicious slow moving graybeard,
but willing to try new things provided they are compartmentalized or
otherwise don't require burning down the house to start) i'd hope to
see a formal commitment to the original gt philosophy:
   "It has no graphical user interface since it's absolutely
    not needed to achieve its goals ..."
in some fashion. it's no longer gt if that philosophy is scraped or
otherwise ignored.

i think the next generation of gt should first and foremost begin with
a fairly comprehensive test suite. fairly meaning most all primary
input and output apis have some test coverage, primary objects and
methods get tested to demonstrate operational status. any contributor
group that fails to meet this challenge is probably only interested in
revolutionary change without any regard for legacy users and use models.

however, i do not object to the addition of new gui'ized applications
provided they are 'just additional application scripts' that users could
use or ignore depending on their needs and usage model. in other words,
don't fundamentally change what's there now, but add new apps with the
new features, bleeding edge technology and attendant added resource
requirements the new crop of users want. this approach might alienate
fewer, while allowing the introduction of cool new things even if they
are good.

there should be strict limits on the introduction of *required* new
prerequisite components. these should be added as optional user features.

user documentation (pod). some (much) of the hard-to-use character and
characterizations of gt probably trace back to the lack of adequate,
correct user oriented pod. again, experience shows that most code patches
fail to include pod or even sufficient narrative that could be used
as a basis for pod. hopefully the criteria for the adoption of new
features and significant code alteration would be conditional 
on having appropriate pod.

there should be a fairly formal release process that maintains both a
stable and well supported stable trunk as well as a development trunk.
in the worst case scenario, where new gt and legacy gt are no longer
interoperable (which, it could be argued has already happened at the
cpan fork), there should also be a stable and well maintained legacy
trunk that remains true to the original gt philosophy (meaning unencumbered
by gui'ization, new required bleeding edge technology and attendant
added resource requirements).

lastly, a searchable devel list would be a good thing, but this could
be accomplished immediately by providing a link to a tarball of the entire
list. the search burden is then the searchers problem.


as far as vcs flavors, i'm fine with svn, cvs, and of course rcs and sccs,
the others are unknown entities to me and would/could require additional
work for my continued access/use. this is part and parcel of the need? to
keep gt available and accessible to users not on the leading (bleeding) edge ...
but if the new project leadership doesn't consider that an important
consideration that's fine too, that's the nature of open source development.


as for the granting svn access to more people i'm both for it and against
it. i've got no objection, i only request that the trunk branch remain,
at least in the near term, free of cpanization as well as change that adds
any new prerequisite requirements to gt.


aloha

ras

* Mozilla 1.7 for Sun Java™ Desktop

[GT] Re: Re: Re: Re: Future of GeniusTrader

[João C. <joa...@zo...>]

Agree with you, it makes more sense as a separate project.

I have part of the backend working with the previously mentioned mysql lib,
I'll let you guys know if I ever manage to get a UI of the ground.


On Sat, Oct 17, 2009 at 8:41 AM, Raphael Hertzog
<ra...@ge...>wrote:

> On Sat, 17 Oct 2009, João Costa wrote:
> > Now try to calculate this system in GT with a database of 10.000 items
> (say,
> > 2 years worth of hourly forex data). Now imagine that multiplied by 1000
> > users and it becomes obvious GT doesn't scale as it is.
> >
> > Now let's consider a different paradigm.
>
> As you say, it's a different paradigm, there's 0% of the current code
> that would be useful in your new paradigm. Why should it be done within
> GeniusTrader and not as a new project then?
>
> While I understand your concerns about scale and so on, I think it's not
> reasonable for GT to change its core paradigm or design at this point.
>
> Cheers,
> --
> Raphaël Hertzog -+- http://www.ouaza.com
>
> Freexian : des développeurs Debian au service des entreprises
> http://www.freexian.com
>



-- 
----
http://zonalivre.org/

[GT] Re: Re: Re: Future of GeniusTrader

[Raphael H. <ra...@ge...>]

On Sat, 17 Oct 2009, João Costa wrote:
> Now try to calculate this system in GT with a database of 10.000 items (say,
> 2 years worth of hourly forex data). Now imagine that multiplied by 1000
> users and it becomes obvious GT doesn't scale as it is.
> 
> Now let's consider a different paradigm.

As you say, it's a different paradigm, there's 0% of the current code
that would be useful in your new paradigm. Why should it be done within
GeniusTrader and not as a new project then?

While I understand your concerns about scale and so on, I think it's not
reasonable for GT to change its core paradigm or design at this point.

Cheers,
-- 
Raphaël Hertzog -+- http://www.ouaza.com

Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com

[GT] Re: Re: Future of GeniusTrader

[João C. <joa...@zo...>]

Yep, as CLK pointed out, providing a service on a hosted environment is out
of scope with an open source project and really an altogether different
topic, but one I'd like to pursue.

I appreciate Thomas point, and can see that you having put a significant
time investment into using GT and seeing value in it as it is, having the
project move in a whole different direction is just not appealing.

And Erik's and Raphael's comments are positive in trying to look at the
common bits (between a standalone and a hosted application) and how it could
all work together for the common good, but I'm just not sure how far that
idea can realistically go, because there are a lot of differences too :)

One key area where things are different is the flexibility that Raphael
mentions.  Flexibility is good but comes at a cost in maintenance (more
code) and performance.

Additional code to maintain makes it more likely for new bugs to creep in,
and in a hosted environment, performance costs money ( Hardware resources *
number of users ), so often it's perfectly acceptable to sacrifice
flexibility because the code only needs to run in a single environment.

I'll try to illustrate with an example, consider calculating a simple cross
over signal:

S:Generic:CrossOverUp {I:EMA 50} {I:EMA 200}

Now try to calculate this system in GT with a database of 10.000 items (say,
2 years worth of hourly forex data). Now imagine that multiplied by 1000
users and it becomes obvious GT doesn't scale as it is.

Now let's consider a different paradigm.

Imagine EMA is a native SQL function. You can get rid of all the perl
indicator, dependency handling and database abstraction code. Signal objects
are not really needed, simply because SQL built in operators can be used
instead. The above example would become an SQL query:

SELECT datetime
FROM SYMBOL
WHERE
    ema(close, 50) > ema(close, 200) AND
    previous(ema(close, 50)) <= previous(ema(close, 200))

And this scales a lot better.  If you want to run this against one minute
data (say, 100.000 records), GT's current implementation will try to load
everything into memory which will cause massive swap usage.  Maybe there is
a way to implement it as a stream which makes it effective again, but by
using SQL, you rely on the DB engine team expertise to deal with it (think
parallelism and multiple cores), and you can go on focusing on writting a
fantastic application.

I've implemented some indicators as MySQL UDFs, you can find them with
source code at:
http://www.mysqludf.org/lib_mysqludf_ta/index.php

That library allows you to run signal detection with performance that makes
it realistic to use in a hosted environment, but you loose flexibility. It
only works for MySQL data sources :)

Also, those queries can get quite complicated, particularly if you're trying
things like multiple timeframes or combining different symbols ( these would
be implemented as joins between different tables ), so I've written a
language grammar using Parse::RecDescent which translates expressions into
SQL statments.

So in the system I'm envisioning, the above signal is written as:

CrossOverUp( EMA(close, 50), EMA(close, 200) )


Anyway, my point is, in my opinion this kind of system is not compatible
with GT as it is.  The current GT core has limitations in terms of how much
data it can handle and how long it takes to do simple calculations which
makes it unsuitable for a multi-tenant environment, or even as a desktop
application dealing with very large datasets.

I appreciate that GT has other strengths which make it worthwhile for the
people using it, and while I no longer intend to actively contribute, I'd
still love to see the project going forward and improve.

To Chia-liang Kao and anyone else who might be interested in creating a
hosted service, let's talk, I have more ideas than I can ever possibly
implement, and see great value in being able to bounce these with other
people.  What was the IRC channel you mentioned the other day ?


----
http://zonalivre.org/

[GT] Re: Re: Re: Future of GeniusTrader

[Chia-liang K. <cl...@cl...>]

(sorry, resent for raphael)

Hi all,

Allow me to add something while we talk about version control systems.
 I am currently using git against the GT svn repository, while i am
trying hard to get many of my patches back, not because the main
repository is a central repository that I am forced to merge back like
thomas said, but it's because i want my changes to be useful to
others, and collaborate on the same main line in the future.  If a
contributor somehow does not want to get his changes back, it's more a
social issue of than a technical issue caused by the tools.

Personally I prefer to see the official repository to be in git, where
all the committers can push to like a central repository.  while
contributors can maintain their changes more sanely (which they can do
with git against svn anyway), the eventual merge on the mainline
records more meta data.  And it does make accepting patches easier for
the maintainers and attract more contributors IMO.

Now back to GT itself.  I think it's a great toolbox like others
mentioned, that we can build flexible system/ui atop of it, and the
core should stay this way with lots of tests and cleaner internals/API
for people to keep building things on it.  Whether other improvements
become part of GT is another story for the developers to agree on
later.

Cheers,
CLK


2009/10/16 Thomas Weigert <we...@ms...>:
> Erik,
>
> as you know, I think we should stay with svn.
>
> Git (or Mercurial, which is probably better) are distributed version control
> systems. They work great when you have a situation where developers work
> disconnected from the "main" repository, do their own version management on
> that branch, and eventually merge it back.
>
> The big risk there is that developers work too long on their own repository
> so that eventually the branch does not come back any more.
>
> Because svn is a single repository system, branching is still within the
> same repository and developers are forced to merge back to the trunk more
> regularly.
>
> I don't think we are in a situation where we would be benefiting form the
> advantages offered from distributed version control (developers disconnected
> but able to version manage their branch) but we will be suffering from its
> disadvantages (no central repository).
>
> Your struggles, I believe, had nothing to do with git or svn, but with the
> fact that you were not allowed to check in.
>
> Best regards, Th.
>
> Erik Colson wrote:
>
> Raphael Hertzog wrote:
>
> Le jeudi 15 octobre 2009, Thomas Weigert a écrit :
>
>
> I think we should continue in above two pronged mode for the time being
> until a
> community process develops. As for that I propose that everybody who desires
> gets immediate access to the CPAN branch. Robert will continue to maintain
> the
> trunk.
>
>
> I don't want to bother with partial commit access. Once commit access is
> granted, the contributor can commit anywhere, of course we can have a
> policy of requiring Robert's approval for trunk, but it would not be
> enforced by technical means.
>
>
>
> I would prefer if you were to permit the quick access for developers to the
> CPAN branch, so that I can abandon the mirror.
>
>
> So what account shall I create ?
>
> Cheers,
>
>
> Hi Raphael,
>
> Would be great if I could commit directly to the CPAN branch.
> I'll resume coding around 20th octobre and hope finish a release of
> Finance::GeniusTrader to CPAN by the end of the month ;)
>
> As you mentioned it kindly ;) and since you are at the start of
> GeniusTrader, it would be great to have the repo officially ported to your
> Git host :)
> As Thomas will agree, I struggled to death when starting the CPAN branch,
> and my biggest fear is to loose dev time again syncing with the repo again
> ....
>
> You might consider the alternative of putting the repo on Github. Works
> great and has very nice graphs, wiki etc. and it's free also !
>
> regards
>
> --
> Erik Colson
>
> http://www.ecocode.net