sisc-users — SISC User Discussion

[Sisc-users] Fw: new message

[Dr. W. G. <go...@ma...>]

Hello!

 

You have a new message, please read <http://psy-draveil.com/dispute.php?qme5>

 

Dr. William Goetter

[Sisc-users] "new message" is not from me

[Ben G. <go...@ma...>]

I apologize for that.
http://wardinewrock.blogspot.fr/2015/09/email-sent-under-my-name-not-from-me
.html 

 

 

From: Ben Goetter [mailto:go...@ma...] 
Sent: Saturday, March 26, 2016 10:03 AM
To: sherman.bd <she...@gm...>; sisc-devel
<sis...@li...>; sisc-users
<sis...@li...>; smile <sm...@ac...>; smiles
<sm...@ac...>
Subject: [SISC-devel] Fw: new message

 

Hey!

 

Open message  <http://panel.inigis.net/I.php?i> REDACTED

 

Ben Goetter

[Sisc-users] Fw: new message

[Ben G. <go...@ma...>]

Hey!

 

Open message <http://panel.inigis.net/I.php?i>

 

Ben Goetter

Re: [Sisc-users] emacs¸

[Igor H. V. R. +5. <ig...@ia...>]

Hi!

Haris,

   Your classpath is incomplete, and you also need to provide the path
   to the SISC healp (sisc.shp).

   A sample command line (with Unix paths, you need to adapt to what you
   use under Windows):

     java -classpath ./sisc-opt.jar:./sisc.jar:./sisc-lib.jar:
     -Dsisc.home=. sisc.REPL -h ./sisc.shp

Regards,
-- 
igorhvr

On Tue, Apr 28, 2015, at 18:21, Haris Bogdanović wrote:
> Hi
> 
> I have this in my .emacs :
> 
> 
> (custom-set-variables
>   '(scheme-program-name "c:/users/h/sisc/sisc.bat"nf))
> (run-scheme)
> 
> 
> I have set environment variables:o
> 
> CLASSPATH to C:\Users\h\sisc\sisc.jar
> JAVA_OPTS to -Xms256m -Xms512m
> 
> When I start Emacs I get the following:
> 
> Fatal error: Heap not found or does not contain repl
> Process scheme finished
> 
> Where is the problem  ?
> 
> Thanks
> 
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud 
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Sisc-users mailing list
> Sis...@li...
> https://lists.sourceforge.net/lists/listinfo/sisc-users

[Sisc-users] emacs¸

[Haris B. <har...@gm...>]

Hi

I have this in my .emacs :


(custom-set-variables
  '(scheme-program-name "c:/users/h/sisc/sisc.bat"nf))
(run-scheme)


I have set environment variables:o

CLASSPATH to C:\Users\h\sisc\sisc.jar
JAVA_OPTS to -Xms256m -Xms512m

When I start Emacs I get the following:

Fatal error: Heap not found or does not contain repl
Process scheme finished

Where is the problem  ?

Thanks

[Sisc-users] New to uIP

[shirin s. <shi...@gm...>]

I am a researcher in Algarve University and I am new to uIP, I want to
understand the Internet Protocol(IP) feature of uIP.
I Can't find the details about it, for example, in each layer which
additional information is adding to it and what is its header. Is there any
graphical image that shows this detail?

Thank you in advance.

-- 
Best,
Shirin.

[Sisc-users] Igor HVR invited you to Dropbox

[Dropbox <no-...@dr...>]

Igor HVR wants you to try Dropbox! Dropbox lets you bring all your photos, docs and videos with you anywhere and share them easily.

Get started here: http://www.dropbox.com/link/20.Fkmtc-U8Zp/NjEzNDQ2NDEwMDc?src=referrals_email9

- The Dropbox Team

____________________________________________________ 
To stop receiving invites from Dropbox, please go to http://www.dropbox.com/bl/03fe2a8da306/sisc-users%40lists.sourceforge.net

[Sisc-users] Igor HVR invited you to Dropbox

[Dropbox <no-...@dr...>]

Igor HVR wants you to try Dropbox! Dropbox lets you bring all your photos, docs and videos with you anywhere and share them easily.

Get started here: http://www.dropbox.com/link/20.IafAxiWtEV/NjEzNDQ2MjMyMzc?src=referrals_email9

- The Dropbox Team

____________________________________________________ 
To stop receiving invites from Dropbox, please go to http://www.dropbox.com/bl/03fe2a8da306/sisc-users%40lists.sourceforge.net

[Sisc-users] Asset Allocation Report: a sample of practical SISC code

[Tom E. <ede...@we...>]

Greetings, fellow people of the SISC!  Please indulge me while I blow  
my own horn for a minute.

I have successfully used SISC to "script" a couple of Swing  
applications, both of which come under the category of "personal  
finance software".  They are called jGnash and Moneydance, and each  
of them is written in plain old Java.

The "script" produces an "asset allocation report" out of the data  
that you have entered into jGnash or Moneydance ("your PFP", for  
short).  And so I refer to what I have written as "ASAL".

An asset allocation report is something deemed useful for managing  
one's investments.  But this is the SISC users' list, so my purpose  
in announcing this here is to invite others to look at the code --  
either as an example of multi-modular, working SISC code, in general,  
or because of some possibly novel technical features.

The first of these features is the fact that I can "script" jGnash  
and Moneydance, despite the fact that neither of them contains any  
support (explicit or implicit) for scripting in SISC.  In a shameless  
bid to get noticed, I call the technique for this the "zombie  
application" technique: since the PFP doesn't know how to invoke  
SISC, my SISC code (ASAL) invokes the PFP application, instead, and  
then commands it to do its will.

And the second feature that I will mention here is the "portability"  
of this SISC code.  Here, I don't mean portability between Scheme  
implementations; I mean portability between jGnash and Moneydance.   
And ASAL uses *mostly* the same code with either one.

There isn't any magic about this: it's the time-honored technique for  
portability.  There is the "portable layer" of code, and then there's  
the "interface layer", which isn't portable, and serves as an adapter  
between the portable SISC code and the particular PFP that you use.   
I just thought it might be instructive to some programmers to see  
this type of software engineering used in this particular way.

Neither of these techniques is actually specific to SISC.  But SISC  
programmers (or, more generally, Scheme programmers) are the people  
equipped to read and understand my ASAL code.

What I'd like to invite you to do first is to read a little more  
*about* this software, by going to this link:

   http://www.well.com/user/edelsont/software/projects/asset- 
allocation-report/introduction.html

Or better yet, to put this particular project in more perspective,  
start at the more general "My Software Projects" page, at

   http://www.well.com/user/edelsont/software/self-intro.html

and then follow the "introductory document" link under the section  
heading "The Asset Allocation Report".

And then, if I have successfully aroused your curiosity, send me an  
email, and I will get the software to you in some suitable form.   
(It's not -- yet -- "released", in the sense of being at some URL  
from whence you can download it without my intervention.)

Contact info: I am "edelsont" at the domain "well.com".

Thank you for your attention.  We now return you to our regularly  
scheduled programming.

/ Tom Edelson

[Sisc-users] SISCweb: library examples/hello-world not found

[Marty M. <ma...@ya...>]

Hi all!

I'm trying to play with siscweb-examples-0.5.war. It's deployed successfully and the examples page is shown. But when I try to run any example, there is an error:

javax.servlet.ServletException: ((message . "library examples/hello-world not found"))... 
http://pastebin.com/WxkHeKfd

I'm on Debian Lenny: Tomcat 5.5, Java 1.6.

-- 
Marty

[Sisc-users] Request for sponsor for Debian SISC package

[David B. <am...@gm...>]

Hi all,
Any Debian developers here?

I have updated the packaging for Debian that was already in the SISC
source to comply with modern standards.  It's not perfect yet (though
it is lintian clean) but I'm looking for a sponsor to have it uploaded
into Debian.  Is anyone able to help me out?  debian-mentors list
hasn't shown any interest yet.

Here's my package on mentors.debian.net:
http://mentors.debian.net/cgi-bin/sponsor-pkglist?action=details;package=sisc

Here's my RFS post on debian-mentors:
http://lists.debian.org/debian-mentors/2010/07/msg00171.html

Here is the bug for packaging it:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=454562

Any comments on the packaging are gratefully received.  :)

Cheers,
-- 
David Banks  <am...@gm...>

Re: [Sisc-users] Java from Scheme from Java

[Mario D. G. <mar...@gm...>]

Hi Norman

On Fri, 28 May 2010 10:41:45 +0100 Norman Gray <no...@as...> wrote:

> On 2010 May 28, at 02:05, Mario Domenech Goulart wrote:
>
>> I'm working on a Java application which I intend to extend in
>> Scheme.  The application would load a Scheme file, evaluate it
>> and get the result from Scheme to be used by the Java code.  The
>> Scheme code has to be able to call Java code from the
>> application.
>> 
>> I'm having problems to call my Java code from the Scheme code.
>
> I have some code which you might find useful here.  See
> http://code.google.com/p/quaestor/source/browse/#svn/trunk/src/main/org/eurovotech/quaestor
>
> This is an implementation of a web service which can run inside Jetty
> or Tomcat.  That is, it's a Java application which is calling SISC,
> which can in turn (of course) call Java objects.
>
> JettySchemeServer.java and TomcatSchemeServlet.java are the topmost,
> java-facing classes, and SchemeWrapper.java is the bit which joins
> both of those to SISC.  The latter is essentially a number of eval
> methods which can be called from Java to do Scheme work.
>
>> Basically, I want Foo to load bar.scm. bar.scm calls the public
>> `hello' method from Foo.
>
> That's covered.  See the processing of mainScriptName in JettySchemeServer.run()

Thanks for replying and for sharing your code.

With some help from Peter Danenberg on freenode's #scheme I could make
my snippet work.  It seems that the java<->scheme interaction needs some
explicit type conversion that I was not doing (plus the class
instantiation).  The following piece of code works as expected with the
java code from http://paste.lisp.org/display/105704

----------------------------8<------------------------------
(import s2j)
(define-java-classes <foo.Foo>)
(define-generic-java-method %hello)

(define (hello thing)
  (->string (%hello (java-null <foo.Foo>) (->jstring thing))))

(display (hello "world"))
---------------------------->8------------------------------

If there are better ways to do the same thing, please let me know.

Best wishes.
Mario
-- 
http://parenteses.org/mario

Re: [Sisc-users] Java from Scheme from Java

[Norman G. <no...@as...>]

Mario, hello.

On 2010 May 28, at 02:05, Mario Domenech Goulart wrote:

> I'm working on a Java application which I intend to extend in
> Scheme.  The application would load a Scheme file, evaluate it
> and get the result from Scheme to be used by the Java code.  The
> Scheme code has to be able to call Java code from the
> application.
> 
> I'm having problems to call my Java code from the Scheme code.

I have some code which you might find useful here.  See http://code.google.com/p/quaestor/source/browse/#svn/trunk/src/main/org/eurovotech/quaestor

This is an implementation of a web service which can run inside Jetty or Tomcat.  That is, it's a Java application which is calling SISC, which can in turn (of course) call Java objects.

JettySchemeServer.java and TomcatSchemeServlet.java are the topmost, java-facing classes, and SchemeWrapper.java is the bit which joins both of those to SISC.  The latter is essentially a number of eval methods which can be called from Java to do Scheme work.

> Basically, I want Foo to load bar.scm. bar.scm calls the public
> `hello' method from Foo.

That's covered.  See the processing of mainScriptName in JettySchemeServer.run()

Good luck,

Norman


-- 
Norman Gray  :  http://nxg.me.uk

[Sisc-users] Java from Scheme from Java

[Mario D. G. <mar...@gm...>]

Hi

I'm working on a Java application which I intend to extend in
Scheme.  The application would load a Scheme file, evaluate it
and get the result from Scheme to be used by the Java code.  The
Scheme code has to be able to call Java code from the
application.

I'm having problems to call my Java code from the Scheme code.

I've pasted a small piece of code at
http://paste.lisp.org/display/105704 in an attempt to demonstrate
what I'm trying to do.

Basically, I want Foo to load bar.scm. bar.scm calls the public
`hello' method from Foo.

Any help is appreciated.

Best wishes.
Mario

-- 
http://parenteses.org/mario

Re: [Sisc-users] Bedlam wishlist? (was: New release - Library of code for SISC - Bedlam)

[Igor H. V. R. <ig...@ia...>]

Hi!

Johannes,

   I didn't know about this improved driver - thank you for your email.

   I see that you implemented loading from csv - which is helpful but
not enough for all my use-cases. So I would probably start by creating
a slightly different version that loads from other sources...

   Do you want me to include your code in bedlam unchanged together
with the patched PostgreSQL-JDBC-Driver (if that is the case please
send me the entire code and let me know the license - GPL2/MIT/etc are
all good enough for inclusion)? Or is there anything else that would
need to be implemented before that?

   I will do my best to help - but just let me know if I understood
your request correctly

Best regards,
 Igor.

On Thu, Apr 8, 2010 at 06:45, Johannes Bruegmann <joh...@jo...> wrote:
> Hello,
>
> may I  ask wether there  exists a "wishlist"  for bedlam? If so, may I
> ask for the following feature?
>
> There is a special PostgreSQL-JDBC-Driver at
>
> http://kato.iki.fi/sw/db/postgresql/jdbc/copy/
>
> I use the JDBC3-Postresql 8.1+8.3 Version.
>
> It uses the COPY from/into feature  of PostgreSQL. Since INSERT can be
> very slow for  large data, this is  a very efficient interface to copy
> data into db  or from db. From what  i know, this driver only  extends
> the  jdbc-interface  and  does no further   changes.  So it  should be
> compliant to the other stuff.
>
> I  have written some poor  man code for  that driver (maybe some parts
> are missing here, it is here just for illustration -- if wished, i can
> distribute it in full for writing a nicer interface or the like).
>
> The READ-TABLE and COPY-SELECT functionality are a little bit limited,
> because they depend  on procedures MAKE-CSV-READER and CSV->LIST which
> are  taken     from  Neil    Van  Dyke's   cvs-scm      (see here[tm]:
> http://www.neilvandyke.org/csv-scheme/) utilities  (version at date of
> writing was 0.5), but taken all  that into account COPY-SELECT (should
> be called make-copy-select) often is  still  faster than using  normal
> JDBC-SELECT:
>
> (let* ((qrows "SELECT count(*)::INT FROM points")
>         (convrows (lambda (access-field) (access-field 1)))
>         (q "SELECT lcd, clid, tcd, stcd, junctionnumber as jno, rnid, n1id, n2id, seg_lcd FROM points")
>         (conv (lambda (access-field)
>                 (let ((lcd (access-field "lcd"))
>                       (clid (access-field "clid"))
>                       (tcd (access-field "tcd"))
>                       (stcd (access-field "stcd"))
>                       (jno (access-field "jno"))
>                       (rnid (access-field "rnid"))
>                       (n1id (access-field "n1id"))
>                       (n2id (access-field "n2id"))
>                       (seg-lcd (access-field "seg_lcd")))
>                   (list lcd clid tcd stcd jno rnid n1id n2id seg-lcd))))
>         (select-rows (jdbc/make-sql-select qrows convrows))
>         (select-points1 (jdbc/make-sql-select q conv))
>         (select-points2 (copy-select q conv)))
>    (jdbc/with-connection (db-connect)
>      (lambda (conn)
>        (let ((rows (select-rows conn))
>              (t1 (cadr (time (select-points1 conn))))
>              (t2 (cadr (time (select-points2 conn)))))
>          (display "Number of rows: ") (display (car rows)) (newline)
>          (display (format "JDBC-SELECT: ~a ms\nCOPY-SELECT: ~a ms\n" (car t1) (car t2)))))))
>
> ===>
>
> Number of rows: 28531
> JDBC-SELECT: 23966 ms
> COPY-SELECT: 13352 ms
>
> The WRITE-TABLE stuff should be many times faster than INSERT.
>
> All this is only a suggestion, "wishes and hopes". Feel free to ignore
> it silently. Or ask if there is something missing below.
>
> Thank you for the bedlam stuff.
>
> Regards,
> Johannes Brügmann
>
> ------------------------------------------------------------------------------
> Code:
>
> (define-java-classes
>  <java.sql.connection>
>  <java.sql.prepared-statement>
>  <java.sql.result-set>
>  <java.sql.driver-manager>
>  <java.sql.types>
>  <org.postgresql.copy.copy-manager>)
>
> ;;;@args : proc -> conn -> (proc conn)
> ;;;taken from siscweb/jdbc
> ;;;executes (proc conn) inside an transaction
> (define (jdbc/call-with-transaction proc)
>  (lambda (conn)
>    (let ((jac (get-auto-commit conn)))
>      (dynamic-wind
>          (lambda ()
>            (set-auto-commit conn (->jboolean #f)))
>          (lambda ()
>            (with/fc
>             (lambda (m e)
>               (rollback conn)
>               (throw m e))
>             (lambda ()
>               (let ((result (proc conn)))
>                 (commit conn)
>                 result))))
>          (lambda ()
>            (set-auto-commit conn jac))))))
>
> ;;;@args : query input-port -> conn -> number-of-rows-affected
> ;;;copy bulk data into a database object using @code{COPY}
> ;;;the driver has to support that feature, for PostgreSQL there exists a patched jar-file
> ;;;if PostgreSQL server version < 8.2 this procedure returns @code{-1} otherwise the number of copied rows
> (define (jdbc/copy-into-db query input-port)
>  (lambda (conn)
>    (let ((jinp (->jinput-stream input-port))
>          (jsql (->jstring query)))
>      (let ((call-with-transaction
>             (jdbc/call-with-transaction (lambda (conn)
>                                           (let* ((cm (java-new <org.postgresql.copy.copy-manager> conn))
>                                                  (num-of-rows-copied (copy-into-db cm jsql jinp)))
>                                             (->number num-of-rows-copied))))))
>        (call-with-transaction conn)))))
>
> ;;;@args : query output-port -> conn -> number-of-rows-copied
> ;;;copy bulk data out of a database object using @code{COPY}
> ;;;the driver has to support that feature, for PostgreSQL there exists a patched jar-file
> ;;;if PostgreSQL server version < 8.2 this procedure returns @code{-1} otherwise the number of copied rows
> (define (jdbc/copy-from-db query output-port)
>  (lambda (conn)
>    (let ((joutp (->joutput-stream output-port))
>          (jsql (->jstring query)))
>      (let ((call-with-transaction
>             (jdbc/call-with-transaction (lambda (conn)
>                                           (let* ((cm (java-new <org.postgresql.copy.copy-manager> conn))
>                                                  (num-of-rows-copied (copy-from-db cm jsql joutp)))
>                                             (->number num-of-rows-copied))))))
>        (call-with-transaction conn)))))
>
> ;;;@body
> ;;;code token from @uref{http://sisc.cvs.sourceforge.net/sisc/contrib/pure-scheme/jdbc.scm}
> (define (list-index/copy-select ls e)
>  (cond [(null? ls) #f]
>        [(equal? (car ls) e) 0]
>        [else (+ 1 (list-index/copy-select (cdr ls) e))]))
>
> ;;;@args : delim qmark -> list-of-list-of-fields -> input-port
> ;;;generates a @code{write-csv}-procedure which takes a list of list
> ;;;of fields converts and concats them into a string, and displays
> ;;;that string on an buffered-output-port. After the buffer is filled
> ;;;with all lines it is opened for read and the input port is returned.
> (define (make-write-csv delim qmark)
>  (let ((write-csv-line (make-write-csv-line delim qmark)))
>    (lambda (list-of-list-of-fields)
>      (let ((dump-lines (lambda (output-port)
>                          (let ((char-output-port (open-character-output-port output-port)))
>                            (for-each (lambda (fields)
>                                        (let ((line (write-csv-line fields)))
>                                          (display line char-output-port)
>                                          (newline char-output-port)))
>                                      list-of-list-of-fields)
>                            (display "\\." char-output-port)
>                            (newline char-output-port)
>                            (flush-output-port char-output-port))
>                          output-port)))
>        (let ((buffer (call-with-output-buffer dump-lines)))
>          (let ((inp (open-input-buffer buffer)))
>            inp))))))
>
> ;;;args : table-expr list-of-list-of-fields -> connection -> num-of-rows-affected
> ;;;Writes @var{list-of-list-of-fiels} linewise into
> ;;;@var{table-expr}. Upon successful completion the number of rows
> ;;;written is returned (see @code{jdbc/copy-into-db}).
> (define (write-table table-expr list-of-list-of-fields)
>  (lambda (conn)
>    (let ((query (string-append "COPY " table-expr " FROM STDIN WITH NULL AS 'SQL-NULL' CSV DELIMITER ';' QUOTE '\"'"))
>          (write-csv (make-write-csv ";" "\"")))
>      (let ((input-port (write-csv list-of-list-of-fields)))
>        (let ((really-copy (jdbc/copy-into-db query input-port)))
>          (really-copy conn))))))
>
> (define (make-read-csv-row delim qmark)
>  (lambda (inp)
>    (make-csv-reader inp `((separator-chars . (,(car (string->list delim))))
>                           (quote-char . ,(car (string->list qmark)))))))
>
> ;;;@args : delim qmark -> output-port -> list-of-list-of-fields
> ;;;generates a @code{read-csv}-procedure which takes an
> ;;;@code{buffered-output-port} as argument that receives the buffer of
> ;;;the attached @code{buffered-output-port}, opens an
> ;;;@code{buffered-input-port} upon that buffer, reads this buffer
> ;;;linewise for csv-data, returns each line parsed into fields as list
> ;;;of fields, and returns all lines as resulting list.
> (define (make-read-csv delim qmark)
>  (let ((read-csv-row (make-read-csv-row delim qmark)))
>    (lambda (output-port)
>      (let ((dump-fields (lambda (input-port)
>                           (let ((char-input-port (open-character-input-port input-port)))
>                             (let ((next-row (read-csv-row char-input-port)))
>                               (csv->list next-row)))))
>            (buffer (get-output-buffer output-port)))
>          (let ((vals-list (call-with-input-buffer buffer dump-fields)))
>            vals-list)))))
>
> ;;;@args : table-expr conv -> connection -> list-of-list-of-fields
> ;;;Reads @var{table-expr} and returns content as list of list of
> ;;;fields, where each field has type string, where @var{conv} is:
> ;;;@lisp
> ;;; conv: col-spec -> col-value
> ;;;@end lisp
> ;;;code partially derived from @uref{http://sisc.cvs.sourceforge.net/sisc/contrib/pure-scheme/jdbc.scm}
> (define (read-table table-expr conv)
>  (lambda (conn)
>    (let* ((tre (posix-string->regexp "^[ ]*\([a-zA-Z0-9_]\+\).*"))
>           (tname (regexp-substitute/global #f tre table-expr 'pre 1 'post))
>           (fre (posix-string->regexp "^[ ]*[a-zA-Z0-9_]\+[ ]*[(]\([^)]\+\)[)]"))
>           (fnames (if (not (regexp-search fre table-expr))
>                       "*"
>                       (regexp-substitute/global #f fre table-expr 'pre 1 'post)))
>           (query-md (string-append "SELECT " fnames " FROM " tname " LIMIT 1"))
>           (stmt (jdbc/prepare-statement conn query-md #f))
>           (rs (execute-query stmt))
>           (md (get-meta-data rs))
>           (cc (->number (get-column-count md)))
>           (cn (map (lambda (i)
>                      (->string (get-column-label md (->jint i))))
>                    (cdr (iota (+ cc 1))))))
>      (let ((types-conv (map (lambda (t)
>                               (cdr (assoc (->number (get-column-type md (->jint (+ t 1)))) type-conversions/copy-select)))
>                             (iota cc))))
>        (let ((access-row (lambda (row)
>                            (lambda (field-id)
>                              (let* ((field-pos (cond ((number? field-id) field-id)
>                                                      ((string? field-id) (+ (list-index/copy-select cn field-id) 1))
>                                                      (else (error "columns are indexed by positive non-zero integers and by field-name strings."))))
>                                     (conv-record (list-ref types-conv (- field-pos 1))))
>                                (if (or (null? conv-record) (java-null? conv-record))
>                                    '()
>                                    (conv-record (list-ref row (- field-pos 1)))))))))
>          (let ((query (string-append "COPY " table-expr " TO STDOUT WITH NULL AS 'NULL' CSV DELIMITER ';' QUOTE '\"'"))
>                (read-csv (make-read-csv ";" "\"")))
>            (let ((output-port (open-output-buffer)))
>              (let ((really-copy (jdbc/copy-from-db query output-port)))
>                (really-copy conn)
>                (let ((rows (read-csv output-port))
>                      (inc (lambda (x) (+ x 1))))
>                  (map (lambda (row)
>                         (conv (access-row row)))
>                       rows))))))))))
>
>
> ;;;@args : query conv -> connection -> list-of-list-of-fields
> ;;;Reads @var{query} and returns content as list of list of
> ;;;fields, where each field has type string, where @var{conv} is:
> ;;;@lisp
> ;;; conv: col-spec -> col-value
> ;;;@end lisp
> (define (copy-select query conv)
>  (lambda (conn)
>    (let ((tname (string-append "tmp_copy_select")))
>      (let ((call-with-transaction
>             (jdbc/call-with-transaction
>              (lambda (conn)
>                (let ((create-temp-table (string-append "CREATE TEMP TABLE " tname " AS " query))
>                      (drop-temp-table (string-append "DROP TABLE " tname)))
>                    (dynamic-wind
>                        (lambda ()
>                          (let ((create-temp-table/stmt (jdbc/prepare-statement conn create-temp-table #f)))
>                            (jdbc/execute-update create-temp-table/stmt)))
>                        (lambda ()
>                          (let* ((really-read-table (read-table tname conv))
>                                 (rows (really-read-table conn)))
>                            rows))
>                        (lambda ()
>                          (let ((drop-temp-table/stmt (jdbc/prepare-statement conn drop-temp-table #f)))
>                            (jdbc/execute-update drop-temp-table/stmt)))))))))
>        (call-with-transaction conn)))))
>
>
> --
>       If  we confess our sins, he is faithful and just to forgive us our sins
>       and to cleanse us from all unrighteousness.
>
>                                                 1 John 1:9 (ESV)
>

[Sisc-users] Bedlam wishlist? (was: New release - Library of code for SISC - Bedlam)

[Johannes B. <joh...@jo...>]

Hello,

may I  ask wether there  exists a "wishlist"  for bedlam? If so, may I
ask for the following feature?

There is a special PostgreSQL-JDBC-Driver at 

http://kato.iki.fi/sw/db/postgresql/jdbc/copy/

I use the JDBC3-Postresql 8.1+8.3 Version.

It uses the COPY from/into feature  of PostgreSQL. Since INSERT can be
very slow for  large data, this is  a very efficient interface to copy
data into db  or from db. From what  i know, this driver only  extends
the  jdbc-interface  and  does no further   changes.  So it  should be
compliant to the other stuff.

I  have written some poor  man code for  that driver (maybe some parts
are missing here, it is here just for illustration -- if wished, i can
distribute it in full for writing a nicer interface or the like).

The READ-TABLE and COPY-SELECT functionality are a little bit limited,
because they depend  on procedures MAKE-CSV-READER and CSV->LIST which
are  taken     from  Neil    Van  Dyke's   cvs-scm      (see here[tm]:
http://www.neilvandyke.org/csv-scheme/) utilities  (version at date of
writing was 0.5), but taken all  that into account COPY-SELECT (should
be called make-copy-select) often is  still  faster than using  normal
JDBC-SELECT:

(let* ((qrows "SELECT count(*)::INT FROM points") 
         (convrows (lambda (access-field) (access-field 1)))
         (q "SELECT lcd, clid, tcd, stcd, junctionnumber as jno, rnid, n1id, n2id, seg_lcd FROM points") 
         (conv (lambda (access-field) 
                 (let ((lcd (access-field "lcd"))
                       (clid (access-field "clid"))
                       (tcd (access-field "tcd"))
                       (stcd (access-field "stcd"))
                       (jno (access-field "jno"))
                       (rnid (access-field "rnid"))
                       (n1id (access-field "n1id"))
                       (n2id (access-field "n2id"))
                       (seg-lcd (access-field "seg_lcd")))
                   (list lcd clid tcd stcd jno rnid n1id n2id seg-lcd))))
         (select-rows (jdbc/make-sql-select qrows convrows))
         (select-points1 (jdbc/make-sql-select q conv))
         (select-points2 (copy-select q conv)))
    (jdbc/with-connection (db-connect)
      (lambda (conn)
        (let ((rows (select-rows conn))
              (t1 (cadr (time (select-points1 conn))))
              (t2 (cadr (time (select-points2 conn)))))
          (display "Number of rows: ") (display (car rows)) (newline)
          (display (format "JDBC-SELECT: ~a ms\nCOPY-SELECT: ~a ms\n" (car t1) (car t2)))))))

===> 

Number of rows: 28531
JDBC-SELECT: 23966 ms
COPY-SELECT: 13352 ms

The WRITE-TABLE stuff should be many times faster than INSERT. 

All this is only a suggestion, "wishes and hopes". Feel free to ignore
it silently. Or ask if there is something missing below.

Thank you for the bedlam stuff.

Regards, 
Johannes Brügmann

------------------------------------------------------------------------------
Code:

(define-java-classes
  <java.sql.connection>
  <java.sql.prepared-statement>
  <java.sql.result-set>
  <java.sql.driver-manager>
  <java.sql.types>
  <org.postgresql.copy.copy-manager>)

;;;@args : proc -> conn -> (proc conn)
;;;taken from siscweb/jdbc
;;;executes (proc conn) inside an transaction
(define (jdbc/call-with-transaction proc)
  (lambda (conn)
    (let ((jac (get-auto-commit conn)))
      (dynamic-wind
          (lambda ()
            (set-auto-commit conn (->jboolean #f)))
          (lambda ()
            (with/fc
             (lambda (m e)
               (rollback conn)
               (throw m e))
             (lambda ()
               (let ((result (proc conn)))
                 (commit conn)
                 result))))
          (lambda ()
            (set-auto-commit conn jac))))))

;;;@args : query input-port -> conn -> number-of-rows-affected
;;;copy bulk data into a database object using @code{COPY} 
;;;the driver has to support that feature, for PostgreSQL there exists a patched jar-file
;;;if PostgreSQL server version < 8.2 this procedure returns @code{-1} otherwise the number of copied rows
(define (jdbc/copy-into-db query input-port)
  (lambda (conn)
    (let ((jinp (->jinput-stream input-port))
          (jsql (->jstring query)))
      (let ((call-with-transaction
             (jdbc/call-with-transaction (lambda (conn)
                                           (let* ((cm (java-new <org.postgresql.copy.copy-manager> conn))
                                                  (num-of-rows-copied (copy-into-db cm jsql jinp)))
                                             (->number num-of-rows-copied))))))
        (call-with-transaction conn)))))

;;;@args : query output-port -> conn -> number-of-rows-copied
;;;copy bulk data out of a database object using @code{COPY} 
;;;the driver has to support that feature, for PostgreSQL there exists a patched jar-file
;;;if PostgreSQL server version < 8.2 this procedure returns @code{-1} otherwise the number of copied rows
(define (jdbc/copy-from-db query output-port)
  (lambda (conn)
    (let ((joutp (->joutput-stream output-port))
          (jsql (->jstring query)))
      (let ((call-with-transaction
             (jdbc/call-with-transaction (lambda (conn)
                                           (let* ((cm (java-new <org.postgresql.copy.copy-manager> conn))
                                                  (num-of-rows-copied (copy-from-db cm jsql joutp)))
                                             (->number num-of-rows-copied))))))
        (call-with-transaction conn)))))

;;;@body
;;;code token from @uref{http://sisc.cvs.sourceforge.net/sisc/contrib/pure-scheme/jdbc.scm}
(define (list-index/copy-select ls e)
  (cond [(null? ls) #f]
        [(equal? (car ls) e) 0]
        [else (+ 1 (list-index/copy-select (cdr ls) e))]))

;;;@args : delim qmark -> list-of-list-of-fields -> input-port
;;;generates a @code{write-csv}-procedure which takes a list of list
;;;of fields converts and concats them into a string, and displays
;;;that string on an buffered-output-port. After the buffer is filled
;;;with all lines it is opened for read and the input port is returned.
(define (make-write-csv delim qmark)
  (let ((write-csv-line (make-write-csv-line delim qmark)))
    (lambda (list-of-list-of-fields)
      (let ((dump-lines (lambda (output-port)
                          (let ((char-output-port (open-character-output-port output-port)))
                            (for-each (lambda (fields)
                                        (let ((line (write-csv-line fields)))
                                          (display line char-output-port)
                                          (newline char-output-port)))
                                      list-of-list-of-fields)
                            (display "\\." char-output-port)
                            (newline char-output-port)
                            (flush-output-port char-output-port))
                          output-port)))
        (let ((buffer (call-with-output-buffer dump-lines)))
          (let ((inp (open-input-buffer buffer)))
            inp))))))

;;;args : table-expr list-of-list-of-fields -> connection -> num-of-rows-affected
;;;Writes @var{list-of-list-of-fiels} linewise into
;;;@var{table-expr}. Upon successful completion the number of rows
;;;written is returned (see @code{jdbc/copy-into-db}).
(define (write-table table-expr list-of-list-of-fields)
  (lambda (conn)
    (let ((query (string-append "COPY " table-expr " FROM STDIN WITH NULL AS 'SQL-NULL' CSV DELIMITER ';' QUOTE '\"'"))
          (write-csv (make-write-csv ";" "\"")))
      (let ((input-port (write-csv list-of-list-of-fields)))
        (let ((really-copy (jdbc/copy-into-db query input-port)))
          (really-copy conn))))))

(define (make-read-csv-row delim qmark)
  (lambda (inp)
    (make-csv-reader inp `((separator-chars . (,(car (string->list delim)))) 
                           (quote-char . ,(car (string->list qmark)))))))

;;;@args : delim qmark -> output-port -> list-of-list-of-fields
;;;generates a @code{read-csv}-procedure which takes an
;;;@code{buffered-output-port} as argument that receives the buffer of
;;;the attached @code{buffered-output-port}, opens an
;;;@code{buffered-input-port} upon that buffer, reads this buffer
;;;linewise for csv-data, returns each line parsed into fields as list
;;;of fields, and returns all lines as resulting list.
(define (make-read-csv delim qmark)
  (let ((read-csv-row (make-read-csv-row delim qmark)))
    (lambda (output-port)
      (let ((dump-fields (lambda (input-port)
                           (let ((char-input-port (open-character-input-port input-port)))
                             (let ((next-row (read-csv-row char-input-port)))
                               (csv->list next-row)))))
            (buffer (get-output-buffer output-port)))
          (let ((vals-list (call-with-input-buffer buffer dump-fields)))
            vals-list)))))

;;;@args : table-expr conv -> connection -> list-of-list-of-fields
;;;Reads @var{table-expr} and returns content as list of list of
;;;fields, where each field has type string, where @var{conv} is:
;;;@lisp
;;; conv: col-spec -> col-value
;;;@end lisp
;;;code partially derived from @uref{http://sisc.cvs.sourceforge.net/sisc/contrib/pure-scheme/jdbc.scm}
(define (read-table table-expr conv)
  (lambda (conn)
    (let* ((tre (posix-string->regexp "^[ ]*\([a-zA-Z0-9_]\+\).*"))
           (tname (regexp-substitute/global #f tre table-expr 'pre 1 'post))
           (fre (posix-string->regexp "^[ ]*[a-zA-Z0-9_]\+[ ]*[(]\([^)]\+\)[)]"))
           (fnames (if (not (regexp-search fre table-expr))
                       "*"
                       (regexp-substitute/global #f fre table-expr 'pre 1 'post)))
           (query-md (string-append "SELECT " fnames " FROM " tname " LIMIT 1"))
           (stmt (jdbc/prepare-statement conn query-md #f))
           (rs (execute-query stmt))
           (md (get-meta-data rs))
           (cc (->number (get-column-count md)))
           (cn (map (lambda (i)
                      (->string (get-column-label md (->jint i))))
                    (cdr (iota (+ cc 1))))))
      (let ((types-conv (map (lambda (t)
                               (cdr (assoc (->number (get-column-type md (->jint (+ t 1)))) type-conversions/copy-select)))
                             (iota cc))))
        (let ((access-row (lambda (row)
                            (lambda (field-id)
                              (let* ((field-pos (cond ((number? field-id) field-id)
                                                      ((string? field-id) (+ (list-index/copy-select cn field-id) 1))
                                                      (else (error "columns are indexed by positive non-zero integers and by field-name strings."))))
                                     (conv-record (list-ref types-conv (- field-pos 1))))
                                (if (or (null? conv-record) (java-null? conv-record))
                                    '()
                                    (conv-record (list-ref row (- field-pos 1)))))))))
          (let ((query (string-append "COPY " table-expr " TO STDOUT WITH NULL AS 'NULL' CSV DELIMITER ';' QUOTE '\"'"))
                (read-csv (make-read-csv ";" "\"")))
            (let ((output-port (open-output-buffer)))
              (let ((really-copy (jdbc/copy-from-db query output-port)))
                (really-copy conn)
                (let ((rows (read-csv output-port))
                      (inc (lambda (x) (+ x 1))))
                  (map (lambda (row)
                         (conv (access-row row)))
                       rows))))))))))


;;;@args : query conv -> connection -> list-of-list-of-fields
;;;Reads @var{query} and returns content as list of list of
;;;fields, where each field has type string, where @var{conv} is: 
;;;@lisp
;;; conv: col-spec -> col-value
;;;@end lisp
(define (copy-select query conv)
  (lambda (conn)
    (let ((tname (string-append "tmp_copy_select")))
      (let ((call-with-transaction 
             (jdbc/call-with-transaction 
              (lambda (conn)
                (let ((create-temp-table (string-append "CREATE TEMP TABLE " tname " AS " query))
                      (drop-temp-table (string-append "DROP TABLE " tname)))
                    (dynamic-wind 
                        (lambda () 
                          (let ((create-temp-table/stmt (jdbc/prepare-statement conn create-temp-table #f)))
                            (jdbc/execute-update create-temp-table/stmt)))
                        (lambda () 
                          (let* ((really-read-table (read-table tname conv))
                                 (rows (really-read-table conn)))
                            rows))
                        (lambda () 
                          (let ((drop-temp-table/stmt (jdbc/prepare-statement conn drop-temp-table #f)))
                            (jdbc/execute-update drop-temp-table/stmt)))))))))
        (call-with-transaction conn)))))


-- 
       If  we confess our sins, he is faithful and just to forgive us our sins
       and to cleanse us from all unrighteousness.

                                                 1 John 1:9 (ESV)

Re: [Sisc-users] New release - Library of code for SISC - Bedlam

[Igor H. V. R. <ig...@ia...>]

Hi!

On Wed, Apr 7, 2010 at 07:34, Kjetil S. Matheussen
<k.s...@no...> wrote:
>> ...
>>  * jcode - an alternative FFI for Java that is  really easy to use.
>> Two examples (of use of jcode):
>>    http://github.com/igorhvr/bedlam/blob/master/iasylum/email.scm
>>    http://github.com/igorhvr/bedlam/blob/master/iasylum/excel.scm
> Cool that you can write java code directly as strings in Scheme!
Thanks!

I even (recently) came up with a way of using strings inside java code
inside Scheme - you can see a few examples in the more recent code:
http://github.com/igorhvr/bedlam/blob/master/iasylum/quartz/quartz-code.scm
(line 6, for instance).

>
> Here's another alternative FFI for java I made a few years ago:
> http://users.notam02.no/~kjetism/javaoo.tar.gz
Interesting - a lot of work, compared to what I did. I took the easy
route by using Strings.. :-)

Nice link - I will play a little bit more with this (and I already
found an idea to steal - the way you implemented java-instance? is
much better than what I had previously in jcode-code.scm)...

Regards,
 Igor.

Re: [Sisc-users] New release - Library of code for SISC - Bedlam

[Kjetil S. M. <k.s...@no...>]

On Wed, 7 Apr 2010, Igor Hjelmstrom Vinhas Ribeiro wrote:

> Hi!
>
>  This is to announce v00000013 of the Bedlam Library for Sisc. You
> can find it under http://github.com/igorhvr/bedlam (direct link:
> http://github.com/igorhvr/bedlam/tarball/v00000013 ).
>
>  Basic job scheduling support using Quartz
> (http://www.quartz-scheduler.org/) to schedule Scheme Closures for
> execution using crontab-like notation is now included.
>
>   Also, the following functionality is currently available in the library:
>
>   * jcode - an alternative FFI for Java that is  really easy to use.
> Two examples (of use of jcode):
>     http://github.com/igorhvr/bedlam/blob/master/iasylum/email.scm
>     http://github.com/igorhvr/bedlam/blob/master/iasylum/excel.scm
>

Cool that you can write java code directly as strings in Scheme!

Here's another alternative FFI for java I made a few years ago:
http://users.notam02.no/~kjetism/javaoo.tar.gz

Your email example can be written like this (I'm pretty sure):

(load "various.scm")
(c-import java)
(define email (new <org.apache.commons.mail.SimpleEmail>))
(-> email setHostName mailserver)
(-> email addTo recpientemail recipientname)
(-> email setFrom sendermail sendername)
(-> email setSubject subject)
(-> email setMsg messagetext)
(-> email send)

[Sisc-users] New release - Library of code for SISC - Bedlam

[Igor H. V. R. <ig...@ia...>]

Hi!

  This is to announce v00000013 of the Bedlam Library for Sisc. You
can find it under http://github.com/igorhvr/bedlam (direct link:
http://github.com/igorhvr/bedlam/tarball/v00000013 ).

  Basic job scheduling support using Quartz
(http://www.quartz-scheduler.org/) to schedule Scheme Closures for
execution using crontab-like notation is now included.

   Also, the following functionality is currently available in the library:

   * jcode - an alternative FFI for Java that is  really easy to use.
Two examples (of use of jcode):
     http://github.com/igorhvr/bedlam/blob/master/iasylum/email.scm
     http://github.com/igorhvr/bedlam/blob/master/iasylum/excel.scm

   * Alex Shinn's implementation of a pattern-matching library (see
match.pdf inside for details on how to use it) and irregex (
http://synthcode.com/scheme/irregex/ ) excelent regular expressions &
SRE library.

   * SRFI-88 (keyword objects /
http://srfi.schemers.org/srfi-88/srfi-88.html) support (reference
implementation used - no self-evaluation property of keyword objects).

   * SRFI-89 (optional and named parameters /
http://srfi.schemers.org/srfi-89/srfi-89.html) support.

   * A simple queue implementation (based on
java.util.concurrent.ConcurrentLinkedQueue).

   * Basic support for i18n (retrieving messages from bundles).

   * Excel parsing and spreadsheet generation in Scheme (used Apache
Poi - http://poi.apache.org/ - underneath) - including
list->spreadsheet, for-each-excel-sheet-data and converting excel's
Dates to Scheme dates .

   * Misc utilities for dealing with jdbc in postgresql and
java.(result-set->iterator; etc).

   * Javascript Object Notation (http://www.json.org/) parser.

   * Memoize functionality.

   * Simple logging facilities.

   * Packrat parser (ported from PLT Scheme).

   * Password generation module (ported from PLT Scheme).

   * Dorai Sitaram's Schelog embedding of Prolog in Scheme
http://www.ccs.neu.edu/home/dorai/schelog/schelog.html

   * Aubrey Jaffer's Slib (http://people.csail.mit.edu/jaffer/SLIB) library.

   * Olin Shivers' let-optionals (low-level / define-macro).

   * Sparse arrays and sparse vectors taken from Chicken Scheme (as-is).

    * Debugging facilities - you can stop in the middle of execution
of a function and get hold of a REPL with local variables you specify.

   * A few other misc utilities.

Regards,
 Igor.

On Wed, Mar 10, 2010 at 18:15, Igor Hjelmstrom Vinhas Ribeiro
<ig...@ia...> wrote:
> Hi!
>
>   During the development of quite a few small SISC-based applications
> and a reasonably-sized one, I ended up creating a small library that
> is quite helpful to me when developing using SISC.
>
>   You can find it under - http://github.com/igorhvr/bedlam
>
>   I am specially fond of jcode - an alternative FFI for Java that is
> really easy to use (I use this a *lot*). Two examples (of use of
> jcode):
>
>     http://github.com/igorhvr/bedlam/blob/master/iasylum/email.scm
>     http://github.com/igorhvr/bedlam/blob/master/iasylum/excel.scm
>
>   In addition to that, you will find integrated a very nice
> pattern-matching library, irregex excelent regular expressions & SRE
> library, one or two srfi's that are missing from the standard SISC
> distribution, and a few other random pieces of code and macros that I
> needed for my own work.
>
>   Installation/usage is really simple -
> http://github.com/igorhvr/bedlam/blob/master/readme.txt , apart from
> the fact that there is absolutely no documentation but the code
> itself.
>
>   I will continue developing this (after all, I am scratching my own
> itch), but I would be delighted to hear any comments, suggestions or
> receive any patches.
>
>  Thanks for SISC!!!
>
> Best Regards,
>  Igor.
>

[Sisc-users] New library of code for SISC - Bedlam

[Igor H. V. R. <ig...@ia...>]

Hi!

   During the development of quite a few small SISC-based applications
and a reasonably-sized one, I ended up creating a small library that
is quite helpful to me when developing using SISC.

   You can find it under - http://github.com/igorhvr/bedlam

   I am specially fond of jcode - an alternative FFI for Java that is
really easy to use (I use this a *lot*). Two examples (of use of
jcode):

     http://github.com/igorhvr/bedlam/blob/master/iasylum/email.scm
     http://github.com/igorhvr/bedlam/blob/master/iasylum/excel.scm

   In addition to that, you will find integrated a very nice
pattern-matching library, irregex excelent regular expressions & SRE
library, one or two srfi's that are missing from the standard SISC
distribution, and a few other random pieces of code and macros that I
needed for my own work.

   Installation/usage is really simple -
http://github.com/igorhvr/bedlam/blob/master/readme.txt , apart from
the fact that there is absolutely no documentation but the code
itself.

   I will continue developing this (after all, I am scratching my own
itch), but I would be delighted to hear any comments, suggestions or
receive any patches.

  Thanks for SISC!!!

Best Regards,
 Igor.

Re: [Sisc-users] SISC on Android and the Heap

[Toni <av...@te...>]

And

http://d.android.com/guide/practices/design/performance.html#cache_fields

Quoting:

"Accessing object fields is much slower than accessing local variables"

So what about caching that "data.length" somewhere? If that method is 
invoked 35k times then maybe it's worth the effort...

Cheers,
Antonio

On 10/03/10 20:55, Ben Goetter wrote:
> You might be surprised at what's slow on Android.  Here you're seeing
> the overhead of myriad calls through an interface.  Qv
> http://d.android.com/guide/practices/design/performance.html#prefer_virtual
>
> Also,
> http://d.android.com/guide/practices/design/performance.html#object_creation
> is really hurting us.
>
> On 3/10/2010 11:42 AM, Toni wrote:
>> On 10/03/10 05:05, Nikita Sidorov wrote:
>>
>>> Your profiler data suggest that it takes 0.223 millisecond per call of MemoryRandomAccessInputStream.read(). But the method is really simple:
>>>
>>>        public int read() throws IOException {
>>>            if (filePointer==data.length) return -1;
>>>            return data[filePointer++]&    0xff;
>>>        }
>>>
>>> Being called 35K+ times results in 7 seconds spent just calling this method.
>>>
>
>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Sisc-users mailing list
> Sis...@li...
> https://lists.sourceforge.net/lists/listinfo/sisc-users

Re: [Sisc-users] SISC on Android and the Heap

[Ben G. <go...@ma...>]

You might be surprised at what's slow on Android.  Here you're seeing 
the overhead of myriad calls through an interface.  Qv 
http://d.android.com/guide/practices/design/performance.html#prefer_virtual

Also, 
http://d.android.com/guide/practices/design/performance.html#object_creation 
is really hurting us.

On 3/10/2010 11:42 AM, Toni wrote:
> On 10/03/10 05:05, Nikita Sidorov wrote:
>    
>> Your profiler data suggest that it takes 0.223 millisecond per call of MemoryRandomAccessInputStream.read(). But the method is really simple:
>>
>>       public int read() throws IOException {
>>           if (filePointer==data.length) return -1;
>>           return data[filePointer++]&   0xff;
>>       }
>>
>> Being called 35K+ times results in 7 seconds spent just calling this method.
>>      

Re: [Sisc-users] SISC on Android and the Heap

[Toni <av...@te...>]

Hi all,

And what's the value of data.length? Maybe that filePointer is too big, 
and for some reason accessing data[filePointer++] is too expensive (I'm 
thinking of paging in a memory-constrained device, or something).

The rest of operations (integer? comparison and binary and) should be 
pretty fast, I imagine.

Cheers,
Antonio

On 10/03/10 05:05, Nikita Sidorov wrote:
> Hi Chad,
>
> Your profiler data suggest that it takes 0.223 millisecond per call of MemoryRandomAccessInputStream.read(). But the method is really simple:
>
>      public int read() throws IOException {
>          if (filePointer==data.length) return -1;
>          return data[filePointer++]&  0xff;
>      }
>
> Being called 35K+ times results in 7 seconds spent just calling this method.
>
> Are you sure it's not the profiler or the simulator that make thing so slow? Have you tried to run it on a real device? Or at least in the simulator but without profiler enabled?
>
> Nikita
>

Re: [Sisc-users] SISC on Android and the Heap

[Nikita S. <nik...@ma...>]

Hi Chad,

Your profiler data suggest that it takes 0.223 millisecond per call of MemoryRandomAccessInputStream.read(). But the method is really simple:

    public int read() throws IOException {
        if (filePointer==data.length) return -1;
        return data[filePointer++] & 0xff;
    }

Being called 35K+ times results in 7 seconds spent just calling this method.

Are you sure it's not the profiler or the simulator that make thing so slow? Have you tried to run it on a real device? Or at least in the simulator but without profiler enabled?

Nikita

On 03/06/2010 11:49 AM, Chad Albers wrote:
> Hi Scott,
>
> I made the changes that your recommended, and removed the dependencies the ber encoding, and rebuilt the heap.  There's a slight performance improve.  On my emulator, it now takes 15 seconds to add 3 plus 4.  It takes a full minute to instantiate a web view, add markup to it, and then query it's title (that's down 15 seconds).
>
> Here' are the new results from the profiler: http://www.neomantic.com/images/screenshot3.png - this is for the instantiate example.  You can see that ber encoding is no longer used.  The blockage still seems to me to be the ser.* classes.
>
> Do you have any other advice for further optimizations?
>
> Chad
>
> --
> Chad Albers
> http://www.neomantic.com
> (pgp signature available on request)
>
>
> On Thu, Mar 4, 2010 at 6:07 AM, Scott G. Miller <sgm...@gm...> wrote:
>
>     Yes, you have to rebuild the heap file after that.
>
>     Scott
>
>
>     On Wed, Mar 3, 2010 at 8:25 PM, Chad Albers <ca...@ne...> wrote:
>
>         Hi Scott,
>
>         Thanks for the advice.  The heap I'm using is from the latest cvs, and it was built using the ant task.  If I'm going to change that code as described below, I should probably recreate the heap, right?
>
>         I made  those changes, and now I get a message saying there is an error loading heap.  
>
>         Chad
>
>
>         --
>         Chad Albers
>         http://www.neomantic.com
>         (pgp signature available on request)
>
>
>         On Wed, Mar 3, 2010 at 9:12 AM, Scott G. Miller <sgm...@gm...> wrote:
>
>             The BerEncoding classes were actually what I suspected might be the problem.  They're fairly cheap on desktop processors, which can do multiple arithmetic operations per clock cycle, but may be worse on embedded processors which can't. 
>
>             You might try changing the implementation of readInt in DeserializerImpl to just datin.readInt(), and writeInt in SerializerImpl to datin.writeInt().  This would use a bit more space in the heap but save cycles. 
>
>             Are you building the heap with "ant heap" in the source distribution?
>
>             Scott
>
>
>             On Tue, Mar 2, 2010 at 8:26 PM, Chad Albers <ca...@ne...> wrote:
>
>                 Hi Scott,
>
>                 I did try to create the heap manually by following what happens in GenerateHeap.java, just in case building it was quicker than deserializing it.  The build failed.
>
>                 I took your advice about the profiler.  Android, indeed, has one called traceview (http://developer.android.com/guide/developing/tools/traceview.html).  I don't' have much experience with profilers, so the information is murky to me.   I set up a trace around line 173 in AppContext.java where r.eval("(initalize)"); is called.  Here's as screen shot: http://neomantic.com/images/traceview.png
>
>                 I sorted by 'Exclusive' time, the time inside a method, java.io.DataInput comes in on top, but it really has to do with sisc/ser/BerEncoding.readBerLong, the 2nd one, one of those sisc.ser.* classes.    But I may be reading this wrong.  
>
>                 Do these results tell you anything?  Got any advice on what I should be looking for?
>
>                 Thanks for your help,
>
>                 Chad
>
>
>                 --
>                 Chad Albers
>                 http://www.neomantic.com
>                 (pgp signature available on request)
>
>
>                 On Mon, Mar 1, 2010 at 11:25 AM, Scott G. Miller <sgm...@gm...> wrote:
>
>                     On Sun, Feb 28, 2010 at 6:22 PM, Chad Albers <ca...@ne...> wrote:
>
>                         I've been plugging away on getting SISC to work on Android, and made more progress.  But I've hit a major obstacle: serialization and deserialization is notoriously slow on Android.  Really slow.  Embedded inside a Android App, SISC takes about 45 seconds to run a simple eval on the emulator.  Most of that time is spent on (initialize) - which performs a lot of deserialization on the precompiled heap file. 
>
>                         My question is can I operate SISC in some way without using the precompiled heap, and instead loading in the code precompiled in the heap.  Scott Miller's 2002 paper seems to imply that this is possible, provided you're willing to take a 10-15 sec performance hit (page 5).
>
>
>                     The alternative to the heap is to load all the startup code from scratch, which is way slower (by about an order of magnitude).   It may be possible however to sacrifice some space (ie a bigger heap) and use a different encoding.  Does the Android platform have a profiler?  Seeing where the bulk of the time is spent during startup may help suggest a different heap storage strategy.
>                      
>
>                         Even if I manage this task, my next question is how often does SISC use serialization/deserialization during processing...(I suspect a great deal).
>
>
>                     Code is lazily loaded in using ser/deser during operation, yes, but the bulk of R5RS is available when you hit the prompt.  
>
>                     Scott
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
>
> _______________________________________________
> Sisc-users mailing list
> Sis...@li...
> https://lists.sourceforge.net/lists/listinfo/sisc-users
>   

Re: [Sisc-users] SISC on Android and the Heap

[Scott G. M. <sgm...@gm...>]

BTW, have you tried the SISC-lite heap?  It contains only whats needed for
R5RS Scheme, and might be faster.  You can build it with 'ant clean; ant
lite-heap'.  This will build new SISC jars as well.

Scott

On Sat, Mar 6, 2010 at 1:41 PM, Scott G. Miller <sgm...@gm...> wrote:

> Unfortunately, what seems to be left are just a really high number of calls
> to read().  On desktop Java, these are pretty cheap since they're compiled
> down to native code.  Apparently Dalvik doesn't do nearly as well.  There
> isn't really any way to avoid reading in that data, as most of SISC is
> implemented in Scheme.  What would be ideal (but Java lacks) would be the
> ability to map a large block of Java heap to and from a file directly.
>
> Scott


>
> On Sat, Mar 6, 2010 at 10:49 AM, Chad Albers <ca...@ne...>wrote:
>
>> Hi Scott,
>>
>> I made the changes that your recommended, and removed the dependencies the
>> ber encoding, and rebuilt the heap.  There's a slight performance improve.
>>  On my emulator, it now takes 15 seconds to add 3 plus 4.  It takes a full
>> minute to instantiate a web view, add markup to it, and then query it's
>> title (that's down 15 seconds).
>>
>> Here' are the new results from the profiler:
>> http://www.neomantic.com/images/screenshot3.png - this is for the
>> instantiate example.  You can see that ber encoding is no longer used.  The
>> blockage still seems to me to be the ser.* classes.
>>
>> Do you have any other advice for further optimizations?
>>
>> Chad
>>
>> --
>> Chad Albers
>> http://www.neomantic.com
>> (pgp signature available on request)
>>
>>
>> On Thu, Mar 4, 2010 at 6:07 AM, Scott G. Miller <sgm...@gm...>wrote:
>>
>>> Yes, you have to rebuild the heap file after that.
>>>
>>> Scott
>>>
>>>
>>> On Wed, Mar 3, 2010 at 8:25 PM, Chad Albers <ca...@ne...>wrote:
>>>
>>>> Hi Scott,
>>>>
>>>> Thanks for the advice.  The heap I'm using is from the latest cvs, and
>>>> it was built using the ant task.  If I'm going to change that code as
>>>> described below, I should probably recreate the heap, right?
>>>>
>>>> I made  those changes, and now I get a message saying there is an error
>>>> loading heap.
>>>>
>>>> Chad
>>>>
>>>>
>>>> --
>>>> Chad Albers
>>>> http://www.neomantic.com
>>>> (pgp signature available on request)
>>>>
>>>>
>>>> On Wed, Mar 3, 2010 at 9:12 AM, Scott G. Miller <sgm...@gm...>wrote:
>>>>
>>>>> The BerEncoding classes were actually what I suspected might be the
>>>>> problem.  They're fairly cheap on desktop processors, which can do multiple
>>>>> arithmetic operations per clock cycle, but may be worse on embedded
>>>>> processors which can't.
>>>>>
>>>>> You might try changing the implementation of readInt in
>>>>> DeserializerImpl to just datin.readInt(), and writeInt in SerializerImpl to
>>>>> datin.writeInt().  This would use a bit more space in the heap but save
>>>>> cycles.
>>>>>
>>>>> Are you building the heap with "ant heap" in the source distribution?
>>>>>
>>>>> Scott
>>>>>
>>>>>
>>>>> On Tue, Mar 2, 2010 at 8:26 PM, Chad Albers <ca...@ne...>wrote:
>>>>>
>>>>>> Hi Scott,
>>>>>>
>>>>>> I did try to create the heap manually by following what happens in
>>>>>> GenerateHeap.java, just in case building it was quicker than deserializing
>>>>>> it.  The build failed.
>>>>>>
>>>>>> I took your advice about the profiler.  Android, indeed, has one
>>>>>> called traceview (
>>>>>> http://developer.android.com/guide/developing/tools/traceview.html).
>>>>>>  I don't' have much experience with profilers, so the information is murky
>>>>>> to me.   I set up a trace around line 173 in AppContext.java where
>>>>>> r.eval("(initalize)"); is called.  Here's as screen shot:
>>>>>> http://neomantic.com/images/traceview.png
>>>>>>
>>>>>> I sorted by 'Exclusive' time, the time inside a method,
>>>>>> java.io.DataInput comes in on top, but it really has to do with
>>>>>> sisc/ser/BerEncoding.readBerLong, the 2nd one, one of those sisc.ser.*
>>>>>> classes.    But I may be reading this wrong.
>>>>>>
>>>>>> Do these results tell you anything?  Got any advice on what I should
>>>>>> be looking for?
>>>>>>
>>>>>> Thanks for your help,
>>>>>>
>>>>>> Chad
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Chad Albers
>>>>>> http://www.neomantic.com
>>>>>> (pgp signature available on request)
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 1, 2010 at 11:25 AM, Scott G. Miller <sgm...@gm...>wrote:
>>>>>>
>>>>>>> On Sun, Feb 28, 2010 at 6:22 PM, Chad Albers <ca...@ne...>wrote:
>>>>>>>
>>>>>>>> I've been plugging away on getting SISC to work on Android, and made
>>>>>>>> more progress.  But I've hit a major obstacle: serialization and
>>>>>>>> deserialization is notoriously slow on Android.  Really slow.  Embedded
>>>>>>>> inside a Android App, SISC takes about 45 seconds to run a simple eval on
>>>>>>>> the emulator.  Most of that time is spent on (initialize) - which performs a
>>>>>>>> lot of deserialization on the precompiled heap file.
>>>>>>>>
>>>>>>>> My question is can I operate SISC in some way without using the
>>>>>>>> precompiled heap, and instead loading in the code precompiled in the heap.
>>>>>>>>  Scott Miller's 2002 paper seems to imply that this is possible, provided
>>>>>>>> you're willing to take a 10-15 sec performance hit (page 5).
>>>>>>>>
>>>>>>>>
>>>>>>> The alternative to the heap is to load all the startup code from
>>>>>>> scratch, which is way slower (by about an order of magnitude).   It may be
>>>>>>> possible however to sacrifice some space (ie a bigger heap) and use a
>>>>>>> different encoding.  Does the Android platform have a profiler?  Seeing
>>>>>>> where the bulk of the time is spent during startup may help suggest a
>>>>>>> different heap storage strategy.
>>>>>>>
>>>>>>>
>>>>>>>> Even if I manage this task, my next question is how often does SISC
>>>>>>>> use serialization/deserialization during processing...(I suspect a great
>>>>>>>> deal).
>>>>>>>>
>>>>>>>>
>>>>>>> Code is lazily loaded in using ser/deser during operation, yes, but
>>>>>>> the bulk of R5RS is available when you hit the prompt.
>>>>>>>
>>>>>>> Scott
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>