apophenia-discuss — The general discussion list for Apophenia.

[Apophenia-discuss] Author Apophenia books-Packt Publishing.

[Kshipra S. <ksh...@pa...>]

Hi All, 

I am writing to you for Packt Publishing, the publishers of computer related books. 

We are planning to extend our catalogue of books based on Scientific Computing Tools and are currently inviting authors interested in writing for Packt. This doesn't need any previous writing experience. Just an expert knowledge of your subject and a passion to share it with others is all that we require. 

So, if you love Apophenia and are interested in authoring a book, here's an opportunity knocking your door. Write to us with your Apophenia book ideas at au...@pa.... Even if you don't have a book idea and are simply interested in authoring a book, we are still keen to hear from you. 

More details about the opportunity are available at: http://authors.packtpub.com/content/scientific-compting-tools-write-Packt

Thanks
Kshipra Singh
Author Relationship Manager
Packt Publishing
www.PacktPub.com
 
Skype: kshiprasingh15
Twitter: http://twitter.com/kshipras
 
Interested in becoming an author? Visit http://authors.packtpub.com for all the information you need about writing for Packt.

[Apophenia-discuss] Bug? Aphophenia 0.22 vasnprintf compilation error on MinGW32

[Matt P. D. <ma...@gm...>]

Hi!

I have encountered an issue when trying to compile the library under
MinGW32 -- one of the files ("vasnprintf.c") seems to expect presence of
POSIX-style "alloc.h" and fails due to its lack. I think that on MinGW32
the relevant header defining the mem. alloc. functions is "malloc.h"
(alternatively, there is also glib-2.0's "galloca.h"), but, as far as I
recall, "alloc.h" is not and has never been a part of MinGW32 (or any
well-known/googleable package/library).

Hence - is MinGW32 even supported or am I missing something (and if so,
what would you suggest on how to proceed)?

Below is the error message resulting from the failed build attempt:

$ make
make  all-am
make[1]: Entering directory `/home/matt/apophenia-0.22'
if /bin/sh ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.
-I.    -g -Wall -I./ -I
/usr/include/mysql -lpthread -g -O2 -MT vasnprintf.lo -MD -MP -MF
".deps/vasnprintf.Tpo" -c -o v
asnprintf.lo `test -f 'vasprintf/vasnprintf.c' || echo
'./'`vasprintf/vasnprintf.c; \
        then mv -f ".deps/vasnprintf.Tpo" ".deps/vasnprintf.Plo"; else
rm -f ".deps/vasnprintf.T
po"; exit 1; fi
 gcc -DHAVE_CONFIG_H -I. -I. -I. -g -Wall -I./ -I/usr/include/mysql
-lpthread -g -O2 -MT vasnpri
ntf.lo -MD -MP -MF .deps/vasnprintf.Tpo -c vasprintf/vasnprintf.c
-DDLL_EXPORT -DPIC -o .libs/v
asnprintf.o
vasprintf/vasnprintf.c:29:21: error: alloca.h: No such file or directory
vasprintf/vasnprintf.c: In function 'vasnprintf':
vasprintf/vasnprintf.c:156: warning: implicit declaration of function
'alloca'
vasprintf/vasnprintf.c:156: warning: incompatible implicit declaration
of built-in function 'all
oca'
make[1]: *** [vasnprintf.lo] Error 1
make[1]: Leaving directory `/home/matt/apophenia-0.22'
make: *** [all] Error 2

Best,

Matt

Re: [Apophenia-discuss] tensors

[Sergey P. <pl...@cs...>]

On 05/07/2009 06:10 PM, B K wrote:
> I haven't used either myself, but they may be more what you have in mind

Thanks for the information. marray and tensor are not very good because 
they are allowing access only to single elements of a multidimentional 
array. This is suboptimal: using GSL's matrix structure fields (tda in 
particular) I can easily slice some of the dimensions of multarrays 
without moving a single bit of their contents. I have my own 
implementation that allows me to slice certain dimensions, but it gets 
hairy, when I need to provide a really arbitrary slicing.

Inspired by your library I googled a little and found this paper:

http://www.springerlink.com/content/2tdwv88rr5m8hl11/

which talks exactly about implementing multidimensional arrays on top of a DBMS system. Compared to the idea you are describing, where each dimension has its separate field (and takes additional memory just for these (computable) indices), the idea of the paper is to pre-compute indices and then just pass sequences to DBMS to extract the matrices (something similar to what
apop_db_to_crosstab does). Too bad SQL queries becomes unwieldy and a special preprocessor is required which sqlite seems to not have :(

In any case, thanks for the library and a different view on data modeling.

--
Sergey

Re: [Apophenia-discuss] tensors

[B K <bkl...@gm...>]

Hi, Sergey.

There is one function for querying out a 2-D matrix,
apop_db_to_crosstab. Given a database table listing dimensions and
values, like

dim1 dim2 val
1      2       8
1      1       7
2      1       5
2      2       3

the function will produce a matrix of the form

7 8
5 3.

The database may have more columns for more dimensions, in which case
you could pull out slices using a query like "select dim1, dim3, val
where dim2=0". Or you could not bother with the crosstabbing function
and just read each row as a multidimensional vector.

The db-to-crosstab is about all Apohenia does readily, because that's
the most common thing one sees in the statistics context. If you want
more from your multidimensional matrices, you may find more use in one
of the two sister projects implementing a multidimensional array
extension to the GSL and a tensor extension:
http://savannah.nongnu.org/projects/marray/
and
http://savannah.nongnu.org/projects/tensor/ .
I haven't used either myself, but they may be more what you have in mind.

Regards,

B Klemens

2009/5/4 Sergey Plis <pl...@cs...>:
> Hi!
>
> I have stumbled upon apophenia library online and got excited by the
> idea of fusing databases and scientific computation. What I can't get
> from documentation is how to create multidimensional arrays. I mean
> databases from which I can query 2D slices as gls_matrices. It seems
> like the feature is provided.
>
> Thank you!
>
> --
> Sergey
>
> ------------------------------------------------------------------------------
> Register Now & Save for Velocity, the Web Performance & Operations
> Conference from O'Reilly Media. Velocity features a full day of
> expert-led, hands-on workshops and two days of sessions from industry
> leaders in dedicated Performance & Operations tracks. Use code vel09scf
> and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
> _______________________________________________
> Apophenia-discuss mailing list
> Apo...@li...
> https://lists.sourceforge.net/lists/listinfo/apophenia-discuss
>

[Apophenia-discuss] Problem with Location of Header Files

[Thomas La B. <bo...@gf...>]

A little background first: I am running

Ubuntu 9.04 64-bit
Codeblocks 8.02
Apophenia 0.22-28

and I am trying to learn c, linux, and apophenia at the same time. I 
have gcc, gsl and sqlite installed. I have successfully compiled and run 
a gsl example, so gcc and gsl are OK. I compiled and installed apophenia 
and then tried to compile the following Apophenia example:


#include <apop.h>
int main(){
   int		draws	= 1e7;
   int		bins	= 100;
   double	mu		= 0.492;    //also try 3./8.
   double	sigmasq = 0.093;    //also try 1./24.
   gsl_rng	*r		= apop_rng_alloc(0);
   apop_data *d		= apop_data_alloc(0, draws, 1);
     apop_model *m = apop_beta_from_mean_var(mu, sigmasq);
     for (int i =0; i < draws; i++)
         apop_draw(apop_data_ptr(d, i, 0), r, m);
     Apop_settings_add_group(&apop_histogram, apop_histogram, d, bins)
     apop_histogram_normalize(&apop_histogram);
     apop_plot_histogram(&apop_histogram, NULL);
}


and got the following errors


/usr/include/apophenia/apop.h|3|error: apophenia/model.h: No such file 
or directory|
/home/thomas/Desktop/drawbeta.c||In function ‘main’:|
/home/thomas/Desktop/drawbeta.c|10|warning: implicit declaration of 
function ‘apop_beta_from_mean_var’|
/home/thomas/Desktop/drawbeta.c|10|warning: initialization makes pointer 
from integer without a cast|
/home/thomas/Desktop/drawbeta.c|11|error: ‘for’ loop initial declaration 
used outside C99 mode|
/home/thomas/Desktop/drawbeta.c|12|warning: implicit declaration of 
function ‘apop_draw’|
/home/thomas/Desktop/drawbeta.c|13|error: ‘apop_histogram’ undeclared 
(first use in this function)|
/home/thomas/Desktop/drawbeta.c|13|error: (Each undeclared identifier is 
reported only once|
/home/thomas/Desktop/drawbeta.c|13|error: for each function it appears in.)|
/home/thomas/Desktop/drawbeta.c|13|error: ‘apop_histogram_settings_free’ 
undeclared (first use in this function)|
/home/thomas/Desktop/drawbeta.c|13|error: ‘apop_histogram_settings_copy’ 
undeclared (first use in this function)|
/home/thomas/Desktop/drawbeta.c|13|warning: implicit declaration of 
function ‘apop_histogram_settings_alloc’|
/home/thomas/Desktop/drawbeta.c|13|warning: passing argument 5 of 
‘apop_settings_group_alloc’ makes pointer from integer without a cast|
/home/thomas/Desktop/drawbeta.c|15|warning: passing argument 2 of 
‘apop_plot_histogram’ makes integer from pointer without a cast|
/home/thomas/Desktop/drawbeta.c|15|error: too few arguments to function 
‘apop_plot_histogram’|
/home/thomas/Desktop/drawbeta.c|16|warning: control reaches end of 
non-void function|
||=== Build finished: 8 errors, 7 warnings ===|


apop.h expands to

#include <apophenia/db.h>
#include <apophenia/asst.h>
#include <apophenia/model.h>
#include <apophenia/types.h>
#include <apophenia/stats.h>
#include <apophenia/output.h>
#include <apophenia/mapply.h>
#include <apophenia/settings.h>
#include <apophenia/bootstrap.h>
#include <apophenia/histogram.h>
#include <apophenia/regression.h>
#include <apophenia/conversions.h>
#include <apophenia/likelihoods.h>
#include <apophenia/missing_data.h>
#include <apophenia/linear_algebra.h>
//Part of the intent of a convenience header like this is that you
//don't have to remember what else you're including. So here are
//some other common GSL headers:
#include <gsl/gsl_rng.h>
#include <gsl/gsl_sort.h>
#include <gsl/gsl_eigen.h>
#include <gsl/gsl_histogram.h>
#include <gsl/gsl_sort_vector.h>
#include <gsl/gsl_permutation.h>
#include <gsl/gsl_integration.h>
//And common headers for other uses (such as seeding an RNG):
#include <time.h>
#include <unistd.h>


All the headers EXCEPT for the offending model.h appear to be in 
/usr/include/apophenia. model.h is in /usr/include/apophenia/model. So, 
the question is, what is wrong here with my install of apophenia and how 
do I fix it?

Tom

[Apophenia-discuss] tensors

[Sergey P. <pl...@cs...>]

Hi!

I have stumbled upon apophenia library online and got excited by the 
idea of fusing databases and scientific computation. What I can't get 
from documentation is how to create multidimensional arrays. I mean 
databases from which I can query 2D slices as gls_matrices. It seems 
like the feature is provided.

Thank you!

--
Sergey

[Apophenia-discuss] Error message 'callback requested query abort' with sqlite and Apophenia on Cygwin

[Damion M. <dam...@ir...>]

Hi,

I am running Cygwin (on XP) with sqlite3 v 3.6.2-1

I compiled Apophenia ok, but when I try to run any program that
uses a sqlite database, I get an error, either:
     callback requested query abort

or:
     assertion "apop_table_exists(tabname, 0)" failed: file "apop_conversions.c"
line 713


I attach the above error messages that I got from the sample C programs that
come with Apophenia.

My guess is that something has changed between sqlite and Apophenia
since about a year ago; back then I had compiled and used Apophenia with sqlite
and had no problem.

(I have tried the sample C program that comes with sqlite and that works fine.)

Many thanks.

All the best,
Damion

Re: [Apophenia-discuss] Error message

[Shige S. <shi...@gm...>]

Hi Ben,

That is exactly where the problem was. After I set the environment variable
(LD_LIBRARY_PATH) correctly, the problem goes away. Many thanks. Great work,
by the way.

Best,
Shige

On Fri, Jan 30, 2009 at 11:11 AM, B K <bkl...@gm...> wrote:

>
>
> 2009/1/30 Shige Song <shi...@gm...>
>
> Dear All,
>>
>> I have just apophenia 0.21 compiled and installed on my Unbuntu 8.10 box.
>> I was able to build the test program (in the "tests" subdirectory) but could
>> not get it running. When I typed "./test" I got error message "./test: error
>> while loading shared libraries: libapophenia.so.0: cannot open shared object
>> file: No such file or directory."
>>
>> I checked the installed directory (/usr/local/lib), I can see "
>> libapophenia.la", "libapophenia.so", "libapophenia.so.0", and
>> "libapophenia.so.0.0.0". Can anybody tell me how to fix this?
>>
>> Many thanks.
>>
>> Best,
>> Shige
>>
>
> My first guess is that this is a problem with your library path, probably
> the environment variable named LD_LIBRARY_PATH. This is what the system
> checks during both compile time and run time to find libraries. The make
> utility sets the libpath during compile time, but you may have to set it
> yourself during run time. If you don't already have a copy of _Modeling with
> Data_, download a PDF of it at modelingwithdata.org and have a look at
> Appendix A, which covers some of the details of path-setting in detail.
> There's also an online appendix on the same site which covers some of these
> issues.
>
> Please let us know if that's the problem (or not),
>
> Ben
>

Re: [Apophenia-discuss] Error message

[B K <bkl...@gm...>]

2009/1/30 Shige Song <shi...@gm...>

> Dear All,
>
> I have just apophenia 0.21 compiled and installed on my Unbuntu 8.10 box. I
> was able to build the test program (in the "tests" subdirectory) but could
> not get it running. When I typed "./test" I got error message "./test: error
> while loading shared libraries: libapophenia.so.0: cannot open shared object
> file: No such file or directory."
>
> I checked the installed directory (/usr/local/lib), I can see "
> libapophenia.la", "libapophenia.so", "libapophenia.so.0", and
> "libapophenia.so.0.0.0". Can anybody tell me how to fix this?
>
> Many thanks.
>
> Best,
> Shige
>

My first guess is that this is a problem with your library path, probably
the environment variable named LD_LIBRARY_PATH. This is what the system
checks during both compile time and run time to find libraries. The make
utility sets the libpath during compile time, but you may have to set it
yourself during run time. If you don't already have a copy of _Modeling with
Data_, download a PDF of it at modelingwithdata.org and have a look at
Appendix A, which covers some of the details of path-setting in detail.
There's also an online appendix on the same site which covers some of these
issues.

Please let us know if that's the problem (or not),

Ben

[Apophenia-discuss] Error message

[Shige S. <shi...@gm...>]

Dear All,

I have just apophenia 0.21 compiled and installed on my Unbuntu 8.10 box. I
was able to build the test program (in the "tests" subdirectory) but could
not get it running. When I typed "./test" I got error message "./test: error
while loading shared libraries: libapophenia.so.0: cannot open shared object
file: No such file or directory."

I checked the installed directory (/usr/local/lib), I can see "
libapophenia.la", "libapophenia.so", "libapophenia.so.0", and
"libapophenia.so.0.0.0". Can anybody tell me how to fix this?

Many thanks.

Best,
Shige

[Apophenia-discuss] issues installing apophenia

[Mary E. F. <mfi...@bu...>]

Hi ,
I am having trouble install apophenia-0.21 on my  64-bit linux  system.  
I have successfully installed gsl-1.11 and sqlite-3.6.6.2 under the 
install dir /data
Below are the errors I get when I run make:
genomics:/data/apophenia-0.21# ./configure --prefix=/data/apophenia-0.21


genomics:/data/apophenia-0.21# make
make  all-am
make[1]: Entering directory `/data/apophenia-0.21'
if /bin/sh ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I. 
-I.    -g -Wall -I./ -I/usr/include/mysql -pthread -g -O2 -MT 
apop_conversions.lo -MD -MP -MF ".deps/apop_conversions.Tpo" -c -o 
apop_conversions.lo apop_conversions.c; \
then mv -f ".deps/apop_conversions.Tpo" ".deps/apop_conversions.Plo"; 
else rm -f ".deps/apop_conversions.Tpo"; exit 1; fi
mkdir .libs

gcc -DHAVE_CONFIG_H -I. -I. -I. -g -Wall -I./ -I/usr/include/mysql 
-pthread -g -O2 -MT apop_conversions.lo -MD -MP -MF 
.deps/apop_conversions.Tpo -c apop_conversions.c  -fPIC -DPIC -o 
.libs/apop_conversions.o
In file included from db.h:4,
                 from conversions.h:11,
                 from apop_conversions.c:4:
types.h:7:25: gsl/gsl_rng.h: No such file or directory
types.h:8:28: gsl/gsl_matrix.h: No such file or directory
In file included from db.h:4,
                 from conversions.h:11,
                 from apop_conversions.c:4:
types.h:85: error: syntax error before "gsl_vector"


The error is that it cannot find gsl/gsl_rng.h, but I have added those 
dirs to my path and created a ld.so.conf.d file for gsl, sqlite, and 
apophenia.

-- 
Thanks
Mary Ellen

[Apophenia-discuss] reworking the setting of settings.

[B K <bkl...@gm...>]

Dear model estimator,

The system for modifying model settings has been changed significantly,
and the shift merits more than a single line in the ChangeLog. So here's
a summary of the changes, and their motivation, pros and cons.

Apophenia is really only based on two objects, the apop_data set and
the apop_model. Data sets come in a pretty standard form, so the data
object is basically settled. But describing a statistical, agent-based,
social, or physical model in a reasonably standardized form is much more
difficult, primarily because every model has significantly different
settings. E.g., an MLE requires a method of search (conjugate gradient,
simplex, simulated annealing), and a histogram needs the number of slots
to be filled with data.

Apophenia's former way of doing things was to have two elements of the
apop_model struct: the method_settings and the model_settings, which was
sufficient for most purposes. You could set your histogram's parameters,
and pick a maximum likelihood method, and there would be no conflict. But
with only two slots, things started to get a bit crowded, and it was
hard to work out how to retain space for applying multiple estimation
methods to one model, or how to put multilevel models into a single model
(rather than chaining them together in various ways).

So, the method_settings and model_settings elements have been replaced
with a single list, which is simply named settings, which can hold an
arbitrary number of groups of settings. For example, you can have a
set of closed-form variables for estimating the log likelihood, and a
histogram for making random draws.

To get/set a setting, you would need to specify the model, the settings
group, and the name of the setting itself.  This is much like a config
file (or for Windows users, a .ini file), which would have settings
divided into sections, like

[MLE]
verbose = 0
method = APOP_CG_PR

[least squares]
weights = your_weight_vector

[histogram]
bins = {0.1, 0.3, 0.2,...}


    Using it

If you don't need to change settings from a model's default, you don't
need to care about any of this; a form like
> apop_model_show(apop_estimate(your_data_here, apop_ols));
still works fine.

If you do need to change settings, then the process takes another step
or two. Here's a sample:

1 apop_data *data = your_data_here;
2 apop_data *w = your_weights_here;

3 apop_model *m = apop_model_copy(apop_wls);
4 Apop_settings_alloc(m, apop_ls, data);
5 Apop_settings_add(m, apop_ls, weights, w);
6 apop_model *est = apop_estimate(data, *m);

Line three establishes the baseline form of the model. Line four adds
a settings group of type apop_ls_settings to the model. If you check the
manual for apop_ls_settings_alloc, you'll see that it takes one
argument---the input data---and the inputs to the alloc function appear
after the model and settings group name. Line five sets the weights
element in that group to w, and looks like the config-file format above.

Also, the output to any of Apophenia's estimations will have an
appropriate group of settings allocated, so you can chain estimations
pretty easily. Continuing the above example, you could re-estimate with
an alternative set of weights via:

> Apop_settings_add(est, apop_ls, weights, weight_set_two);
> apop_model *est2 = apop_estimate(data, *est);

[Notice that the 'settings_add' function really just modifies the setting.]

Many people are incredibly averse to additional lines of code, so there's
a convenience macro to allocate and modify a setting on the same line
(joining lines 4 and 5 above):

> Apop_settings_alloc_add(m, apop_ls, weights, w, data);

If you need to read a setting, such as to check whether things have
changed after an estimation, use:

Apop_settings_get(m, apop_ls, weights);

Notice the use of a single capital to remind you that you are using a
macro, and so surprising errors can crop up. Here in the modern day, we
read things like APOP_SETTINGS_ALLOC_ADD as yelling, but if you prefer
all caps to indicate macros, those work as well.

For just using a model, that's about 100% of what you need to know, and
you can just skip to the optimistic conclusion below.


    Writing new settings

To store the settings for your own models, you don't necessarily
need any of this. The apop_model structure has a void pointer named
more which you can use as you see fit. If more_size is larger than zero
(i.e., you set it to sizeof(your_struct)), then it will be copied via
memcpy as necessary. Apohenia's estimation routines will never impinge
on this item, so do what you feel with it.

If you do want to use the settings system to set up a new model, then
you will need four items.  This is the sort of boilerplate that will
be familiar to users of object oriented languages in the style of C++
or Java. Let your settings group be named ysg; then you will need

--The settings struct
typedef struct {
...
} ysg_settings


--The allocate function
ysg_settings *ysg_settings_alloc(...){
    ysg_settings *out = malloc(sizeof(ysg_settings));
    /* Give default values for all elements here. */
    return out; }


--The copy function
void *ysg_settings_copy(ysg_settings *copyme) {
    ysg_settings *out = malloc(sizeof(ysg_settings));
    ...
    return out; }


--The free function, which can be as brief as:
void ysg_settings_free(ysg_settings *copyme) {
    free(copyme);
}
(but may include a freeing of pointed-to subelements as necessary).

The names are not negotiable: when you call
> Apop_settings_alloc(m, ysg, ...)
the macro will look for ysg_settings, ysg_settings_alloc, et cetera.

The lines-of-code averse will cringe at having to write such boilerplate
code (I do), but we've tried a few alternatives, and it's the least of
all evils.

You can retrieve the whole group or individual elements via:
> Apop_settings_get_group(m, ysg)
or the one-element macro mentioned above,
> Apop_settings_get(m, ysg, an_element)

As you saw above, once the typedef/alloc/copy/free machinery is written,
you can declare, get, and set in a reasonably graceful manner.


    In conclusion

OK, thanks for bearing with these exceptionally user-visible changes, and
I hope you find that beyond the main goal of allowing models to go through
an arbitrary number of filters, it does make your code a little cleaner.
This has good odds of being the last major reform the basic system sees,
meaning that from here on out it's just a question of adding more models,
tests, and estimation routines.

Bk

PS: On to new business, the next item on the reform agenda is
representations of multidimensional histograms. The GSL only provides 1-
and 2-d histograms, which is a somewhat low limit. The problem is that
in many dimensions, writing down the full grid becomes infeasible, and we
need to instead use a list representing a sparse matrix---but smoothing,
integrating over a range, or otherwise giving a good representation
of a high-dimensional surface is nontrivial. Suggestions (or better,
contributions of code) are welcome.

[Apophenia-discuss] Question about Apophenia and Dev-C

[<jan...@fr...>]

Hi All,
=20
My question is how to compile the Apophenia library with  Dev-c++
environment ?=20
=20
Regards,
=20
Janos
=20
=20
=20
______________________________________________________

Kor=E1n keltem, hol az arany? :-)

=20

[Apophenia-discuss] Question about Apophenia and Dev-C

[<ja...@gy...>]

Hi All,
=20
My question is how to compile the Apophenia library with  Dev-c++
environment ?=20
=20
Regards,
=20
Janos
=20
=20
=20
______________________________________________________

Kor=E1n keltem, hol az arany? :-)

=20