pdl-stats-help — General purpose Q and A

Re: [pdl-stats] PDL::Stats::GLM - Uninvertible matrix given to inv

[Maggie X <mag...@gm...>]

Hi Daniel,

Can you show an example of the data that produced the uninvertible matrix?

Maggie
On Dec 8, 2014 2:10 PM, "Daniel Kitchener" <dan...@gm...>
wrote:

> I am trying to use the logistic function in PDL::Stats::GLM, but sometimes
> get the error "Uninvertible matrix given to inv". I have traced this to
> PDL::Fit::LM trying to invert a matrix to estimate covariance, and found
> this related post from 2009:
> http://mailman.jach.hawaii.edu/pipermail/perldl/2009-November/002477.html
>
> However, it seems that there's not any resolution to that post. Is there a
> way to work around this issue or make it not appear?
>
> For now, I've resorted to locally removing the matinv call
> in PDL::Fit::LM::lmfit, as the output is never used by
> PDL::Stats::GLM::logistic, but having a dependency on a forked module is
> obviously not an ideal solution.
>
> Thanks
> Daniel
>
>
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> pdl-stats-help mailing list
> pdl...@li...
> https://lists.sourceforge.net/lists/listinfo/pdl-stats-help
>
>

[pdl-stats] PDL::Stats::GLM - Uninvertible matrix given to inv

[Daniel K. <dan...@gm...>]

I am trying to use the logistic function in PDL::Stats::GLM, but sometimes
get the error "Uninvertible matrix given to inv". I have traced this to
PDL::Fit::LM trying to invert a matrix to estimate covariance, and found
this related post from 2009:
http://mailman.jach.hawaii.edu/pipermail/perldl/2009-November/002477.html

However, it seems that there's not any resolution to that post. Is there a
way to work around this issue or make it not appear?

For now, I've resorted to locally removing the matinv call
in PDL::Fit::LM::lmfit, as the output is never used by
PDL::Stats::GLM::logistic, but having a dependency on a forked module is
obviously not an ideal solution.

Thanks
Daniel

Re: [pdl-stats] ols and missing values

[Maggie X. <mag...@us...>]

Hi Ingo,

Thanks for working out the issue! If you fork from the original pdl-stats
repo and push your change to the forked repo, you can then send me a link
to the commit where you've made the change. Or, if you don't want to deal
with git, you can email me the pp code and I'll take a look. Thanks!

Here<http://net.tutsplus.com/tutorials/other/easy-version-control-with-git/>'s
a simple git tutorial if you want to know the basics. There's a section on
github.


Best,
Maggie


On Mon, Mar 18, 2013 at 6:23 AM, Ingo Schmid <in...@gm...> wrote:

>  Hi,
>
> recently I've needed some simple pearson's correlations and found out
> that ols can't handle missing values elegantly. fill_m doesn't do the job
> if you need also p-values. So I changed ols to filter rows from y and ivs
> that contain bad values, then clear the flag before continuing. This is
> based on a flag.
>
> Thing is, I've never used git so far, I managed to clone and pull, but am
> unsure about how to commit GLM.pp and push it to the server for you to
> review.
>
> Thanks
> Ingo
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> pdl-stats-help mailing list
> pdl...@li...
> https://lists.sourceforge.net/lists/listinfo/pdl-stats-help
>
>

[pdl-stats] ols and missing values

[Ingo S. <in...@gm...>]

Hi,

recently I've needed some simple pearson's correlations and found out
that ols can't handle missing values elegantly. fill_m doesn't do the
job if you need also p-values. So Ichanged ols to filter rows from y and
ivs that contain bad values, then clear the flag before continuing. This
is based ona flag.

Thing is, I've never used git so far,I managed to clone and pull, butam
unsure about how to commit GLM.pp and push it to the server for you to
review.

Thanks
Ingo

Re: [pdl-stats] PDL::Stats::Distr

[Maggie X <mag...@gm...>]

Both modules are part of the PDL::Stats package. You get them when you
install PDL::Stats. On windows the installation process often skips these
two modules because it can't find the gsl libs. You can download the
pdl-stats ppm from sysphus' repo where it's compiled against gsl so have
all the modules.

As to gaps, you can use bad value. Or if it's something like discreet vs.
continuous distribution you can also warn the user about the expected input
or value range, then bail out on it.

Best,
Maggie
On Jul 23, 2012 5:01 PM, "Boyd Duffee" <b.d...@is...> wrote:

> Hi all,
>
> In the pod for PDL::Stats, it says that PDL::Stats is a short cut for
> <modules> and _if installed_ use PDL::Stats::Distr and use PDL::GSL::CDF
> Both of these modules sound like something that I'd like to use, but I
> can't find them on cpan.  Any idea where I could get my hands on them?
>
> I'm building up probability distributions in order to fit power law
> exponents to the curves.  I can't guarantee to have a point for each x
> value in the curve, so I'm assuming that I'd need to use bad values for
> those gaps.  Any suggestions?
>
> thanks,
> --
> Boyd Duffee                             Keele University
> Systems Guy                             (01782) 734225
>    It is amazing what you can accomplish if you are
>    willing to let someone else take the credit.
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> pdl-stats-help mailing list
> pdl...@li...
> https://lists.sourceforge.net/lists/listinfo/pdl-stats-help
>

[pdl-stats] PDL::Stats::Distr

[Boyd D. <b.d...@ke...>]

Hi all,

In the pod for PDL::Stats, it says that PDL::Stats is a short cut for 
<modules> and _if installed_ use PDL::Stats::Distr and use PDL::GSL::CDF
Both of these modules sound like something that I'd like to use, but I 
can't find them on cpan.  Any idea where I could get my hands on them?

I'm building up probability distributions in order to fit power law 
exponents to the curves.  I can't guarantee to have a point for each x 
value in the curve, so I'm assuming that I'd need to use bad values for 
those gaps.  Any suggestions?

thanks,
-- 
Boyd Duffee				Keele University
Systems Guy				(01782) 734225	
   It is amazing what you can accomplish if you are
   willing to let someone else take the credit.

[pdl-stats] PDL::Stats::Distr

[Boyd D. <b.d...@is...>]

Hi all,

In the pod for PDL::Stats, it says that PDL::Stats is a short cut for 
<modules> and _if installed_ use PDL::Stats::Distr and use PDL::GSL::CDF
Both of these modules sound like something that I'd like to use, but I 
can't find them on cpan.  Any idea where I could get my hands on them?

I'm building up probability distributions in order to fit power law 
exponents to the curves.  I can't guarantee to have a point for each x 
value in the curve, so I'm assuming that I'd need to use bad values for 
those gaps.  Any suggestions?

thanks,
-- 
Boyd Duffee				Keele University
Systems Guy				(01782) 734225	
   It is amazing what you can accomplish if you are
   willing to let someone else take the credit.

Re: [pdl-stats] PDL::Slatec warning

[Maggie X. <mag...@us...>]

Thank you for the feedback! The warning was in place primarily for debugging
build failures. I should be able to either make the warning optional or move
it to the make test process in the next release.

As to PDL::Slatec (and PDL::GSL::CDF) with Strawberry Perl, you can try the
ppm builds (the ppm utility <http://search.cpan.org/dist/PPM/> is available
from cpan):

    ppm install http://www.sisyphusion.tk/ppm/PGPLOT.ppd
    ppm install http://www.sisyphusion.tk/ppm/PDL.ppd
    ppm install http://www.sisyphusion.tk/ppm/PDL-Stats.ppd


Best,
Maggie


On Tue, Oct 11, 2011 at 5:02 PM, Bruce Ravel <br...@bn...> wrote:

>
> Hi,
>
> I have been using PDL::Stats for some time now.  It's great!  Thanks a
> million!
>
> I don't love that PDL::Stats::GLM complains about the absence of
> PDL::Slatec.  The documentation states that PDL::SLatec is only
> *required* for the logistic method (which I am not using).
>
> I find it to be hit or miss as to whether PDL::Slatec installs easily.
> On most of my linux machines -- no problem.  On all of my Windows
> machines using Strawberry, not so much.  Since I do not use the
> logistic method, I find the warning to be unnecessary.
>
> I have a lot of unsavvy users of my software for whom a
> not-quite-necessary warning is bound to be pretty scary.  My
> preference would be for PDL::Stats::GLM to croak or die if logistic is
> called in the absence of PDL::Stats, but for it to be silent in any
> other case.
>
> Thanks!
> Bruce
>
> --
>
>  Bruce Ravel  ------------------------------------ br...@bn...
>
>  National Institute of Standards and Technology
>  Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2
>  Building 535A
>  Upton NY, 11973
>
>  My homepage:    http://xafs.org/BruceRavel
>  EXAFS software:  http://cars9.uchicago.edu/ifeffit/Demeter
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> pdl-stats-help mailing list
> pdl...@li...
> https://lists.sourceforge.net/lists/listinfo/pdl-stats-help
>

[pdl-stats] PDL::Slatec warning

[Bruce R. <br...@bn...>]

Hi,

I have been using PDL::Stats for some time now.  It's great!  Thanks a
million!

I don't love that PDL::Stats::GLM complains about the absence of
PDL::Slatec.  The documentation states that PDL::SLatec is only
*required* for the logistic method (which I am not using).

I find it to be hit or miss as to whether PDL::Slatec installs easily.
On most of my linux machines -- no problem.  On all of my Windows
machines using Strawberry, not so much.  Since I do not use the
logistic method, I find the warning to be unnecessary.

I have a lot of unsavvy users of my software for whom a
not-quite-necessary warning is bound to be pretty scary.  My
preference would be for PDL::Stats::GLM to croak or die if logistic is
called in the absence of PDL::Stats, but for it to be silent in any
other case.

Thanks!
Bruce

-- 

 Bruce Ravel  ------------------------------------ br...@bn...

 National Institute of Standards and Technology
 Synchrotron Methods Group at NSLS --- Beamlines U7A, X24A, X23A2
 Building 535A
 Upton NY, 11973

 My homepage:    http://xafs.org/BruceRavel
 EXAFS software:  http://cars9.uchicago.edu/ifeffit/Demeter

Re: [pdl-stats] A PDL::Stats change I'd love to see

[Maggie X. <mag...@us...>]

Thank you David. I'll add it to the todo list for the next release.

Best,
Maggie


On Wed, Apr 27, 2011 at 8:35 AM, <dw...@em...> wrote:

>
> Thank you Maggie.
>
>  What I'd like to be able to do, with things like the logistic function
> (any curve fitter
> in PDL::Stats, actually) is something like this:
>
>  my %m = $y->logistic( COV => 1 );
> print $m{cov}->corr_table();
>
>  I don't think logistic should return the covariance matrix by default,
> but I would like
> to be able to get to it.
>
>  A single line patch in GLM.pm, after %ret is declared, around line 2000
> or so..
>
>  $ret{cov} = $cov if ( $opt{COV} );
>
>  would do the trick for logistic, I believe. I don't think $cov is out of
> scope by that point.
>
>  Sincerely,
>
>  David Myers.
>
>
>
>
> ------------------------------------------------------------------------------
> WhatsUp Gold - Download Free Network Management Software
> The most intuitive, comprehensive, and cost-effective network
> management toolset available today.  Delivers lowest initial
> acquisition cost and overall TCO of any competing solution.
> http://p.sf.net/sfu/whatsupgold-sd
> _______________________________________________
> pdl-stats-help mailing list
> pdl...@li...
> https://lists.sourceforge.net/lists/listinfo/pdl-stats-help
>
>

[pdl-stats] A PDL::Stats change I'd love to see

[<dw...@em...>]

Thank you Maggie.



What I'd like to be able to do, with things like the logistic function (any curve fitter
in PDL::Stats, actually) is something like this:


my %m = $y->logistic( COV => 1 );
print $m{cov}->corr_table();


I don't think logistic should return the covariance matrix by default, but I would like
to be able to get to it.


A single line patch in GLM.pm, after %ret is declared, around line 2000 or so..


$ret{cov} = $cov if ( $opt{COV} );


would do the trick for logistic, I believe. I don't think $cov is out of scope by that point.


Sincerely,


David Myers.

Re: [pdl-stats] Detecting cross correlation in logistic regression independent parms.

[Maggie X. <mag...@us...>]

In the context of logistic regression, by cross correlation you mean Pearson
correlation matrix for a set of variables? The function corr_table() in
PDL::Stats::Basic can give you that. You can also calculate it from
covariance matrix by dividing the covariance with the product of of the
standard deviations of the two variables you are correlating with.


Best,
Maggie


On Mon, Apr 25, 2011 at 10:54 PM, <dw...@em...> wrote:

>
> I apologize in advance if this is obvious. I'm an ex-scientist, from a
> generation  for which Bevington's "Data Reduction
> and Error Analysis in the Physical Sciences" was our Bible. To determine
> cross correlation of a fit, we'd extract
> the covariance matrix from our fitters, do some math presto magico things,
> and then could determine if
> we were fitting cross correlated data (then drop one of the correlated
> coefficients from our models..).
>
>  I'm working with PDL::Stats logistic method. Works great, but haven't a
> clue how to detect cross correlation.
> I suspect the "how" is obvious to a stats major, but not to an
> ex-biochemist like me. The underlying fitter
> returns $cov, but no clue how to get to it.
>
>  Any help appreciated.
>
>  Sincerely,
>
>  David Myers.
>
>
> ------------------------------------------------------------------------------
> WhatsUp Gold - Download Free Network Management Software
> The most intuitive, comprehensive, and cost-effective network
> management toolset available today.  Delivers lowest initial
> acquisition cost and overall TCO of any competing solution.
> http://p.sf.net/sfu/whatsupgold-sd
> _______________________________________________
> pdl-stats-help mailing list
> pdl...@li...
> https://lists.sourceforge.net/lists/listinfo/pdl-stats-help
>
>

[pdl-stats] Detecting cross correlation in logistic regression independent parms.

[<dw...@em...>]

I apologize in advance if this is obvious. I'm an ex-scientist, from a generation  for which Bevington's "Data Reduction

and Error Analysis in the Physical Sciences" was our Bible. To determine cross correlation of a fit, we'd extract
the covariance matrix from our fitters, do some math presto magico things, and then could determine if
we were fitting cross correlated data (then drop one of the correlated coefficients from our models..).


I'm working with PDL::Stats logistic method. Works great, but haven't a clue how to detect cross correlation.
I suspect the "how" is obvious to a stats major, but not to an ex-biochemist like me. The underlying fitter 
returns $cov, but no clue how to get to it.


Any help appreciated.


Sincerely,


David Myers.