infomap-nlp-users

Re: [infomap-nlp-users] svdinterface on 64-bit systems

[Bertram, A. <ber...@um...>]

Hi,

I don't know if anyone else has looked at infomap-build, or more
specifically svdinterface, segmentation faulting on 64 bit systems in
the past few months, but what I've come to determine is this:

On the 64-bit system to which I have access, calls to
myutils.c::mymalloc() from within las2.c truncate results to 32-bits.
This produces very unfortunate behavior such as: free()ing memory
immediately after mymalloc() causes a segmentation fault.  In testing a
small driver file which used myutils.c::mymalloc(), everything works
fine.  Regular malloc() in las2.c works fine.  I haven't yet looked into
why this was happening, but I don't really plan to at this point, as I
have a solution which works for me at the moment.

My solution is to replace mymalloc() calls with malloc().  This means
you lose all the error testing on malloc, but that's somewhat better
than causing segmentation faults.

If someone figures out why this is going on I'd be delighted to know,
but until then, I figured I'd share what I saw going on.

Regards,

Albert Bertram

Re: [infomap-nlp-users] svdinterface on 64-bit systems

[Bertram, A. <ber...@um...>]

Hi again,

For anyone interested, I identified a simpler solution, and the root
cause of this problem.

Most, and maybe all, C compilers will assume type int for undeclared
functions.  In las2.c, the mymalloc() function is not declared, thus it
was assumed to be of type int, which on most systems is 32 bits long,
and the 64 bit addresses were truncated.  On 32 bit compilers this
doesn't present a problem.

The more elegant solution to this problem is to add:

#include "myutils.h"=20

to las2.c, which contains declarations for mymalloc().  I performed a
quick test, and found this to work for me on the 64 bit linux system I
use.

Regards,

Albert Bertram

> -----Original Message-----
> From: Bertram, Albert=20
> Sent: Tuesday, September 25, 2007 3:12 PM
> To: 'inf...@li...'
> Subject: Re: [infomap-nlp-users] svdinterface on 64-bit systems
>=20
> Hi,
>=20
> I don't know if anyone else has looked at infomap-build, or=20
> more specifically svdinterface, segmentation faulting on 64=20
> bit systems in the past few months, but what I've come to=20
> determine is this:
>=20
> On the 64-bit system to which I have access, calls to=20
> myutils.c::mymalloc() from within las2.c truncate results to=20
> 32-bits.  This produces very unfortunate behavior such as:=20
> free()ing memory immediately after mymalloc() causes a=20
> segmentation fault.  In testing a small driver file which=20
> used myutils.c::mymalloc(), everything works fine.  Regular=20
> malloc() in las2.c works fine.  I haven't yet looked into why=20
> this was happening, but I don't really plan to at this point,=20
> as I have a solution which works for me at the moment.
>=20
> My solution is to replace mymalloc() calls with malloc(). =20
> This means you lose all the error testing on malloc, but=20
> that's somewhat better than causing segmentation faults.
>=20
> If someone figures out why this is going on I'd be delighted=20
> to know, but until then, I figured I'd share what I saw going on.
>=20
> Regards,
>=20
> Albert Bertram
>=20

[infomap-nlp-users] New Semantic Vectors Package, and request for beta testers

[Dominic W. <dwi...@cs...>]

Dear All,

I'd like to draw people's attention to a new Semantic Vectors package that
I've been working on. The project is hosted at
http://code.google.com/p/semanticvectors/
and there is a beta release available along with some test data.

As many of you are aware, the Infomap NLP package has suffered from two
main problems over the years, which are ease of installation and use, and
scalability. The new Semantic Vectors has a good chance or providing
significant improvements in both these areas. For ease of use, it's
implemented entirely in Java, and its only dependencies are Apache Lucene,
Ant, and having Java itself installed. For scalability, it uses Random
Projection instead of Singular Value Decomposition, which is much more
scalable and drawing increasing interest in the research community.
For a more thorough comparison with the Infomap NLP package, see
http://code.google.com/p/semanticvectors/wiki/InfomapComparison

For anyone on the list interested in the new package, I would really
appreciate it if you'd give it a try, kick the tires, let me know if it
works for you. It would be great to have a couple of reports from beta
testers before I announce the package on the corpora mailing list.

While it's way premature to consider the Infomap NLP package as
deprecated, I think it may become gradually superseded by the new Semantic
Vectors package, and I will probably be putting more of my own development
efforts into the new package than into any further releases of Infomap
NLP. However, I'd really like to thank everyone who's contributed code,
fixes, support and collaboration with Infomap NLP, and I hope people
continue to find good uses for the software. I will continue to answer
questions and give help where possible.

Happy semantic modelling!
Best wishes,
Dominic

Re: [infomap-nlp-users] svdinterface on 64-bit systems

[Scott C. <ced...@gm...>]

Hi Albert,

    Thank you!  I looked into this a while back, and what you're
saying corresponds to what I found, although I didn't get as far as an
actual fix.  Have you been able to run the modified code successfully
on 64-bit systems?


               Scott

On 9/25/07, Bertram, Albert <ber...@um...> wrote:
> Hi,
>
> I don't know if anyone else has looked at infomap-build, or more
> specifically svdinterface, segmentation faulting on 64 bit systems in
> the past few months, but what I've come to determine is this:
>
> On the 64-bit system to which I have access, calls to
> myutils.c::mymalloc() from within las2.c truncate results to 32-bits.
> This produces very unfortunate behavior such as: free()ing memory
> immediately after mymalloc() causes a segmentation fault.  In testing a
> small driver file which used myutils.c::mymalloc(), everything works
> fine.  Regular malloc() in las2.c works fine.  I haven't yet looked into
> why this was happening, but I don't really plan to at this point, as I
> have a solution which works for me at the moment.
>
> My solution is to replace mymalloc() calls with malloc().  This means
> you lose all the error testing on malloc, but that's somewhat better
> than causing segmentation faults.
>
> If someone figures out why this is going on I'd be delighted to know,
> but until then, I figured I'd share what I saw going on.
>
> Regards,
>
> Albert Bertram
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> infomap-nlp-users mailing list
> inf...@li...
> https://lists.sourceforge.net/lists/listinfo/infomap-nlp-users
>

Re: [infomap-nlp-users] svdinterface on 64-bit systems

[<lei...@gm...>]

Hi,

I had a similar error but I think it's fixed in the latest version of
Infomap. The problem was that integers and pointers were stored in the same
arrays. This is fine as long as pointers are 32 bits long but on a 64 bit
SUN system I had to update this. If this bug is still around I could try to
find the updated source.

Regards,
Leif

On 9/25/07, Bertram, Albert <ber...@um...> wrote:
>
> Hi,
>
> I don't know if anyone else has looked at infomap-build, or more
> specifically svdinterface, segmentation faulting on 64 bit systems in
> the past few months, but what I've come to determine is this:
>
> On the 64-bit system to which I have access, calls to
> myutils.c::mymalloc() from within las2.c truncate results to 32-bits.
> This produces very unfortunate behavior such as: free()ing memory
> immediately after mymalloc() causes a segmentation fault.  In testing a
> small driver file which used myutils.c::mymalloc(), everything works
> fine.  Regular malloc() in las2.c works fine.  I haven't yet looked into
> why this was happening, but I don't really plan to at this point, as I
> have a solution which works for me at the moment.
>
> My solution is to replace mymalloc() calls with malloc().  This means
> you lose all the error testing on malloc, but that's somewhat better
> than causing segmentation faults.
>
> If someone figures out why this is going on I'd be delighted to know,
> but until then, I figured I'd share what I saw going on.
>
> Regards,
>
> Albert Bertram
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> infomap-nlp-users mailing list
> inf...@li...
> https://lists.sourceforge.net/lists/listinfo/infomap-nlp-users
>



--=20
Leif Gr=F6nqvist lei...@gm...
www.gslt.hum.gu.se/~leifg   0707-164380

Re: [infomap-nlp-users] svdinterface on 64-bit systems

[Bertram, A. <ber...@um...>]

Hi Scott,

The modified code works for me.  Using some sample data, I'm able to
finish the infomap-build, and search using associate. The results look
reasonable.  This is all on 64-bit linux.

%uname -a
Linux <hostname> 2.6.17.6 #1 SMP Thu Feb 8 16:09:22 EST 2007 x86_64
GNU/Linux=20

I don't think I have access to any other 64 bit environments though.  So
I can't guarantee it works on solaris or whatever else may be out there.

Regards

Albert

> -----Original Message-----
> From: Scott Cederberg [mailto:ced...@gm...]=20
> Sent: Tuesday, September 25, 2007 3:30 PM
> To: Bertram, Albert
> Cc: inf...@li...
> Subject: Re: [infomap-nlp-users] svdinterface on 64-bit systems
>=20
> Hi Albert,
>=20
>     Thank you!  I looked into this a while back, and what=20
> you're saying corresponds to what I found, although I didn't=20
> get as far as an actual fix.  Have you been able to run the=20
> modified code successfully on 64-bit systems?
>=20
>=20
>                Scott
>=20
> On 9/25/07, Bertram, Albert <ber...@um...> wrote:
> > Hi,
> >
> > I don't know if anyone else has looked at infomap-build, or more=20
> > specifically svdinterface, segmentation faulting on 64 bit=20
> systems in=20
> > the past few months, but what I've come to determine is this:
> >
> > On the 64-bit system to which I have access, calls to
> > myutils.c::mymalloc() from within las2.c truncate results=20
> to 32-bits.
> > This produces very unfortunate behavior such as: free()ing memory=20
> > immediately after mymalloc() causes a segmentation fault. =20
> In testing=20
> > a small driver file which used myutils.c::mymalloc(),=20
> everything works=20
> > fine.  Regular malloc() in las2.c works fine.  I haven't yet looked=20
> > into why this was happening, but I don't really plan to at=20
> this point,=20
> > as I have a solution which works for me at the moment.
> >
> > My solution is to replace mymalloc() calls with malloc(). =20
> This means=20
> > you lose all the error testing on malloc, but that's=20
> somewhat better=20
> > than causing segmentation faults.
> >
> > If someone figures out why this is going on I'd be=20
> delighted to know,=20
> > but until then, I figured I'd share what I saw going on.
> >
> > Regards,
> >
> > Albert Bertram
> >
> >=20
> ----------------------------------------------------------------------
> > --- This SF.net email is sponsored by: Microsoft Defy all=20
> challenges.=20
> > Microsoft(R) Visual Studio 2005.
> > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > _______________________________________________
> > infomap-nlp-users mailing list
> > inf...@li...
> > https://lists.sourceforge.net/lists/listinfo/infomap-nlp-users
> >
>=20

Re: [infomap-nlp-users] svdinterface on 64-bit systems

[<lei...@gm...>]

On Solaris-64bit I had to change the structure of the xv1-array to a
2-dimensional array. The allocation in my code looks like:

  /* compute eigenvectors */
  if (vectors) {
    int i;

    fprintf(stderr, "nrows: %d, ncols: %d, steps: %d\n", nrow, ncol, j);
    if (!(xv1 =3D (double **) mymalloc((j+1)*sizeof(double), "las2: xv1")) =
||
!(xv2 =3D (double *) mymalloc(ncol * sizeof(double), "las2: xv2")))
errormessage("SECOND MALLOC FAILED in LANDR()","",NON);
    for(i=3D0; i<=3Dj; i++) {
      char t[25];
      sprintf(t, "las2: xv1 %d", i);
      if(!(xv1[i]=3Dmymalloc((nrow+ncol)*sizeof(double), t)))
errormessage("Third MALLOC FAILED in LANDR()","",NON);
    }
    kappa =3D max(fabs(kappa), eps34);
    ritvec(n,kappa,ritz,bnd,wptr[6],wptr[9],wptr[4],wptr[5],right,maxprs);
  }

instead of:

  /* compute eigenvectors */
  if (vectors) {
    if (!(xv1 =3D (double *) mymalloc((nrow+ncol)*(j+1)*sizeof(double))) ||
    !(xv2 =3D (double *) mymalloc(ncol * sizeof(double))))
      errormessage("SECOND MALLOC FAILED in LANDR()","",NON);
    kappa =3D dmax(fabs(kappa), eps34);
    ritvec(n,kappa,ritz,bnd,wptr[6],wptr[9],wptr[4],wptr[5],right,maxprs);
  }


And in ritvec():

       * Obs! xv1 is now 2-dimensional to avoid 32-bit memory
       * problems. Th shape is xv1[0 - j][0 - (nrow+ncol-1)] */

      for (i =3D 0; i < n; i++) xv1[kk][i] =3D w1[i];
      nsig +=3D 1;
      kk++;
    }
    id2++;

instead of:

      for (i =3D 0; i < n; i++) xv1[id++] =3D w1[i];
      id +=3D nrow;
      nsig +=3D 1;
    }
    id2++;

Maybe there are some more places, but it's a bit difficult to find since I
made more changes to be able to scale up.

Regards,
Leif


On 9/25/07, Bertram, Albert <ber...@um...> wrote:
>
> Hi Scott,
>
> The modified code works for me.  Using some sample data, I'm able to
> finish the infomap-build, and search using associate. The results look
> reasonable.  This is all on 64-bit linux.
>
> %uname -a
> Linux <hostname> 2.6.17.6 #1 SMP Thu Feb 8 16:09:22 EST 2007 x86_64
> GNU/Linux
>
> I don't think I have access to any other 64 bit environments though.  So
> I can't guarantee it works on solaris or whatever else may be out there.
>
> Regards
>
> Albert
>
> > -----Original Message-----
> > From: Scott Cederberg [mailto:ced...@gm...]
> > Sent: Tuesday, September 25, 2007 3:30 PM
> > To: Bertram, Albert
> > Cc: inf...@li...
> > Subject: Re: [infomap-nlp-users] svdinterface on 64-bit systems
> >
> > Hi Albert,
> >
> >     Thank you!  I looked into this a while back, and what
> > you're saying corresponds to what I found, although I didn't
> > get as far as an actual fix.  Have you been able to run the
> > modified code successfully on 64-bit systems?
> >
> >
> >                Scott
> >
> > On 9/25/07, Bertram, Albert <ber...@um...> wrote:
> > > Hi,
> > >
> > > I don't know if anyone else has looked at infomap-build, or more
> > > specifically svdinterface, segmentation faulting on 64 bit
> > systems in
> > > the past few months, but what I've come to determine is this:
> > >
> > > On the 64-bit system to which I have access, calls to
> > > myutils.c::mymalloc() from within las2.c truncate results
> > to 32-bits.
> > > This produces very unfortunate behavior such as: free()ing memory
> > > immediately after mymalloc() causes a segmentation fault.
> > In testing
> > > a small driver file which used myutils.c::mymalloc(),
> > everything works
> > > fine.  Regular malloc() in las2.c works fine.  I haven't yet looked
> > > into why this was happening, but I don't really plan to at
> > this point,
> > > as I have a solution which works for me at the moment.
> > >
> > > My solution is to replace mymalloc() calls with malloc().
> > This means
> > > you lose all the error testing on malloc, but that's
> > somewhat better
> > > than causing segmentation faults.
> > >
> > > If someone figures out why this is going on I'd be
> > delighted to know,
> > > but until then, I figured I'd share what I saw going on.
> > >
> > > Regards,
> > >
> > > Albert Bertram
> > >
> > >
> > ----------------------------------------------------------------------
> > > --- This SF.net email is sponsored by: Microsoft Defy all
> > challenges.
> > > Microsoft(R) Visual Studio 2005.
> > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > > _______________________________________________
> > > infomap-nlp-users mailing list
> > > inf...@li...
> > > https://lists.sourceforge.net/lists/listinfo/infomap-nlp-users
> > >
> >
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> infomap-nlp-users mailing list
> inf...@li...
> https://lists.sourceforge.net/lists/listinfo/infomap-nlp-users
>



--=20
Leif Gr=F6nqvist lei...@gm...
www.gslt.hum.gu.se/~leifg   0707-164380