KlustaKwik spike sorting

Hi Peter,

The reason for the update was that Michael Zugaro (CC'ed) found and fixed a bug. 

I'm afraid the versions have got in a bit of a mess. When Michael found the bug, he asked me what was the latest version and I had to admit I didn't know. 

We are going to be working on it here soon (trying to get it to work for large channel counts). We will only work on the C++ version, and will do our best to avoid more version forks. 

Does the latest C++ version compile on a Mac? (Warnings are one thing, errors another...)

All the best,
Kenneth.

-----Original Message-----
From: Peter Nathan Steinmetz [mailto:peternsteinmetz@...] 
Sent: 20 February 2011 20:23
To: klustakwik-develop@...
Subject: [Klustakwik-develop] could always reverse master/ warnings

Hi All,

I suppose we could always change the master branch to be the C++ development line and branch the java line, just a matter of the branch you checkout and work on.

BTW, I just tried compiling the C++ version here on a Mac and it throws a log of warnings about a redefinition of M_PI and a lot about conversions from a string constant to char* (in general, variables that are string constants should be const char* to avoid this).

cheers,
Peter



------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
Klustakwik-develop mailing list
Klustakwik-develop@...
https://lists.sourceforge.net/lists/listinfo/klustakwik-develop

Hi Ken & Michael,

In terms of your question:

> Does the latest C++ version compile on a Mac? (Warnings are one thing, errors another...)

Yes, it compiles and runs, it just throws a whole bunch of these warnings, mostly about const char *

In terms of the bug which was fixed, can you indicate which lines or files were affected? I'd like to make sure the java version incorporates any fix, if needed.

thanks,
Peter

> Hi Ken & Michael,
>
> In terms of your question:
> > Does the latest C++ version compile on a Mac? (Warnings are one thing,
> > errors another...)
>
> Yes, it compiles and runs, it just throws a whole bunch of these warnings,
> mostly about const char *
>
> In terms of the bug which was fixed, can you indicate which lines or files
> were affected? I'd like to make sure the java version incorporates any fix,
> if needed.
>
> thanks,
> Peter

Hi Peter and Ken,

Yes, that was my concern too. Some versions of KlustaKwik had subdirectories 
(linux, mac, windows) and others didn't. But there were many versions 
floating around (some with a number, e.g. 2.0, others with a suffix, 
e.g. Subsetter), and nobody seemed to remember which was the latest.

Anyway, although debugging the recurrent problems we and others were 
experiencing did take some effort, in the end the fix turned out to be *very* 
simple. The problem arises when KlustaKwik tries to split a cluster, ends up 
with all points in the same cluster, then later aborts because splitting 
should have produced two clusters.

To correct this, it is sufficient to replace the following code (on line
700 of KlustaKwik 1.5, but this may vary depending on the version):

   if(SplitScore<UnsplitScore) {

with:

   if(K2.nClustersAlive<2) Output("Split failed - leaving alone\n");
   if(SplitScore<UnsplitScore&K2.nClustersAlive>=2) {

That's it!

Michaël

-- 
Michaël Zugaro
CNRS - Collège de France, LPPA
11, place Marcelin Berthelot
75005 Paris
Tel (33) 1 44 27 12 93
Fax (33) 1 44 27 13 82

Hi Michael,

Thanks for the update. I've committed the same change in the jKlustaKwik version. 

As you are working on the R1.5 C++ branch, let me know if you'd like any assistance with check in and check out of the source code from the git repository. I'd never heard of a version 2 and if we can keep things checked in, it may flow more smoothly for the next person or group in the future.

cheers,
Peter


On Feb 24, 2011, at 4:35 AM, Michael Zugaro wrote:

> 
> Hi Peter and Ken,
> 
> Yes, that was my concern too. Some versions of KlustaKwik had subdirectories 
> (linux, mac, windows) and others didn't. But there were many versions 
> floating around (some with a number, e.g. 2.0, others with a suffix, 
> e.g. Subsetter), and nobody seemed to remember which was the latest.
> 
> Anyway, although debugging the recurrent problems we and others were 
> experiencing did take some effort, in the end the fix turned out to be *very* 
> simple. The problem arises when KlustaKwik tries to split a cluster, ends up 
> with all points in the same cluster, then later aborts because splitting 
> should have produced two clusters.
> 
> To correct this, it is sufficient to replace the following code (on line
> 700 of KlustaKwik 1.5, but this may vary depending on the version):
> 
>   if(SplitScore<UnsplitScore) {
> 
> with:
> 
>   if(K2.nClustersAlive<2) Output("Split failed - leaving alone\n");
>   if(SplitScore<UnsplitScore&K2.nClustersAlive>=2) {
> 
> That's it!
> 
> Michaël
> 
> -- 
> Michaël Zugaro
> CNRS - Collège de France, LPPA
> 11, place Marcelin Berthelot
> 75005 Paris
> Tel (33) 1 44 27 12 93
> Fax (33) 1 44 27 13 82

Hi Peter,

Actually, I am not even sure versions 1.8 and 2.0 are actually different. 
Also, there is the 'subsetter' version which may or may not be the same as 
these. Finally, it's the first time I've heard of a Java port... Why was 
KlustaKwik ported to Java (I could guess 'platform independence', but 
gui-less C/C++ is portable)? How efficient is the Java code compared to 
C/C++? How easy is it to have jobs managed by a queuing system?

Cheers,

-- 
Michaël Zugaro
CNRS - Collège de France, LPPA
11, place Marcelin Berthelot
75005 Paris
Tel (33) 1 44 27 12 93
Fax (33) 1 44 27 13 82

> Hi Michael,
>
> Thanks for the update. I've committed the same change in the jKlustaKwik
> version.
>
> As you are working on the R1.5 C++ branch, let me know if you'd like any
> assistance with check in and check out of the source code from the git
> repository. I'd never heard of a version 2 and if we can keep things
> checked in, it may flow more smoothly for the next person or group in the
> future.
>
> cheers,
> Peter
>
> On Feb 24, 2011, at 4:35 AM, Michael Zugaro wrote:
> > Hi Peter and Ken,
> >
> > Yes, that was my concern too. Some versions of KlustaKwik had
> > subdirectories (linux, mac, windows) and others didn't. But there were
> > many versions floating around (some with a number, e.g. 2.0, others with
> > a suffix, e.g. Subsetter), and nobody seemed to remember which was the
> > latest.
> >
> > Anyway, although debugging the recurrent problems we and others were
> > experiencing did take some effort, in the end the fix turned out to be
> > *very* simple. The problem arises when KlustaKwik tries to split a
> > cluster, ends up with all points in the same cluster, then later aborts
> > because splitting should have produced two clusters.
> >
> > To correct this, it is sufficient to replace the following code (on line
> > 700 of KlustaKwik 1.5, but this may vary depending on the version):
> >
> >   if(SplitScore<UnsplitScore) {
> >
> > with:
> >
> >   if(K2.nClustersAlive<2) Output("Split failed - leaving alone\n");
> >   if(SplitScore<UnsplitScore&K2.nClustersAlive>=2) {
> >
> > That's it!
> >
> > Michaël
> >
> > --
> > Michaël Zugaro
> > CNRS - Collège de France, LPPA
> > 11, place Marcelin Berthelot
> > 75005 Paris
> > Tel (33) 1 44 27 12 93
> > Fax (33) 1 44 27 13 82

Hi Michael,

I was just looking at this a bit in the repository available at: http://klustakwik.git.sourceforge.net/git/gitweb-index.cgi

I've also never heard of a 2.0, and there wasn't such a tag in the repository. Perhaps it was done outside the project itself. 

In the old release system, a release could have a name as well as a number. Looking at the dates, I think Subsetter was just the name for the 1.8 release.

It was just after this that I decided to start on the Java port. The changes made to the C++ version along the 1.8 branch had been made for a number of reasons. Mostly to allow use of the core computing functionality in a more flexible way (e.g. not having to have features in a file named .fet.1, not having to set binary flags for the features to use, etc) as well as to try and clarify the structure of the code. Since we use primarily Java in the lab, I just decided it would be easier to have a Java version that would fit in with our other code and be more readily callable from it.

There were a number of other perceived advantages that also motivated this: Easy use of the JUnit testing framework to have explicit tests for things like the M-step working correctly, even after changes are made; binary multi-platform compatibility (don't need to recompile can just run it from the jar on any system with a JVM); and easier programming (I think I program ten times more quickly in Java than in C++, though I am expert in both, likely due to the simpler syntax of Java and only one file for each class, instead of a .h and .cpp that have to be kept in sync).

In terms of performance, the Java version runs perhaps 10-30% longer than the C++ version, really not bad at all.

We've used Java on a number of clustering systems, the commands queue up just fine. Another reason for this move was our desire to migrate toward using Hadoop (http://hadoop.apache.org/), which permits loosely coupled cluster computing on disparate hardware, and this is more easily done with Java (though can also be done with C++).

The Java port was really a background project, way down in priority below grants and papers, and so I just released R1.0 last week. Your testing or and comments on it would certainly be most welcome if you'd like to give it a spin.

cheers,
Peter

On Feb 25, 2011, at 2:51 AM, Michael Zugaro wrote:

> Hi Peter,
> 
> Actually, I am not even sure versions 1.8 and 2.0 are actually different. 
> Also, there is the 'subsetter' version which may or may not be the same as 
> these. Finally, it's the first time I've heard of a Java port... Why was 
> KlustaKwik ported to Java (I could guess 'platform independence', but 
> gui-less C/C++ is portable)? How efficient is the Java code compared to 
> C/C++? How easy is it to have jobs managed by a queuing system?
> 
> Cheers,
> 
> -- 
> Michaël Zugaro
> CNRS - Collège de France, LPPA
> 11, place Marcelin Berthelot
> 75005 Paris
> Tel (33) 1 44 27 12 93
> Fax (33) 1 44 27 13 82

-- 
Peter N. Steinmetz, M.D.,Ph.D.
Program Director, Neuroengineering
Barrow Neurological Institute
PeterNSteinmetz@...
602-406-3258
http://steinmetz.org/peter