Re: [Denovoassembler-devel] git diff kmersmatrix branch

[Sébastien B. <se...@bo...>]

I merged your work.

Cheer.

----------------------------------------
> Date: Fri, 7 Mar 2014 10:57:14 +0000
> From: max...@gm...
> To: se...@bo...; den...@li...
> Subject: Re: [Denovoassembler-devel] git diff kmersmatrix branch
>
> Hi Seb,
>
> You can find these modifications in the same branch as before :
>
> https://github.com/Zorino/ray.git
>
> patch-kmermatrix
>
>
> On 03/07/2014 02:18 AM, Sébastien Boisvert wrote:
>> Hi Maxime,
>>
>> I can't merge your code in master without making some changes first.
>>
>> As the maintainer, here are the changes that I will need to do to increase the plus-value of your work (on
>> your next pull request, you can think about some of these points if you want):
>>
>> Keep in mind that Coding style is very important for readability.
>> (you can check https://github.com/sebhtml/ray/blob/master/Documentation/CodingStyle.txt )
>>
>> Here are the changes that I will make tomorrow:
>>
>> - (C0) As discussed, we don't want to generate Surveyor/KmerMatrix.tsv by default because this is
>> quite large file. There needs to be an option.
>>
>> - (C1) Obviously, if you want people to use your Surveyor workflow, you need to add documentation.
>> You just need to add some code in code/Mock/Parameters.cpp. After that you build Ray,
>> then you write ./Ray -help> MANUAL_PAGE.txt
>
> Yes there was an option needed to launch it so it wasn't by default but
> I forgot to add it in the Parameter file (now -run-kmer-matrix).Done.
>
>> - (C2) KmerMatrixOwner.h/cpp was done by you. So the copyright in the header belongs to you, not
>> me.
>
> done.
>
>>
>> - (C2) there is some red showing up in 'git diff --color', nothing important though
>> relevant commits ce6c272 & 4e4949
>>
>> - (C3) Copyright for code/Surveyor/SequenceKmerReader.cpp/.h should be in 2014
>
> done.
>
>> - (C4) In code/Surveyor/StoreKeeper.h you must use tabulations and no spaces for indentation.
>> for example, the line with MERGE_KMER_MATRIX uses spaces in StoreKeeper.h
>
> Yes, my indent-mode in emacs suddenly decided to put 8 spaces instead of
> tabs I corrected this and re-indented it all.
>
>> - (C5) code/Surveyor/KmerMatrixOwner.cpp was using spaces instead of tabulations.
>>
>> - (C6) the method name KmerMatrixOwner::printLocalKmersMatrix is meaningless as it does not
>> write a matrix, it writes a kmer.
>
> now called dumpKmerMatrixBuffer since it is not writing a single kmer
> but a couple of it with a utility function of yours called
> flushFileOperationBuffer
>
>> - (C7) you hard-coded the kmer value (31) !!! m_kmerMatrix << kmer.idToWord(31,0);
>> you must avoid that ! [KmerMatrixOwner.cpp]
>
> big mistake here fixed it with m_parameters->getWordSize()
>
>> - (C8) The FIRST_TAG for your reader is the same that is used by the graph reader. I think we said it
>> was OK as long as you added a comment about the reason.
>> code/Surveyor/GenomeAssemblyReader.h: FIRST_TAG = 10200,
>> code/Surveyor/GenomeGraphReader.h: FIRST_TAG = 10200,
>
> added this comment :
> // Using the same tag as GenomeGraphReader
> // because we can mix an assembly reader with a graph reader
>
>>
>> Keep up the great work, but next time, I think you can make progress on respecting the coding style,
>> among other things.
>
> thanks, I understand the importance you give to coding style.
>
>> I will do a bunch of commits in my branch patch-kmermatrix (fetched from
>> remotes/zorino/patch-kmermatrix) to address the comments above.
>>
>>
>> [boiseb01@ls30 ray]$ git diff master..remotes/zorino/patch-kmermatrix --stat
>> code/Surveyor/GenomeAssemblyReader.cpp | 3 +-
>> code/Surveyor/KmerMatrixOwner.cpp | 157 ++++++++++++++++++++
>> code/Surveyor/KmerMatrixOwner.h | 72 +++++++++
>> code/Surveyor/Makefile | 1 +
>> code/Surveyor/MatrixOwner.cpp | 19 +--
>> code/Surveyor/MatrixOwner.h | 3 +-
>> code/Surveyor/Mother.cpp | 253 +++++++++++++++++++++----------
>> code/Surveyor/Mother.h | 17 ++-
>> code/Surveyor/SequenceKmerReader.cpp | 53 ++++++-
>> code/Surveyor/SequenceKmerReader.h | 2 +
>> code/Surveyor/StoreKeeper.cpp | 117 +++++++++++----
>> code/Surveyor/StoreKeeper.h | 20 +++-
>> 12 files changed, 580 insertions(+), 137 deletions(-)
>>
>> Link: http://sourceforge.net/p/denovoassembler/mailman/denovoassembler-devel/thread/017C19ECDB98F64F99200D83876C44520111A7FC4FE7%40EXCH-MBX-B.ulaval.ca/#msg31999514
>> Link: http://sourceforge.net/p/denovoassembler/mailman/denovoassembler-devel/thread/COL131-W760CDDC5071A415AB9C5EDAC870%40phx.gbl/#msg32015211
>> Link: http://sourceforge.net/p/denovoassembler/mailman/denovoassembler-devel/thread/f2d8f53ef133555e36529047f4e95084%40boisvert.info/#msg31993547
>> Link: http://sourceforge.net/p/denovoassembler/mailman/denovoassembler-devel/thread/5301F42B.6060904%40gmail.com/#msg31988453
>>
>> ----------------------------------------
>>> Date: Wed, 5 Mar 2014 13:41:39 +0000
>>> From: max...@gm...
>>> To: se...@bo...
>>> Subject: Re: [Denovoassembler-devel] git diff kmersmatrix branch
>>>
>>> Hi Sebastien,
>>>
>>> I have fix the issue when scaffolds would be given in entry instead of
>>> contigs in the SequenceKmerReader class.
>>>
>>> I also made the edition according to this review.
>>>
>>> Please now pull from :
>>>
>>> https://github.com/Zorino/ray.git
>>>
>>> patch-kmermatrix
>>>
>>> Cheers,
>>>
>>> Maxime
>>>
>>>
>>>
>>> On 02/23/2014 12:16 PM, Sébastien Boisvert wrote:
>>>> Hey Maxime,
>>>>
>>>> You did not provide KmersMatrixOwner.h, KmersMatrixOwner.cpp, and changes
>>>> to the Surveyor Makefile.
>>>>
>>>>
>>>> OTher comments are below.
>>>>
>>>> ----------------------------------------
>>>>> Date: Sat, 22 Feb 2014 10:03:30 +0000
>>>>> From: ma...@de...
>>>>> To: se...@bo...
>>>>> Subject: git diff kmersmatrix branch
>>>>>
>>>>> diff --git a/code/Surveyor/MatrixOwner.cpp b/code/Surveyor/MatrixOwner.cpp
>>>>> index ffaae00..47cf84a 100644
>>>>> --- a/code/Surveyor/MatrixOwner.cpp
>>>>> +++ b/code/Surveyor/MatrixOwner.cpp
>>>>> @@ -65,9 +65,12 @@ void MatrixOwner::receive(Message & message) {
>>>>> assert(m_parameters != NULL);
>>>>> assert(m_sampleNames != NULL);
>>>>> #endif
>>>>> -
>>>>> m_mother = source;
>>>>>
>>>>> + //open the buffer of the file
>>>>> + // createKmersMatrixOutputFile();
>>>>> +
>>>>> +
>>>>> } else if(tag == PUSH_PAYLOAD) {
>>>>>
>>>>> SampleIdentifier sample1 = -1;
>>>>> @@ -89,10 +92,10 @@ void MatrixOwner::receive(Message & message) {
>>>>> assert(count>= 0);
>>>>> #endif
>>>>>
>>>>> - /*
>>>>> +
>>>>> printName();
>>>>> - cout << "DEBUG add " << sample1 << " " << sample2 << " "
>>>>> << count << endl;
>>>>> -*/
>>>>> + // cout << "DEBUG add " << sample1 << " " << sample2 <<
>>>> Commented lines should be removed.
>>>>
>>>>> " " << count << endl;
>>>>> +
>>>>> m_receivedPayloads ++;
>>>>>
>>>>> m_localGramMatrix[sample1][sample2] += count;
>>>>> @@ -100,14 +103,14 @@ void MatrixOwner::receive(Message & message) {
>>>>> Message response;
>>>>> response.setTag(PUSH_PAYLOAD_OK);
>>>>> send(source, response);
>>>>> + }
>>>>> + else if(tag == PUSH_PAYLOAD_END) {
>>>> Use '} else if (' and not '}
>>>> else if'
>>>>
>>>>
>>>> This is the coding style of the project.
>>>> see https://github.com/sebhtml/ray/blob/master/Documentation/CodingStyle.txt
>>>>
>>>> * Kernighan and Ritchie style, variant "The One True Brace Style" (1TBS)
>>>> http://en.wikipedia.org/wiki/Indent_style#Variant:_1TBS
>>>>
>>>>
>>>> (you used K&R Variant: Stroustrup).
>>>>
>>>>> - } else if(tag == PUSH_PAYLOAD_END) {
>>>>> -
>>>>> + cout << "PUSH_PAYLOAD_END" <<endl;
>>>> Remove this debug message.
>>>>
>>>>> m_completedStoreActors++;
>>>>>
>>>>> if(m_completedStoreActors == getSize()) {
>>>>>
>>>>> -
>>>>> printName();
>>>>> cout << "MatrixOwner received " <<
>>>>> m_receivedPayloads << " payloads" << endl;
>>>>>
>>>>> @@ -151,10 +154,9 @@ void MatrixOwner::receive(Message & message) {
>>>>>
>>>>>
>>>>> // tell Mother that the matrix is ready now.
>>>>> -
>>>>> - Message coolMessage;
>>>>> - coolMessage.setTag(MATRIX_IS_READY);
>>>>> - send(m_mother, coolMessage);
>>>>> + Message coolMessage;
>>>>> + coolMessage.setTag(GRAM_MATRIX_IS_READY);
>>>>> + send(m_mother, coolMessage);
>>>>>
>>>>>
>>>>> // clear matrices
>>>>> @@ -275,3 +277,4 @@ void
>>>>> MatrixOwner::printLocalGramMatrixWithHash(ostream & stream, map<SampleIdent
>>>>> stream << endl;
>>>>> }
>>>>> }
>>>>> +
>>>>> diff --git a/code/Surveyor/MatrixOwner.h b/code/Surveyor/MatrixOwner.h
>>>>> index ceb17e2..afa9278 100644
>>>>> --- a/code/Surveyor/MatrixOwner.h
>>>>> +++ b/code/Surveyor/MatrixOwner.h
>>>>> @@ -28,6 +28,7 @@
>>>>>
>>>>> #include <map>
>>>>> #include <iostream>
>>>>> +#include <sstream>
>>>>> using namespace std;
>>>>>
>>>>> class MatrixOwner : public Actor {
>>>>> @@ -62,7 +63,7 @@ public:
>>>>> PUSH_PAYLOAD,
>>>>> PUSH_PAYLOAD_OK,
>>>>> PUSH_PAYLOAD_END,
>>>>> - MATRIX_IS_READY,
>>>>> + GRAM_MATRIX_IS_READY,
>>>>> LAST_TAG
>>>>> };
>>>>>
>>>>> diff --git a/code/Surveyor/Mother.cpp b/code/Surveyor/Mother.cpp
>>>>> index 4d2ef9c..103a583 100644
>>>>> --- a/code/Surveyor/Mother.cpp
>>>>> +++ b/code/Surveyor/Mother.cpp
>>>>> @@ -27,6 +27,7 @@
>>>>> #include "GenomeGraphReader.h"
>>>>> #include "GenomeAssemblyReader.h"
>>>>> #include "MatrixOwner.h"
>>>>> +#include "KmersMatrixOwner.h"
>>>>>
>>>>> #include <RayPlatform/cryptography/crypto.h>
>>>>>
>>>>> @@ -39,11 +40,13 @@ using namespace std;
>>>>> #define INPUT_TYPE_GRAPH 0
>>>>> #define INPUT_TYPE_ASSEMBLY 1
>>>>>
>>>>> -
>>>>> Mother::Mother() {
>>>>>
>>>>> m_coalescenceManager = -1;
>>>>> m_matrixOwner = -1;
>>>>> + m_kmersMatrixOwner = -1;
>>>>> +
>>>>> + // m_matricesAreReady = true;
>>>> Remove this commented line.
>>>>
>>>>> m_parameters = NULL;
>>>>> m_bigMother = -1;
>>>>> @@ -91,7 +94,7 @@ void Mother::receive(Message & message) {
>>>>> notifyController();
>>>>> }
>>>>>
>>>>> - } else if(tag == MERGE) {
>>>>> + } else if(tag == MERGE_GRAM_MATRIX) {
>>>>>
>>>>> int matrixOwner = -1;
>>>>> memcpy(&matrixOwner, buffer, sizeof(matrixOwner));
>>>>> @@ -102,7 +105,7 @@ void Mother::receive(Message & message) {
>>>>> #endif
>>>>>
>>>>> Message theMessage;
>>>>> - theMessage.setTag(StoreKeeper::MERGE);
>>>>> + theMessage.setTag(StoreKeeper::MERGE_GRAM_MATRIX);
>>>>> theMessage.setBuffer(&matrixOwner);
>>>>> theMessage.setNumberOfBytes(sizeof(matrixOwner));
>>>>>
>>>>> @@ -111,10 +114,33 @@ void Mother::receive(Message & message) {
>>>>> send(destination, theMessage);
>>>>>
>>>>> Message response;
>>>>> - response.setTag(MERGE_OK);
>>>>> + response.setTag(MERGE_GRAM_MATRIX_OK);
>>>>> + send(source, response);
>>>>> +
>>>>> + } else if (tag == MERGE_KMERS_MATRIX) {
>>>>> +
>>>>> + int kmersMatrixOwner = -1;
>>>>> + memcpy(&kmersMatrixOwner, buffer, sizeof(kmersMatrixOwner));
>>>>> +
>>>>> +#ifdef CONFIG_ASSERT
>>>>> + assert(kmersMatrixOwner>= 0);
>>>>> + assert(m_storeKeepers.size() == 1);
>>>>> +#endif
>>>>> +
>>>>> + Message theMessage;
>>>>> + theMessage.setTag(StoreKeeper::MERGE_KMERS_MATRIX);
>>>> The name should be MERGE_KMER_MATRIX and not MERGE_KMERS_MATRIX.
>>>>
>>>>> + theMessage.setBuffer(&kmersMatrixOwner);
>>>>> + theMessage.setNumberOfBytes(sizeof(kmersMatrixOwner));
>>>>> +
>>>>> + int destination = m_storeKeepers[0];
>>>>> +
>>>>> + send(destination, theMessage);
>>>>> +
>>>>> + Message response;
>>>>> + response.setTag(MERGE_KMERS_MATRIX_OK);
>>>>> send(source, response);
>>>>>
>>>>> - } else if(tag == SHUTDOWN) {
>>>>> + } else if(tag == SHUTDOWN) {
>>>>>
>>>>> Message response;
>>>>> response.setTag(SHUTDOWN_OK);
>>>>> @@ -122,18 +148,16 @@ void Mother::receive(Message & message) {
>>>>>
>>>>> stop();
>>>>>
>>>>> - } else if(tag == StoreKeeper::MERGE_OK) {
>>>>> + } else if(tag == StoreKeeper::MERGE_GRAM_MATRIX_OK) {
>>>>>
>>>>> // TODO: the bug https://github.com/sebhtml/ray/issues/216
>>>>> // is caused by the fact that this message is not
>>>>> // received .
>>>>>
>>>>> - /*
>>>>> - Message newMessage;
>>>>> - newMessage.setTag(MERGE_OK);
>>>>> + // Message newMessage;
>>>>> + // newMessage.setTag(MERGE_OK);
>>>>>
>>>>> - send(m_bigMother, newMessage);
>>>>> - */
>>>>> + // send(m_bigMother, newMessage);
>>>> Remove these commented lines.
>>>>
>>>>> } else if(tag == FINISH_JOB) {
>>>>>
>>>>> @@ -153,6 +177,7 @@ void Mother::receive(Message & message) {
>>>>>
>>>>> sendToFirstMother(FLUSH_AGGREGATOR,
>>>>> FLUSH_AGGREGATOR_RETURN);
>>>>> }
>>>>> +
>>>>> } else if(tag == FLUSH_AGGREGATOR) {
>>>>>
>>>>> /*
>>>>> @@ -188,64 +213,52 @@ void Mother::receive(Message & message) {
>>>>> cout << "DEBUG sending FLUSH_AGGREGATOR_OK to
>>>>> m_bigMother" << endl;
>>>>> */
>>>>>
>>>>> - } else if(tag == MatrixOwner::MATRIX_IS_READY) {
>>>>> + } else if(tag == MatrixOwner::GRAM_MATRIX_IS_READY) {
>>>>> +
>>>>> + //TODO : check if all matrices are ready
>>>>> + if(m_matricesAreReady){
>>>>> + sendToFirstMother(SHUTDOWN, SHUTDOWN_OK);
>>>>> + }else {
>>>>> + cout << "GRAM_MATRIX_IS_READY" << endl;
>>>> When an actor speak, you must print its name too in stdout.
>>>> (with printName()).
>>>>
>>>>> + m_matricesAreReady = true;
>>>>> + }
>>>>>
>>>>> - sendToFirstMother(SHUTDOWN, SHUTDOWN_OK);
>>>>> + }
>>>>> + else if(tag == KmersMatrixOwner::KMERS_MATRIX_IS_READY) {
>>>>> +
>>>>> + cout << "KMERS_MATRIX_IS_READY" << endl;
>>>>> + if(m_matricesAreReady){
>>>>> + sendToFirstMother(SHUTDOWN, SHUTDOWN_OK);
>>>>> + }else {
>>>>> + cout << "KMERS_MATRIX_IS_READY" << endl;
>>>>> + m_matricesAreReady = true;
>>>> In one comment above, I saw that m_matricesAreReady = false
>>>> was commented. Check that out.
>>>>
>>>>> + }
>>>>>
>>>>> } else if(tag == FLUSH_AGGREGATOR_OK) {
>>>>>
>>>>> - /*
>>>>> printName();
>>>>> cout << "DEBUG received FLUSH_AGGREGATOR_OK" << endl;
>>>>> - */
>>>>>
>>>>> m_flushedMothers++;
>>>>>
>>>>> if(m_flushedMothers < getSize())
>>>>> return;
>>>>>
>>>>> - // spawn the MatrixOwner here !
>>>>> -
>>>>> - MatrixOwner * matrixOwner = new MatrixOwner();
>>>>> - spawn(matrixOwner);
>>>>> -
>>>>> - m_matrixOwner = matrixOwner->getName();
>>>>> -
>>>>> - printName();
>>>>> - cout << "Spawned MatrixOwner actor !" << endl;
>>>>> -
>>>>> - // tell the StoreKeeper actors to send their stuff to the
>>>>> - // MatrixOwner actor
>>>>> - // The Mother of Mother will wait for a signal from
>>>>> MatrixOwner
>>>>> -
>>>>> - Message greetingMessage;
>>>>> -
>>>>> - vector<string> * names = & m_sampleNames;
>>>>> -
>>>>> - char buffer[32];
>>>>> - int offset = 0;
>>>>> - memcpy(buffer + offset, &m_parameters,
>>>>> sizeof(m_parameters));
>>>>> - offset += sizeof(m_parameters);
>>>>> - memcpy(buffer + offset, &names, sizeof(names));
>>>>> - offset += sizeof(names);
>>>>> -
>>>>> - greetingMessage.setBuffer(&buffer);
>>>>> - greetingMessage.setNumberOfBytes(offset);
>>>>> -
>>>>> - greetingMessage.setTag(MatrixOwner::GREETINGS);
>>>>> - send(m_matrixOwner, greetingMessage);
>>>>> -
>>>>> - sendToFirstMother(MERGE, MERGE_OK);
>>>>> -
>>>>> + spawnMatrixOwner();
>>>> I like that. A method to spawn an actor. Good !
>>>>
>>>>> } else if(tag == m_responseTag) {
>>>>>
>>>>> -
>>>>> if(m_responseTag == SHUTDOWN_OK) {
>>>>>
>>>>> - } else if(m_responseTag == MERGE_OK) {
>>>>> -
>>>>> - } else if(m_responseTag == FLUSH_AGGREGATOR_RETURN) {
>>>>> + } else if(m_responseTag == MERGE_GRAM_MATRIX_OK) {
>>>>> + // All mothers merged their GRAM MATRIX
>>>>> + // Spawn KmersMatrixOwner to print
>>>>> + if(m_motherToKill < getSize() &&
>>>>> m_printKmersMatrix){
>>>>> + spawnKmersMatrixOwner();
>>>>> + }
>>>>> + } else if(m_responseTag == MERGE_KMERS_MATRIX_OK) {
>>>>> + }
>>>>> + else if(m_responseTag == FLUSH_AGGREGATOR_RETURN) {
>>>> Again, put closing brace on same line (} else if ( ...) {
>>>>
>>>>> /*
>>>>> printName();
>>>>> @@ -254,9 +267,8 @@ void Mother::receive(Message & message) {
>>>>> */
>>>>> }
>>>>>
>>>>> - // every mother was informed.
>>>>> + // every mother was not informed.
>>>> Good catch !
>>>>
>>>>> if(m_motherToKill>= getSize()) {
>>>>> -
>>>>> sendMessageWithReply(m_motherToKill, m_forwardTag);
>>>>> m_motherToKill--;
>>>>> }
>>>>> @@ -284,11 +296,15 @@ void Mother::sendMessageWithReply(int & actor, int
>>>>> tag) {
>>>>> Message message;
>>>>> message.setTag(tag);
>>>>>
>>>>> - if(tag == MERGE) {
>>>>> + if(tag == MERGE_GRAM_MATRIX) {
>>>>> message.setBuffer(&m_matrixOwner);
>>>>> message.setNumberOfBytes(sizeof(m_matrixOwner));
>>>>> -
>>>>> - } else if(tag == FLUSH_AGGREGATOR) {
>>>>> + }
>>>>> + else if(tag == MERGE_KMERS_MATRIX) {
>>>>> + message.setBuffer(&m_kmersMatrixOwner);
>>>>> + message.setNumberOfBytes(sizeof(m_kmersMatrixOwner));
>>>>> + }
>>>>> + else if(tag == FLUSH_AGGREGATOR) {
>>>>>
>>>>> /*
>>>>> printName();
>>>>> @@ -328,6 +344,10 @@ void Mother::stop() {
>>>>> m_matrixOwner = -1;
>>>>> }
>>>>>
>>>>> + if(m_kmersMatrixOwner>= 0) {
>>>>> + send(m_kmersMatrixOwner, kill);
>>>>> + m_kmersMatrixOwner = -1;
>>>>> + }
>>>>>
>>>>> die();
>>>>>
>>>>> @@ -410,39 +430,44 @@ void Mother::startSurveyor() {
>>>>>
>>>>> bool isRoot = (getName() % getSize()) == 0;
>>>>>
>>>>> - //cout << "DEBUG startSurveyor isRoot" << isRoot << endl;
>>>>> -
>>>>> - // get a list of files.
>>>>> + // Set matricesAreReady to true in case user doesn't want
>>>>> + // to print out kmers matrix.
>>>>> + m_matricesAreReady = true;
>>>>>
>>>>> vector<string> * commands = m_parameters->getCommands();
>>>>>
>>>>> -
>>>>> for(int i = 0 ; i < (int) commands->size() ; ++i) {
>>>>>
>>>>> string & element = commands->at(i);
>>>>>
>>>>> - // DONE: Check bounds for file names
>>>>> + if (element != "-print-kmers-matrix") {
>>>> The name should be kmer-matrix, not kmers-matrix.
>>>>
>>>> It is like groceries store vs grocery store.
>>>>
>>>>> + // DONE: Check bounds for file names
>>>>>
>>>>> - map<string,int> fastTable;
>>>>> + map<string,int> fastTable;
>>>>>
>>>>> - fastTable["-read-sample-graph"] = INPUT_TYPE_GRAPH;
>>>>> - fastTable["-read-sample-assembly"] = INPUT_TYPE_ASSEMBLY;
>>>>> + fastTable["-read-sample-graph"] = INPUT_TYPE_GRAPH;
>>>>> + fastTable["-read-sample-assembly"] =
>>>>> INPUT_TYPE_ASSEMBLY;
>>>>>
>>>>> - // Unsupported option
>>>>> - if(fastTable.count(element) == 0 || i+2> (int)
>>>>> commands->size())
>>>>> - continue;
>>>>> + // Unsupported option
>>>>> + if(fastTable.count(element) == 0 || i+2> (int)
>>>>> commands->size())
>>>>> + continue;
>>>>>
>>>>> - string sampleName = commands->at(++i);
>>>>> - string fileName = commands->at(++i);
>>>>> + string sampleName = commands->at(++i);
>>>>> + string fileName = commands->at(++i);
>>>>>
>>>>> - m_sampleNames.push_back(sampleName);
>>>>> + m_sampleNames.push_back(sampleName);
>>>>>
>>>>> - // DONE implement this m_assemblyFileNames + type
>>>>> - m_inputFileNames.push_back(fileName);
>>>>> + // DONE implement this m_assemblyFileNames + type
>>>>> + m_inputFileNames.push_back(fileName);
>>>>>
>>>>> - int type = fastTable[element];
>>>>> + int type = fastTable[element];
>>>>>
>>>>> - m_sampleInputTypes.push_back(type);
>>>>> + m_sampleInputTypes.push_back(type);
>>>>> +
>>>>> + } else {
>>>>> + m_matricesAreReady = false;
>>>>> + m_printKmersMatrix = true;
>>>> Question: if m_printKmersMatrix is false, I suppose the code
>>>> follows the usual path of printing just one matrix, right ?
>>>>
>>>>> + }
>>>>>
>>>>> }
>>>>>
>>>>> @@ -468,6 +493,9 @@ void Mother::startSurveyor() {
>>>>>
>>>>> m_storeKeepers.push_back(actor->getName());
>>>>>
>>>>> + actor->setOutputKmersMatrixPath(m_parameters->getPrefix());
>>>> The path should be prefix/Surveyor/<whatever the kmer matrix's name is>
>>>>
>>>>> + actor->setSamplesSize(m_sampleNames.size());
>>>> sample size, not samples size.
>>>>
>>>>> +
>>>>> // tell the CoalescenceManager about the local StoreKeeper
>>>>> Message dummyMessage;
>>>>> int localStore = actor->getName();
>>>>> @@ -568,6 +596,80 @@ void Mother::spawnReader() {
>>>>> }
>>>>> }
>>>>>
>>>>> +
>>>>> +void Mother::spawnMatrixOwner() {
>>>>> +
>>>>> + // spawn the MatrixOwner here !
>>>>> + MatrixOwner * matrixOwner = new MatrixOwner();
>>>>> + spawn(matrixOwner);
>>>>> +
>>>>> + m_matrixOwner = matrixOwner->getName();
>>>>> +
>>>>> + printName();
>>>>> + cout << "Spawned MatrixOwner actor !" << m_matrixOwner << endl;
>>>>> +
>>>>> + // tell the StoreKeeper actors to send their stuff to the
>>>>> + // MatrixOwner actor
>>>>> + // The Mother of Mother will wait for a signal from MatrixOwner
>>>>> +
>>>>> + Message greetingMessage;
>>>>> +
>>>>> + vector<string> * names = & m_sampleNames;
>>>>> +
>>>>> + char buffer[32];
>>>>> + int offset = 0;
>>>>> + memcpy(buffer + offset, &m_parameters, sizeof(m_parameters));
>>>>> + offset += sizeof(m_parameters);
>>>>> + memcpy(buffer + offset, &names, sizeof(names));
>>>>> + offset += sizeof(names);
>>>>> +
>>>>> + greetingMessage.setBuffer(&buffer);
>>>>> + greetingMessage.setNumberOfBytes(offset);
>>>>> +
>>>>> + greetingMessage.setTag(MatrixOwner::GREETINGS);
>>>>> + send(m_matrixOwner, greetingMessage);
>>>>> +
>>>>> + sendToFirstMother(MERGE_GRAM_MATRIX, MERGE_GRAM_MATRIX_OK);
>>>>> +}
>>>>> +
>>>>> +void Mother::spawnKmersMatrixOwner() {
>>>>> +
>>>>> + // spawn the MatrixOwner here !
>>>>> + KmersMatrixOwner * kmersMatrixOwner = new KmersMatrixOwner();
>>>>> + spawn(kmersMatrixOwner);
>>>>> +
>>>>> + m_kmersMatrixOwner = kmersMatrixOwner->getName();
>>>>> +
>>>>> + printName();
>>>>> + cout << "Spawned KmersMatrixOwner actor !" <<
>>>>> m_kmersMatrixOwner << endl;
>>>>> +
>>>>> + // tell the StoreKeeper actors to send their stuff to the
>>>>> + // KmersMatrixOwner actor
>>>>> + // The Mother of Mother will wait for a signal from MatrixOwner
>>>>> +
>>>>> + Message greetingMessage;
>>>>> +
>>>>> + vector<string> * names = & m_sampleNames;
>>>>> +
>>>>> + char buffer[32];
>>>>> + int offset = 0;
>>>>> + memcpy(buffer + offset, &m_parameters, sizeof(m_parameters));
>>>>> + offset += sizeof(m_parameters);
>>>>> + memcpy(buffer + offset, &names, sizeof(names));
>>>>> + offset += sizeof(names);
>>>>> +
>>>>> + greetingMessage.setBuffer(&buffer);
>>>>> + greetingMessage.setNumberOfBytes(offset);
>>>>> +
>>>>> + greetingMessage.setTag(KmersMatrixOwner::GREETINGS);
>>>>> + send(m_kmersMatrixOwner, greetingMessage);
>>>>> +
>>>>> + sendToFirstMother(MERGE_KMERS_MATRIX, MERGE_KMERS_MATRIX_OK);
>>>>> +
>>>>> +}
>>>>> +
>>>>> +
>>>>> void Mother::setParameters(Parameters * parameters) {
>>>>> m_parameters = parameters;
>>>>> }
>>>>> +
>>>>> diff --git a/code/Surveyor/Mother.h b/code/Surveyor/Mother.h
>>>>> index 092920f..9774c4b 100644
>>>>> --- a/code/Surveyor/Mother.h
>>>>> +++ b/code/Surveyor/Mother.h
>>>>> @@ -28,6 +28,7 @@
>>>>>
>>>>> #include <vector>
>>>>> #include <string>
>>>>> +#include <iostream>
>>>>> using namespace std;
>>>>>
>>>>> /**
>>>>> @@ -55,9 +56,12 @@ class Mother: public Actor {
>>>>> private:
>>>>>
>>>>> int m_matrixOwner;
>>>>> + int m_kmersMatrixOwner;
>>>>>
>>>>> int m_flushedMothers;
>>>>> int m_finishedMothers;
>>>>> + bool m_matricesAreReady;
>>>>> + bool m_printKmersMatrix;
>>>>>
>>>>> Parameters * m_parameters;
>>>>>
>>>>> @@ -93,6 +97,13 @@ private:
>>>>> */
>>>>> void sendToFirstMother(int forwardTag, int responseTag);
>>>>>
>>>>> + /* int m_kmersMatrixBlocNumber; */
>>>>> + void printLocalKmersMatrix(string & kmer, string &
>>>>> samples_kmers, bool force);
>>>>> + void createKmersMatrixOutputFile();
>>>>> +
>>>>> + void spawnMatrixOwner();
>>>>> + void spawnKmersMatrixOwner();
>>>>> +
>>>> That's a good design -- private methods for private uses.
>>>>
>>>>> public:
>>>>>
>>>>> Mother();
>>>>> @@ -109,8 +120,10 @@ public:
>>>>> FLUSH_AGGREGATOR,
>>>>> FLUSH_AGGREGATOR_OK,
>>>>> FLUSH_AGGREGATOR_RETURN,
>>>>> - MERGE,
>>>>> - MERGE_OK,
>>>>> + MERGE_GRAM_MATRIX,
>>>>> + MERGE_GRAM_MATRIX_OK,
>>>>> + MERGE_KMERS_MATRIX,
>>>>> + MERGE_KMERS_MATRIX_OK,
>>>> kmer matrix, not kmers matrix.
>>>>
>>>>> LAST_TAG,
>>>>> };
>>>>>
>>>>> diff --git a/code/Surveyor/StoreKeeper.cpp b/code/Surveyor/StoreKeeper.cpp
>>>>> index 84eef34..492208c 100644
>>>>> --- a/code/Surveyor/StoreKeeper.cpp
>>>>> +++ b/code/Surveyor/StoreKeeper.cpp
>>>>> @@ -22,10 +22,16 @@
>>>>> #include "StoreKeeper.h"
>>>>> #include "CoalescenceManager.h"
>>>>> #include "MatrixOwner.h"
>>>>> +#include "KmersMatrixOwner.h"
>>>>>
>>>>> #include <code/VerticesExtractor/Vertex.h>
>>>>> +#include <RayPlatform/structures/MyHashTableIterator.h>
>>>>> +#include <RayPlatform/core/OperatingSystem.h>
>>>>>
>>>>> #include <iostream>
>>>>> +#include <sstream>
>>>>> +#include <iomanip>
>>>>> +#include <fstream>
>>>>> using namespace std;
>>>>>
>>>>> #include <string.h>
>>>>> @@ -83,15 +89,21 @@ void StoreKeeper::receive(Message & message) {
>>>>>
>>>>> die();
>>>>>
>>>>> - } else if(tag == MERGE) {
>>>>> + } else if(tag == MERGE_GRAM_MATRIX) {
>>>>>
>>>>>
>>>>> - printName();
>>>>> - cout << "DEBUG at MERGE message reception ";
>>>>> - cout << "(StoreKeeper) received " << m_receivedObjects
>>>>> << " objects in total";
>>>>> - cout << " with " << m_receivedPushes << " push
>>>>> operations" << endl;
>>>>> + // printName();
>>>>> + // cout << "DEBUG at MERGE_GRAM_MATRIX message reception ";
>>>>> + // cout << "(StoreKeeper) received " <<
>>>>> m_receivedObjects << " objects in total";
>>>>> + // cout << " with " << m_receivedPushes << " push
>>>> You can remove commented lines.
>>>>
>>>>> operations" << endl;
>>>>> computeLocalGramMatrix();
>>>>>
>>>>> +
>>>>> + // TODEL Print matrix bloc
>>>>> + // m_kmersMatrixBlocNumber = 0;
>>>>> + // printLocalKmersMatrix();
>>>>> +
>>>> You can remove commented lines.
>>>>
>>>>> +
>>>>> m_mother = source;
>>>>>
>>>>> memcpy(&m_matrixOwner, buffer, sizeof(m_matrixOwner));
>>>>> @@ -108,19 +120,32 @@ void StoreKeeper::receive(Message & message) {
>>>>> m_iterator2 = m_iterator1->second.begin();
>>>>> }
>>>>>
>>>>> - /*
>>>>> - printName();
>>>>> - cout << "DEBUG printLocalGramMatrix before first
>>>>> sendMatrixCell" << endl;
>>>>> - printLocalGramMatrix();
>>>>> - */
>>>>> -
>>>>> + // printName();
>>>>> + // cout << "DEBUG printLocalGramMatrix before first
>>>>> sendMatrixCell" << endl;
>>>>> + // printLocalGramMatrix();
>>>> You can remove commented lines.
>>>>
>>>>> sendMatrixCell();
>>>>>
>>>>> } else if(tag == MatrixOwner::PUSH_PAYLOAD_OK) {
>>>>> -
>>>>> sendMatrixCell();
>>>>> + } else if(tag == MERGE_KMERS_MATRIX) {
>>>>> + // cout << "DEBUG at MERGE_GRAM_MATRIX message reception ";
>>>>> + // cout << "(StoreKeeper) received " <<
>>>>> m_receivedObjects << " objects in total";
>>>>> + // cout << " with " << m_receivedPushes << " push
>>>>> operations" << endl;
>>>> You can remove commented lines.
>>>>
>>>> Otherwise, add a "#ifdef DEBUG_SOMETHING_SOMETHING / #endif around that lines".
>>>>
>>>>
>>>>> +
>>>>> + m_mother = source;
>>>>>
>>>>> - } else if(tag == CoalescenceManager::SET_KMER_LENGTH) {
>>>>> + memcpy(&m_kmersMatrixOwner, buffer,
>>>>> sizeof(m_kmersMatrixOwner));
>>>>> +
>>>>> + m_hashTableIterator.constructor(&m_hashTable);
>>>>> +
>>>>> + sendKmersSamples();
>>>>> + }
>>>>> + else if (tag == KmersMatrixOwner::PUSH_KMER_SAMPLES_END) {
>>>>> + }
>>>>> + else if(tag == KmersMatrixOwner::PUSH_KMER_SAMPLES_OK) {
>>>>> + sendKmersSamples();
>>>>> + }
>>>>> + else if(tag == CoalescenceManager::SET_KMER_LENGTH) {
>>>>>
>>>>> int kmerLength = 0;
>>>>> int position = 0;
>>>>> @@ -181,8 +206,6 @@ void StoreKeeper::sendMatrixCell() {
>>>>> message.setNumberOfBytes(offset);
>>>>> message.setTag(MatrixOwner::PUSH_PAYLOAD);
>>>>>
>>>>> - //cout << " DEBUG send PUSH_PAYLOAD to " <<
>>>>> m_matrixOwner << endl;
>>>>> -
>>>>> send(m_matrixOwner, message);
>>>>>
>>>>> m_iterator2++;
>>>>> @@ -207,10 +230,7 @@ void StoreKeeper::sendMatrixCell() {
>>>>> // free memory.
>>>>> m_localGramMatrix.clear();
>>>>>
>>>>> - /*
>>>>> printName();
>>>>> - cout << "DEBUG send PUSH_PAYLOAD_END to " << m_matrixOwner << endl;
>>>>> - */
>>>>>
>>>>> Message response;
>>>>> response.setTag(MatrixOwner::PUSH_PAYLOAD_END);
>>>>> @@ -236,6 +256,7 @@ void StoreKeeper::configureHashTable() {
>>>>> );
>>>>>
>>>>> m_configured = true;
>>>>> +
>>>>> }
>>>>>
>>>>> void StoreKeeper::printColorReport() {
>>>>> @@ -375,6 +396,7 @@ void StoreKeeper::computeLocalGramMatrix() {
>>>>> //printLocalGramMatrix();
>>>>> }
>>>>>
>>>>> +
>>>>> void StoreKeeper::printLocalGramMatrix() {
>>>>>
>>>>> printName();
>>>>> @@ -623,3 +645,73 @@ void StoreKeeper::storeData(Vertex & vertex, int &
>>>>> sample) {
>>>>>
>>>>> */
>>>>> }
>>>>> +
>>>>> +
>>>>> +void StoreKeeper::setSamplesSize(int sampleSize) {
>>>>> + m_sampleSize = sampleSize;
>>>>> +}
>>>>> +
>>>>> +void StoreKeeper::setOutputKmersMatrixPath(string pathPrefix) {
>>>>> + // m_outputKmersMatrixPath = pathPrefix;
>>>>> + // m_outputKmersMatrixPath += "/KmersMatrixDump/";
>>>>> + // createDirectory(m_outputKmersMatrixPath.c_str());
>>>> You can remove commented lines.
>>>>
>>>>
>>>> This file could be in prefix/Surveyor/<...>
>>>>
>>>>> +}
>>>>> +
>>>>> +
>>>>> +void StoreKeeper::sendKmersSamples() {
>>>>> +
>>>>> + char buffer[4000];
>>>> For portability, use MAXIMUM_MESSAGE_SIZE_IN_BYTES instead of 4000.
>>>>
>>>>> + int bytes = 0;
>>>>> +
>>>>> + ExperimentVertex * currentVertex = NULL;
>>>>> + VirtualKmerColorHandle currentVirtualColor = NULL_VIRTUAL_COLOR;
>>>>> +
>>>>> + vector<bool> samplesVector (m_sampleSize, false);
>>>>> +
>>>>> + if(m_hashTableIterator.hasNext()){
>>>>> +
>>>>> + // fill(samplesVector.begin(),samplesVector.end(),false);
>>>>> +
>>>> You can remove commented lines.
>>>>
>>>>> + currentVertex = m_hashTableIterator.next();
>>>>> + Kmer kmer = currentVertex->getKey();
>>>>> +
>>>>> + bytes += kmer.dump(buffer);
>>>>> +
>>>>> + currentVirtualColor = currentVertex->getVirtualColor();
>>>>> + set<PhysicalKmerColor> * samples =
>>>>> m_colorSet.getPhysicalColors(currentVirtualColor);
>>>>> +
>>>>> + for(set<PhysicalKmerColor>:: iterator sampleIterator =
>>>>> samples->begin();
>>>>> + sampleIterator != samples->end(); ++sampleIterator) {
>>>>> + PhysicalKmerColor value = *sampleIterator;
>>>>> + samplesVector[value] = true;
>>>>> + // cout << " " << value;
>>>>> + }
>>>>> +
>>>>> + for (std::vector<bool>::iterator it =
>>>>> samplesVector.begin();
>>>>> + it != samplesVector.end(); ++it) {
>>>>> + buffer[bytes] = *it;
>>>>> + bytes++;
>>>>> + }
>>>>> + // buffer[bytes] = '\0';
>>>> You can remove commented lines.
>>>>
>>>>> + }
>>>>> +
>>>>> +
>>>>> + Message message;
>>>>> + message.setNumberOfBytes(bytes);
>>>>> + message.setBuffer(buffer);
>>>>> +
>>>>> + // message.setTag(MatrixOwner::PUSH_KMERS_SAMPLES);
>>>>> + if(m_hashTableIterator.hasNext()){
>>>>> + message.setTag(KmersMatrixOwner::PUSH_KMER_SAMPLES);
>>>>> + }else{
>>>>> + message.setTag(KmersMatrixOwner::PUSH_KMER_SAMPLES_END);
>>>>> + }
>>>>> +
>>>>> + // Message response;
>>>>> + // response.setTag(MatrixOwner::PUSH_PAYLOAD_END);
>>>>> + // send(m_matrixOwner, response);
>>>>> +
>>>>> + send(m_kmersMatrixOwner, message);
>>>>> +
>>>>> +}
>>>>> +
>>>>> diff --git a/code/Surveyor/StoreKeeper.h b/code/Surveyor/StoreKeeper.h
>>>>> index e44cf98..36adf77 100644
>>>>> --- a/code/Surveyor/StoreKeeper.h
>>>>> +++ b/code/Surveyor/StoreKeeper.h
>>>>> @@ -34,6 +34,10 @@
>>>>>
>>>>> #include <RayPlatform/actors/Actor.h>
>>>>> #include <RayPlatform/structures/MyHashTable.h>
>>>>> +#include <RayPlatform/structures/MyHashTableIterator.h>
>>>>> +
>>>>> +#include <iostream>
>>>>> +#include <sstream>
>>>>>
>>>>> /**
>>>>> * Provides genomic storage.
>>>>> @@ -55,6 +59,7 @@ private:
>>>>>
>>>>> int m_mother;
>>>>> int m_matrixOwner;
>>>>> + int m_kmersMatrixOwner;
>>>>>
>>>>> bool m_configured;
>>>>>
>>>>> @@ -64,6 +69,8 @@ private:
>>>>> */
>>>>> MyHashTable<Kmer,ExperimentVertex> m_hashTable;
>>>>>
>>>>> + MyHashTableIterator<Kmer,ExperimentVertex> m_hashTableIterator;
>>>>> +
>>>>> int m_kmerLength;
>>>>> bool m_colorSpaceMode;
>>>>>
>>>>> @@ -79,6 +86,13 @@ private:
>>>>> void printLocalGramMatrix();
>>>>> void printColorReport();
>>>>>
>>>>> + /* ostringstream m_currentKmer; */
>>>>> + /* ostringstream m_currentSamplesKmers; */
>>>>> + int m_sampleSize;
>>>>> + string m_outputKmersMatrixPath;
>>>>> + void printLocalKmersMatrix(string & m_kmer, string &
>>>>> m_samplesKmers);
>>>>> + void sendKmersSamples();
>>>>> +
>>>>> void sendMatrixCell();
>>>>>
>>>>> public:
>>>>> @@ -86,14 +100,19 @@ public:
>>>>> StoreKeeper();
>>>>> ~StoreKeeper();
>>>>>
>>>>> + void setOutputKmersMatrixPath(string pathPrefix);
>>>>> + void setSamplesSize(int sampleSize);
>>>>> +
>>>>> void receive(Message & message);
>>>>>
>>>>> enum {
>>>>> FIRST_TAG = 10250,
>>>>> PUSH_SAMPLE_VERTEX,
>>>>> PUSH_SAMPLE_VERTEX_OK,
>>>>> - MERGE,
>>>>> - MERGE_OK,
>>>>> + MERGE_GRAM_MATRIX,
>>>>> + MERGE_GRAM_MATRIX_OK,
>>>>> + MERGE_KMERS_MATRIX,
>>>>> + MERGE_KMERS_MATRIX_OK,
>>>>> LAST_TAG
>>>>> };
>>>>> };
>>>> ------------------------------------------------------------------------------
>>>> Managing the Performance of Cloud-Based Applications
>>>> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
>>>> Read the Whitepaper.
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Denovoassembler-devel mailing list
>>>> Den...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/denovoassembler-devel
>>>
>