Re: [Denovoassembler-devel] git diff kmersmatrix branch

[Maxime D. <max...@gm...>]

Hi Seb,

You can find these modifications in the same branch as before :

https://github.com/Zorino/ray.git

patch-kmermatrix


On 03/07/2014 02:18 AM, Sébastien Boisvert wrote:
> Hi Maxime,
>
> I can't merge your code in master without making some changes first.
>
> As the maintainer, here are the changes that I will need to do to increase the plus-value of your work (on
> your next pull request, you can think about some of these points if you want):
>
> Keep in mind that Coding style is very important for readability.
> (you can check https://github.com/sebhtml/ray/blob/master/Documentation/CodingStyle.txt )
>
> Here are the changes that I will make tomorrow:
>
> - (C0) As discussed, we don't want to generate Surveyor/KmerMatrix.tsv by default because this is 
> quite large file. There needs to be an option.
>
> - (C1) Obviously, if you want people to use your Surveyor workflow, you need to add documentation.
> You just need to add some code in code/Mock/Parameters.cpp. After that you build Ray,
> then you write ./Ray -help> MANUAL_PAGE.txt

Yes there was an option needed to launch it so it wasn't by default but
I forgot to add it in the Parameter file (now -run-kmer-matrix).Done.

> - (C2) KmerMatrixOwner.h/cpp was done by you. So the copyright in the header belongs to you, not
> me.

done.

>
> - (C2) there is some red showing up in 'git diff --color', nothing important though
>     relevant commits ce6c272 & 4e4949
>
> - (C3) Copyright for code/Surveyor/SequenceKmerReader.cpp/.h should be in 2014

done.

> - (C4) In code/Surveyor/StoreKeeper.h you must use tabulations and no spaces for indentation.
> for example, the line with MERGE_KMER_MATRIX uses spaces in StoreKeeper.h

Yes, my indent-mode in emacs suddenly decided to put 8 spaces instead of
tabs I corrected this and re-indented it all.

> - (C5) code/Surveyor/KmerMatrixOwner.cpp was using spaces instead of tabulations.
>
> - (C6) the method name  KmerMatrixOwner::printLocalKmersMatrix is meaningless as it does not
> write a matrix, it writes a kmer.

now called dumpKmerMatrixBuffer since it is not writing a single kmer
but a couple of it with a utility function of yours called
flushFileOperationBuffer

> - (C7) you hard-coded the kmer value (31) !!!         m_kmerMatrix << kmer.idToWord(31,0);
> you must avoid that ! [KmerMatrixOwner.cpp]

big mistake here fixed it with m_parameters->getWordSize()

> - (C8) The FIRST_TAG for your reader is the same that is used by the graph reader. I think we said it 
> was OK as long as you added a comment about the reason.
> code/Surveyor/GenomeAssemblyReader.h:           FIRST_TAG = 10200,
> code/Surveyor/GenomeGraphReader.h:              FIRST_TAG = 10200,

added this comment :
        // Using the same tag as GenomeGraphReader
        // because we can mix an assembly reader with a graph reader

>
> Keep up the great work, but next time, I think you can make progress on respecting the coding style,
> among other things.

thanks, I understand the importance you give to coding style.

> I will do a bunch of commits in my branch patch-kmermatrix (fetched from
> remotes/zorino/patch-kmermatrix) to address the comments above.
>
>
> [boiseb01@ls30 ray]$ git diff master..remotes/zorino/patch-kmermatrix --stat
>  code/Surveyor/GenomeAssemblyReader.cpp |    3 +-
>  code/Surveyor/KmerMatrixOwner.cpp      |  157 ++++++++++++++++++++
>  code/Surveyor/KmerMatrixOwner.h        |   72 +++++++++
>  code/Surveyor/Makefile                 |    1 +
>  code/Surveyor/MatrixOwner.cpp          |   19 +--
>  code/Surveyor/MatrixOwner.h            |    3 +-
>  code/Surveyor/Mother.cpp               |  253 +++++++++++++++++++++----------
>  code/Surveyor/Mother.h                 |   17 ++-
>  code/Surveyor/SequenceKmerReader.cpp   |   53 ++++++-
>  code/Surveyor/SequenceKmerReader.h     |    2 +
>  code/Surveyor/StoreKeeper.cpp          |  117 +++++++++++----
>  code/Surveyor/StoreKeeper.h            |   20 +++-
>  12 files changed, 580 insertions(+), 137 deletions(-)
>
> Link: http://sourceforge.net/p/denovoassembler/mailman/denovoassembler-devel/thread/017C19ECDB98F64F99200D83876C44520111A7FC4FE7%40EXCH-MBX-B.ulaval.ca/#msg31999514
> Link: http://sourceforge.net/p/denovoassembler/mailman/denovoassembler-devel/thread/COL131-W760CDDC5071A415AB9C5EDAC870%40phx.gbl/#msg32015211
> Link: http://sourceforge.net/p/denovoassembler/mailman/denovoassembler-devel/thread/f2d8f53ef133555e36529047f4e95084%40boisvert.info/#msg31993547
> Link: http://sourceforge.net/p/denovoassembler/mailman/denovoassembler-devel/thread/5301F42B.6060904%40gmail.com/#msg31988453
>
> ----------------------------------------
>> Date: Wed, 5 Mar 2014 13:41:39 +0000
>> From: max...@gm...
>> To: se...@bo...
>> Subject: Re: [Denovoassembler-devel] git diff kmersmatrix branch
>>
>> Hi Sebastien,
>>
>> I have fix the issue when scaffolds would be given in entry instead of
>> contigs in the SequenceKmerReader class.
>>
>> I also made the edition according to this review.
>>
>> Please now pull from :
>>
>> https://github.com/Zorino/ray.git
>>
>> patch-kmermatrix
>>
>> Cheers,
>>
>> Maxime
>>
>>
>>
>> On 02/23/2014 12:16 PM, Sébastien Boisvert wrote:
>>> Hey Maxime,
>>>
>>> You did not provide KmersMatrixOwner.h, KmersMatrixOwner.cpp, and changes
>>> to the Surveyor Makefile.
>>>
>>>
>>> OTher comments are below.
>>>
>>> ----------------------------------------
>>>> Date: Sat, 22 Feb 2014 10:03:30 +0000
>>>> From: ma...@de...
>>>> To: se...@bo...
>>>> Subject: git diff kmersmatrix branch
>>>>
>>>> diff --git a/code/Surveyor/MatrixOwner.cpp b/code/Surveyor/MatrixOwner.cpp
>>>> index ffaae00..47cf84a 100644
>>>> --- a/code/Surveyor/MatrixOwner.cpp
>>>> +++ b/code/Surveyor/MatrixOwner.cpp
>>>> @@ -65,9 +65,12 @@ void MatrixOwner::receive(Message & message) {
>>>> assert(m_parameters != NULL);
>>>> assert(m_sampleNames != NULL);
>>>> #endif
>>>> -
>>>> m_mother = source;
>>>>
>>>> + //open the buffer of the file
>>>> + // createKmersMatrixOutputFile();
>>>> +
>>>> +
>>>> } else if(tag == PUSH_PAYLOAD) {
>>>>
>>>> SampleIdentifier sample1 = -1;
>>>> @@ -89,10 +92,10 @@ void MatrixOwner::receive(Message & message) {
>>>> assert(count>= 0);
>>>> #endif
>>>>
>>>> - /*
>>>> +
>>>> printName();
>>>> - cout << "DEBUG add " << sample1 << " " << sample2 << " "
>>>> << count << endl;
>>>> -*/
>>>> + // cout << "DEBUG add " << sample1 << " " << sample2 <<
>>> Commented lines should be removed.
>>>
>>>> " " << count << endl;
>>>> +
>>>> m_receivedPayloads ++;
>>>>
>>>> m_localGramMatrix[sample1][sample2] += count;
>>>> @@ -100,14 +103,14 @@ void MatrixOwner::receive(Message & message) {
>>>> Message response;
>>>> response.setTag(PUSH_PAYLOAD_OK);
>>>> send(source, response);
>>>> + }
>>>> + else if(tag == PUSH_PAYLOAD_END) {
>>> Use '} else if (' and not '}
>>> else if'
>>>
>>>
>>> This is the coding style of the project.
>>> see https://github.com/sebhtml/ray/blob/master/Documentation/CodingStyle.txt
>>>
>>> * Kernighan and Ritchie style, variant "The One True Brace Style" (1TBS)
>>> http://en.wikipedia.org/wiki/Indent_style#Variant:_1TBS
>>>
>>>
>>> (you used K&R Variant: Stroustrup).
>>>
>>>> - } else if(tag == PUSH_PAYLOAD_END) {
>>>> -
>>>> + cout << "PUSH_PAYLOAD_END" <<endl;
>>> Remove this debug message.
>>>
>>>> m_completedStoreActors++;
>>>>
>>>> if(m_completedStoreActors == getSize()) {
>>>>
>>>> -
>>>> printName();
>>>> cout << "MatrixOwner received " <<
>>>> m_receivedPayloads << " payloads" << endl;
>>>>
>>>> @@ -151,10 +154,9 @@ void MatrixOwner::receive(Message & message) {
>>>>
>>>>
>>>> // tell Mother that the matrix is ready now.
>>>> -
>>>> - Message coolMessage;
>>>> - coolMessage.setTag(MATRIX_IS_READY);
>>>> - send(m_mother, coolMessage);
>>>> + Message coolMessage;
>>>> + coolMessage.setTag(GRAM_MATRIX_IS_READY);
>>>> + send(m_mother, coolMessage);
>>>>
>>>>
>>>> // clear matrices
>>>> @@ -275,3 +277,4 @@ void
>>>> MatrixOwner::printLocalGramMatrixWithHash(ostream & stream, map<SampleIdent
>>>> stream << endl;
>>>> }
>>>> }
>>>> +
>>>> diff --git a/code/Surveyor/MatrixOwner.h b/code/Surveyor/MatrixOwner.h
>>>> index ceb17e2..afa9278 100644
>>>> --- a/code/Surveyor/MatrixOwner.h
>>>> +++ b/code/Surveyor/MatrixOwner.h
>>>> @@ -28,6 +28,7 @@
>>>>
>>>> #include <map>
>>>> #include <iostream>
>>>> +#include <sstream>
>>>> using namespace std;
>>>>
>>>> class MatrixOwner : public Actor {
>>>> @@ -62,7 +63,7 @@ public:
>>>> PUSH_PAYLOAD,
>>>> PUSH_PAYLOAD_OK,
>>>> PUSH_PAYLOAD_END,
>>>> - MATRIX_IS_READY,
>>>> + GRAM_MATRIX_IS_READY,
>>>> LAST_TAG
>>>> };
>>>>
>>>> diff --git a/code/Surveyor/Mother.cpp b/code/Surveyor/Mother.cpp
>>>> index 4d2ef9c..103a583 100644
>>>> --- a/code/Surveyor/Mother.cpp
>>>> +++ b/code/Surveyor/Mother.cpp
>>>> @@ -27,6 +27,7 @@
>>>> #include "GenomeGraphReader.h"
>>>> #include "GenomeAssemblyReader.h"
>>>> #include "MatrixOwner.h"
>>>> +#include "KmersMatrixOwner.h"
>>>>
>>>> #include <RayPlatform/cryptography/crypto.h>
>>>>
>>>> @@ -39,11 +40,13 @@ using namespace std;
>>>> #define INPUT_TYPE_GRAPH 0
>>>> #define INPUT_TYPE_ASSEMBLY 1
>>>>
>>>> -
>>>> Mother::Mother() {
>>>>
>>>> m_coalescenceManager = -1;
>>>> m_matrixOwner = -1;
>>>> + m_kmersMatrixOwner = -1;
>>>> +
>>>> + // m_matricesAreReady = true;
>>> Remove this commented line.
>>>
>>>> m_parameters = NULL;
>>>> m_bigMother = -1;
>>>> @@ -91,7 +94,7 @@ void Mother::receive(Message & message) {
>>>> notifyController();
>>>> }
>>>>
>>>> - } else if(tag == MERGE) {
>>>> + } else if(tag == MERGE_GRAM_MATRIX) {
>>>>
>>>> int matrixOwner = -1;
>>>> memcpy(&matrixOwner, buffer, sizeof(matrixOwner));
>>>> @@ -102,7 +105,7 @@ void Mother::receive(Message & message) {
>>>> #endif
>>>>
>>>> Message theMessage;
>>>> - theMessage.setTag(StoreKeeper::MERGE);
>>>> + theMessage.setTag(StoreKeeper::MERGE_GRAM_MATRIX);
>>>> theMessage.setBuffer(&matrixOwner);
>>>> theMessage.setNumberOfBytes(sizeof(matrixOwner));
>>>>
>>>> @@ -111,10 +114,33 @@ void Mother::receive(Message & message) {
>>>> send(destination, theMessage);
>>>>
>>>> Message response;
>>>> - response.setTag(MERGE_OK);
>>>> + response.setTag(MERGE_GRAM_MATRIX_OK);
>>>> + send(source, response);
>>>> +
>>>> + } else if (tag == MERGE_KMERS_MATRIX) {
>>>> +
>>>> + int kmersMatrixOwner = -1;
>>>> + memcpy(&kmersMatrixOwner, buffer, sizeof(kmersMatrixOwner));
>>>> +
>>>> +#ifdef CONFIG_ASSERT
>>>> + assert(kmersMatrixOwner>= 0);
>>>> + assert(m_storeKeepers.size() == 1);
>>>> +#endif
>>>> +
>>>> + Message theMessage;
>>>> + theMessage.setTag(StoreKeeper::MERGE_KMERS_MATRIX);
>>> The name should be MERGE_KMER_MATRIX and not MERGE_KMERS_MATRIX.
>>>
>>>> + theMessage.setBuffer(&kmersMatrixOwner);
>>>> + theMessage.setNumberOfBytes(sizeof(kmersMatrixOwner));
>>>> +
>>>> + int destination = m_storeKeepers[0];
>>>> +
>>>> + send(destination, theMessage);
>>>> +
>>>> + Message response;
>>>> + response.setTag(MERGE_KMERS_MATRIX_OK);
>>>> send(source, response);
>>>>
>>>> - } else if(tag == SHUTDOWN) {
>>>> + } else if(tag == SHUTDOWN) {
>>>>
>>>> Message response;
>>>> response.setTag(SHUTDOWN_OK);
>>>> @@ -122,18 +148,16 @@ void Mother::receive(Message & message) {
>>>>
>>>> stop();
>>>>
>>>> - } else if(tag == StoreKeeper::MERGE_OK) {
>>>> + } else if(tag == StoreKeeper::MERGE_GRAM_MATRIX_OK) {
>>>>
>>>> // TODO: the bug https://github.com/sebhtml/ray/issues/216
>>>> // is caused by the fact that this message is not
>>>> // received .
>>>>
>>>> - /*
>>>> - Message newMessage;
>>>> - newMessage.setTag(MERGE_OK);
>>>> + // Message newMessage;
>>>> + // newMessage.setTag(MERGE_OK);
>>>>
>>>> - send(m_bigMother, newMessage);
>>>> - */
>>>> + // send(m_bigMother, newMessage);
>>> Remove these commented lines.
>>>
>>>> } else if(tag == FINISH_JOB) {
>>>>
>>>> @@ -153,6 +177,7 @@ void Mother::receive(Message & message) {
>>>>
>>>> sendToFirstMother(FLUSH_AGGREGATOR,
>>>> FLUSH_AGGREGATOR_RETURN);
>>>> }
>>>> +
>>>> } else if(tag == FLUSH_AGGREGATOR) {
>>>>
>>>> /*
>>>> @@ -188,64 +213,52 @@ void Mother::receive(Message & message) {
>>>> cout << "DEBUG sending FLUSH_AGGREGATOR_OK to
>>>> m_bigMother" << endl;
>>>> */
>>>>
>>>> - } else if(tag == MatrixOwner::MATRIX_IS_READY) {
>>>> + } else if(tag == MatrixOwner::GRAM_MATRIX_IS_READY) {
>>>> +
>>>> + //TODO : check if all matrices are ready
>>>> + if(m_matricesAreReady){
>>>> + sendToFirstMother(SHUTDOWN, SHUTDOWN_OK);
>>>> + }else {
>>>> + cout << "GRAM_MATRIX_IS_READY" << endl;
>>> When an actor speak, you must print its name too in stdout.
>>> (with printName()).
>>>
>>>> + m_matricesAreReady = true;
>>>> + }
>>>>
>>>> - sendToFirstMother(SHUTDOWN, SHUTDOWN_OK);
>>>> + }
>>>> + else if(tag == KmersMatrixOwner::KMERS_MATRIX_IS_READY) {
>>>> +
>>>> + cout << "KMERS_MATRIX_IS_READY" << endl;
>>>> + if(m_matricesAreReady){
>>>> + sendToFirstMother(SHUTDOWN, SHUTDOWN_OK);
>>>> + }else {
>>>> + cout << "KMERS_MATRIX_IS_READY" << endl;
>>>> + m_matricesAreReady = true;
>>> In one comment above, I saw that m_matricesAreReady = false
>>> was commented. Check that out.
>>>
>>>> + }
>>>>
>>>> } else if(tag == FLUSH_AGGREGATOR_OK) {
>>>>
>>>> - /*
>>>> printName();
>>>> cout << "DEBUG received FLUSH_AGGREGATOR_OK" << endl;
>>>> - */
>>>>
>>>> m_flushedMothers++;
>>>>
>>>> if(m_flushedMothers < getSize())
>>>> return;
>>>>
>>>> - // spawn the MatrixOwner here !
>>>> -
>>>> - MatrixOwner * matrixOwner = new MatrixOwner();
>>>> - spawn(matrixOwner);
>>>> -
>>>> - m_matrixOwner = matrixOwner->getName();
>>>> -
>>>> - printName();
>>>> - cout << "Spawned MatrixOwner actor !" << endl;
>>>> -
>>>> - // tell the StoreKeeper actors to send their stuff to the
>>>> - // MatrixOwner actor
>>>> - // The Mother of Mother will wait for a signal from
>>>> MatrixOwner
>>>> -
>>>> - Message greetingMessage;
>>>> -
>>>> - vector<string> * names = & m_sampleNames;
>>>> -
>>>> - char buffer[32];
>>>> - int offset = 0;
>>>> - memcpy(buffer + offset, &m_parameters,
>>>> sizeof(m_parameters));
>>>> - offset += sizeof(m_parameters);
>>>> - memcpy(buffer + offset, &names, sizeof(names));
>>>> - offset += sizeof(names);
>>>> -
>>>> - greetingMessage.setBuffer(&buffer);
>>>> - greetingMessage.setNumberOfBytes(offset);
>>>> -
>>>> - greetingMessage.setTag(MatrixOwner::GREETINGS);
>>>> - send(m_matrixOwner, greetingMessage);
>>>> -
>>>> - sendToFirstMother(MERGE, MERGE_OK);
>>>> -
>>>> + spawnMatrixOwner();
>>> I like that. A method to spawn an actor. Good !
>>>
>>>> } else if(tag == m_responseTag) {
>>>>
>>>> -
>>>> if(m_responseTag == SHUTDOWN_OK) {
>>>>
>>>> - } else if(m_responseTag == MERGE_OK) {
>>>> -
>>>> - } else if(m_responseTag == FLUSH_AGGREGATOR_RETURN) {
>>>> + } else if(m_responseTag == MERGE_GRAM_MATRIX_OK) {
>>>> + // All mothers merged their GRAM MATRIX
>>>> + // Spawn KmersMatrixOwner to print
>>>> + if(m_motherToKill < getSize() &&
>>>> m_printKmersMatrix){
>>>> + spawnKmersMatrixOwner();
>>>> + }
>>>> + } else if(m_responseTag == MERGE_KMERS_MATRIX_OK) {
>>>> + }
>>>> + else if(m_responseTag == FLUSH_AGGREGATOR_RETURN) {
>>> Again, put closing brace on same line (} else if ( ...) {
>>>
>>>> /*
>>>> printName();
>>>> @@ -254,9 +267,8 @@ void Mother::receive(Message & message) {
>>>> */
>>>> }
>>>>
>>>> - // every mother was informed.
>>>> + // every mother was not informed.
>>> Good catch !
>>>
>>>> if(m_motherToKill>= getSize()) {
>>>> -
>>>> sendMessageWithReply(m_motherToKill, m_forwardTag);
>>>> m_motherToKill--;
>>>> }
>>>> @@ -284,11 +296,15 @@ void Mother::sendMessageWithReply(int & actor, int
>>>> tag) {
>>>> Message message;
>>>> message.setTag(tag);
>>>>
>>>> - if(tag == MERGE) {
>>>> + if(tag == MERGE_GRAM_MATRIX) {
>>>> message.setBuffer(&m_matrixOwner);
>>>> message.setNumberOfBytes(sizeof(m_matrixOwner));
>>>> -
>>>> - } else if(tag == FLUSH_AGGREGATOR) {
>>>> + }
>>>> + else if(tag == MERGE_KMERS_MATRIX) {
>>>> + message.setBuffer(&m_kmersMatrixOwner);
>>>> + message.setNumberOfBytes(sizeof(m_kmersMatrixOwner));
>>>> + }
>>>> + else if(tag == FLUSH_AGGREGATOR) {
>>>>
>>>> /*
>>>> printName();
>>>> @@ -328,6 +344,10 @@ void Mother::stop() {
>>>> m_matrixOwner = -1;
>>>> }
>>>>
>>>> + if(m_kmersMatrixOwner>= 0) {
>>>> + send(m_kmersMatrixOwner, kill);
>>>> + m_kmersMatrixOwner = -1;
>>>> + }
>>>>
>>>> die();
>>>>
>>>> @@ -410,39 +430,44 @@ void Mother::startSurveyor() {
>>>>
>>>> bool isRoot = (getName() % getSize()) == 0;
>>>>
>>>> - //cout << "DEBUG startSurveyor isRoot" << isRoot << endl;
>>>> -
>>>> - // get a list of files.
>>>> + // Set matricesAreReady to true in case user doesn't want
>>>> + // to print out kmers matrix.
>>>> + m_matricesAreReady = true;
>>>>
>>>> vector<string> * commands = m_parameters->getCommands();
>>>>
>>>> -
>>>> for(int i = 0 ; i < (int) commands->size() ; ++i) {
>>>>
>>>> string & element = commands->at(i);
>>>>
>>>> - // DONE: Check bounds for file names
>>>> + if (element != "-print-kmers-matrix") {
>>> The name should be kmer-matrix, not kmers-matrix.
>>>
>>> It is like groceries store vs grocery store.
>>>
>>>> + // DONE: Check bounds for file names
>>>>
>>>> - map<string,int> fastTable;
>>>> + map<string,int> fastTable;
>>>>
>>>> - fastTable["-read-sample-graph"] = INPUT_TYPE_GRAPH;
>>>> - fastTable["-read-sample-assembly"] = INPUT_TYPE_ASSEMBLY;
>>>> + fastTable["-read-sample-graph"] = INPUT_TYPE_GRAPH;
>>>> + fastTable["-read-sample-assembly"] =
>>>> INPUT_TYPE_ASSEMBLY;
>>>>
>>>> - // Unsupported option
>>>> - if(fastTable.count(element) == 0 || i+2> (int)
>>>> commands->size())
>>>> - continue;
>>>> + // Unsupported option
>>>> + if(fastTable.count(element) == 0 || i+2> (int)
>>>> commands->size())
>>>> + continue;
>>>>
>>>> - string sampleName = commands->at(++i);
>>>> - string fileName = commands->at(++i);
>>>> + string sampleName = commands->at(++i);
>>>> + string fileName = commands->at(++i);
>>>>
>>>> - m_sampleNames.push_back(sampleName);
>>>> + m_sampleNames.push_back(sampleName);
>>>>
>>>> - // DONE implement this m_assemblyFileNames + type
>>>> - m_inputFileNames.push_back(fileName);
>>>> + // DONE implement this m_assemblyFileNames + type
>>>> + m_inputFileNames.push_back(fileName);
>>>>
>>>> - int type = fastTable[element];
>>>> + int type = fastTable[element];
>>>>
>>>> - m_sampleInputTypes.push_back(type);
>>>> + m_sampleInputTypes.push_back(type);
>>>> +
>>>> + } else {
>>>> + m_matricesAreReady = false;
>>>> + m_printKmersMatrix = true;
>>> Question: if m_printKmersMatrix is false, I suppose the code
>>> follows the usual path of printing just one matrix, right ?
>>>
>>>> + }
>>>>
>>>> }
>>>>
>>>> @@ -468,6 +493,9 @@ void Mother::startSurveyor() {
>>>>
>>>> m_storeKeepers.push_back(actor->getName());
>>>>
>>>> + actor->setOutputKmersMatrixPath(m_parameters->getPrefix());
>>> The path should be prefix/Surveyor/<whatever the kmer matrix's name is>
>>>
>>>> + actor->setSamplesSize(m_sampleNames.size());
>>> sample size, not samples size.
>>>
>>>> +
>>>> // tell the CoalescenceManager about the local StoreKeeper
>>>> Message dummyMessage;
>>>> int localStore = actor->getName();
>>>> @@ -568,6 +596,80 @@ void Mother::spawnReader() {
>>>> }
>>>> }
>>>>
>>>> +
>>>> +void Mother::spawnMatrixOwner() {
>>>> +
>>>> + // spawn the MatrixOwner here !
>>>> + MatrixOwner * matrixOwner = new MatrixOwner();
>>>> + spawn(matrixOwner);
>>>> +
>>>> + m_matrixOwner = matrixOwner->getName();
>>>> +
>>>> + printName();
>>>> + cout << "Spawned MatrixOwner actor !" << m_matrixOwner << endl;
>>>> +
>>>> + // tell the StoreKeeper actors to send their stuff to the
>>>> + // MatrixOwner actor
>>>> + // The Mother of Mother will wait for a signal from MatrixOwner
>>>> +
>>>> + Message greetingMessage;
>>>> +
>>>> + vector<string> * names = & m_sampleNames;
>>>> +
>>>> + char buffer[32];
>>>> + int offset = 0;
>>>> + memcpy(buffer + offset, &m_parameters, sizeof(m_parameters));
>>>> + offset += sizeof(m_parameters);
>>>> + memcpy(buffer + offset, &names, sizeof(names));
>>>> + offset += sizeof(names);
>>>> +
>>>> + greetingMessage.setBuffer(&buffer);
>>>> + greetingMessage.setNumberOfBytes(offset);
>>>> +
>>>> + greetingMessage.setTag(MatrixOwner::GREETINGS);
>>>> + send(m_matrixOwner, greetingMessage);
>>>> +
>>>> + sendToFirstMother(MERGE_GRAM_MATRIX, MERGE_GRAM_MATRIX_OK);
>>>> +}
>>>> +
>>>> +void Mother::spawnKmersMatrixOwner() {
>>>> +
>>>> + // spawn the MatrixOwner here !
>>>> + KmersMatrixOwner * kmersMatrixOwner = new KmersMatrixOwner();
>>>> + spawn(kmersMatrixOwner);
>>>> +
>>>> + m_kmersMatrixOwner = kmersMatrixOwner->getName();
>>>> +
>>>> + printName();
>>>> + cout << "Spawned KmersMatrixOwner actor !" <<
>>>> m_kmersMatrixOwner << endl;
>>>> +
>>>> + // tell the StoreKeeper actors to send their stuff to the
>>>> + // KmersMatrixOwner actor
>>>> + // The Mother of Mother will wait for a signal from MatrixOwner
>>>> +
>>>> + Message greetingMessage;
>>>> +
>>>> + vector<string> * names = & m_sampleNames;
>>>> +
>>>> + char buffer[32];
>>>> + int offset = 0;
>>>> + memcpy(buffer + offset, &m_parameters, sizeof(m_parameters));
>>>> + offset += sizeof(m_parameters);
>>>> + memcpy(buffer + offset, &names, sizeof(names));
>>>> + offset += sizeof(names);
>>>> +
>>>> + greetingMessage.setBuffer(&buffer);
>>>> + greetingMessage.setNumberOfBytes(offset);
>>>> +
>>>> + greetingMessage.setTag(KmersMatrixOwner::GREETINGS);
>>>> + send(m_kmersMatrixOwner, greetingMessage);
>>>> +
>>>> + sendToFirstMother(MERGE_KMERS_MATRIX, MERGE_KMERS_MATRIX_OK);
>>>> +
>>>> +}
>>>> +
>>>> +
>>>> void Mother::setParameters(Parameters * parameters) {
>>>> m_parameters = parameters;
>>>> }
>>>> +
>>>> diff --git a/code/Surveyor/Mother.h b/code/Surveyor/Mother.h
>>>> index 092920f..9774c4b 100644
>>>> --- a/code/Surveyor/Mother.h
>>>> +++ b/code/Surveyor/Mother.h
>>>> @@ -28,6 +28,7 @@
>>>>
>>>> #include <vector>
>>>> #include <string>
>>>> +#include <iostream>
>>>> using namespace std;
>>>>
>>>> /**
>>>> @@ -55,9 +56,12 @@ class Mother: public Actor {
>>>> private:
>>>>
>>>> int m_matrixOwner;
>>>> + int m_kmersMatrixOwner;
>>>>
>>>> int m_flushedMothers;
>>>> int m_finishedMothers;
>>>> + bool m_matricesAreReady;
>>>> + bool m_printKmersMatrix;
>>>>
>>>> Parameters * m_parameters;
>>>>
>>>> @@ -93,6 +97,13 @@ private:
>>>> */
>>>> void sendToFirstMother(int forwardTag, int responseTag);
>>>>
>>>> + /* int m_kmersMatrixBlocNumber; */
>>>> + void printLocalKmersMatrix(string & kmer, string &
>>>> samples_kmers, bool force);
>>>> + void createKmersMatrixOutputFile();
>>>> +
>>>> + void spawnMatrixOwner();
>>>> + void spawnKmersMatrixOwner();
>>>> +
>>> That's a good design -- private methods for private uses.
>>>
>>>> public:
>>>>
>>>> Mother();
>>>> @@ -109,8 +120,10 @@ public:
>>>> FLUSH_AGGREGATOR,
>>>> FLUSH_AGGREGATOR_OK,
>>>> FLUSH_AGGREGATOR_RETURN,
>>>> - MERGE,
>>>> - MERGE_OK,
>>>> + MERGE_GRAM_MATRIX,
>>>> + MERGE_GRAM_MATRIX_OK,
>>>> + MERGE_KMERS_MATRIX,
>>>> + MERGE_KMERS_MATRIX_OK,
>>> kmer matrix, not kmers matrix.
>>>
>>>> LAST_TAG,
>>>> };
>>>>
>>>> diff --git a/code/Surveyor/StoreKeeper.cpp b/code/Surveyor/StoreKeeper.cpp
>>>> index 84eef34..492208c 100644
>>>> --- a/code/Surveyor/StoreKeeper.cpp
>>>> +++ b/code/Surveyor/StoreKeeper.cpp
>>>> @@ -22,10 +22,16 @@
>>>> #include "StoreKeeper.h"
>>>> #include "CoalescenceManager.h"
>>>> #include "MatrixOwner.h"
>>>> +#include "KmersMatrixOwner.h"
>>>>
>>>> #include <code/VerticesExtractor/Vertex.h>
>>>> +#include <RayPlatform/structures/MyHashTableIterator.h>
>>>> +#include <RayPlatform/core/OperatingSystem.h>
>>>>
>>>> #include <iostream>
>>>> +#include <sstream>
>>>> +#include <iomanip>
>>>> +#include <fstream>
>>>> using namespace std;
>>>>
>>>> #include <string.h>
>>>> @@ -83,15 +89,21 @@ void StoreKeeper::receive(Message & message) {
>>>>
>>>> die();
>>>>
>>>> - } else if(tag == MERGE) {
>>>> + } else if(tag == MERGE_GRAM_MATRIX) {
>>>>
>>>>
>>>> - printName();
>>>> - cout << "DEBUG at MERGE message reception ";
>>>> - cout << "(StoreKeeper) received " << m_receivedObjects
>>>> << " objects in total";
>>>> - cout << " with " << m_receivedPushes << " push
>>>> operations" << endl;
>>>> + // printName();
>>>> + // cout << "DEBUG at MERGE_GRAM_MATRIX message reception ";
>>>> + // cout << "(StoreKeeper) received " <<
>>>> m_receivedObjects << " objects in total";
>>>> + // cout << " with " << m_receivedPushes << " push
>>> You can remove commented lines.
>>>
>>>> operations" << endl;
>>>> computeLocalGramMatrix();
>>>>
>>>> +
>>>> + // TODEL Print matrix bloc
>>>> + // m_kmersMatrixBlocNumber = 0;
>>>> + // printLocalKmersMatrix();
>>>> +
>>> You can remove commented lines.
>>>
>>>> +
>>>> m_mother = source;
>>>>
>>>> memcpy(&m_matrixOwner, buffer, sizeof(m_matrixOwner));
>>>> @@ -108,19 +120,32 @@ void StoreKeeper::receive(Message & message) {
>>>> m_iterator2 = m_iterator1->second.begin();
>>>> }
>>>>
>>>> - /*
>>>> - printName();
>>>> - cout << "DEBUG printLocalGramMatrix before first
>>>> sendMatrixCell" << endl;
>>>> - printLocalGramMatrix();
>>>> - */
>>>> -
>>>> + // printName();
>>>> + // cout << "DEBUG printLocalGramMatrix before first
>>>> sendMatrixCell" << endl;
>>>> + // printLocalGramMatrix();
>>> You can remove commented lines.
>>>
>>>> sendMatrixCell();
>>>>
>>>> } else if(tag == MatrixOwner::PUSH_PAYLOAD_OK) {
>>>> -
>>>> sendMatrixCell();
>>>> + } else if(tag == MERGE_KMERS_MATRIX) {
>>>> + // cout << "DEBUG at MERGE_GRAM_MATRIX message reception ";
>>>> + // cout << "(StoreKeeper) received " <<
>>>> m_receivedObjects << " objects in total";
>>>> + // cout << " with " << m_receivedPushes << " push
>>>> operations" << endl;
>>> You can remove commented lines.
>>>
>>> Otherwise, add a "#ifdef DEBUG_SOMETHING_SOMETHING / #endif around that lines".
>>>
>>>
>>>> +
>>>> + m_mother = source;
>>>>
>>>> - } else if(tag == CoalescenceManager::SET_KMER_LENGTH) {
>>>> + memcpy(&m_kmersMatrixOwner, buffer,
>>>> sizeof(m_kmersMatrixOwner));
>>>> +
>>>> + m_hashTableIterator.constructor(&m_hashTable);
>>>> +
>>>> + sendKmersSamples();
>>>> + }
>>>> + else if (tag == KmersMatrixOwner::PUSH_KMER_SAMPLES_END) {
>>>> + }
>>>> + else if(tag == KmersMatrixOwner::PUSH_KMER_SAMPLES_OK) {
>>>> + sendKmersSamples();
>>>> + }
>>>> + else if(tag == CoalescenceManager::SET_KMER_LENGTH) {
>>>>
>>>> int kmerLength = 0;
>>>> int position = 0;
>>>> @@ -181,8 +206,6 @@ void StoreKeeper::sendMatrixCell() {
>>>> message.setNumberOfBytes(offset);
>>>> message.setTag(MatrixOwner::PUSH_PAYLOAD);
>>>>
>>>> - //cout << " DEBUG send PUSH_PAYLOAD to " <<
>>>> m_matrixOwner << endl;
>>>> -
>>>> send(m_matrixOwner, message);
>>>>
>>>> m_iterator2++;
>>>> @@ -207,10 +230,7 @@ void StoreKeeper::sendMatrixCell() {
>>>> // free memory.
>>>> m_localGramMatrix.clear();
>>>>
>>>> - /*
>>>> printName();
>>>> - cout << "DEBUG send PUSH_PAYLOAD_END to " << m_matrixOwner << endl;
>>>> - */
>>>>
>>>> Message response;
>>>> response.setTag(MatrixOwner::PUSH_PAYLOAD_END);
>>>> @@ -236,6 +256,7 @@ void StoreKeeper::configureHashTable() {
>>>> );
>>>>
>>>> m_configured = true;
>>>> +
>>>> }
>>>>
>>>> void StoreKeeper::printColorReport() {
>>>> @@ -375,6 +396,7 @@ void StoreKeeper::computeLocalGramMatrix() {
>>>> //printLocalGramMatrix();
>>>> }
>>>>
>>>> +
>>>> void StoreKeeper::printLocalGramMatrix() {
>>>>
>>>> printName();
>>>> @@ -623,3 +645,73 @@ void StoreKeeper::storeData(Vertex & vertex, int &
>>>> sample) {
>>>>
>>>> */
>>>> }
>>>> +
>>>> +
>>>> +void StoreKeeper::setSamplesSize(int sampleSize) {
>>>> + m_sampleSize = sampleSize;
>>>> +}
>>>> +
>>>> +void StoreKeeper::setOutputKmersMatrixPath(string pathPrefix) {
>>>> + // m_outputKmersMatrixPath = pathPrefix;
>>>> + // m_outputKmersMatrixPath += "/KmersMatrixDump/";
>>>> + // createDirectory(m_outputKmersMatrixPath.c_str());
>>> You can remove commented lines.
>>>
>>>
>>> This file could be in prefix/Surveyor/<...>
>>>
>>>> +}
>>>> +
>>>> +
>>>> +void StoreKeeper::sendKmersSamples() {
>>>> +
>>>> + char buffer[4000];
>>> For portability, use MAXIMUM_MESSAGE_SIZE_IN_BYTES instead of 4000.
>>>
>>>> + int bytes = 0;
>>>> +
>>>> + ExperimentVertex * currentVertex = NULL;
>>>> + VirtualKmerColorHandle currentVirtualColor = NULL_VIRTUAL_COLOR;
>>>> +
>>>> + vector<bool> samplesVector (m_sampleSize, false);
>>>> +
>>>> + if(m_hashTableIterator.hasNext()){
>>>> +
>>>> + // fill(samplesVector.begin(),samplesVector.end(),false);
>>>> +
>>> You can remove commented lines.
>>>
>>>> + currentVertex = m_hashTableIterator.next();
>>>> + Kmer kmer = currentVertex->getKey();
>>>> +
>>>> + bytes += kmer.dump(buffer);
>>>> +
>>>> + currentVirtualColor = currentVertex->getVirtualColor();
>>>> + set<PhysicalKmerColor> * samples =
>>>> m_colorSet.getPhysicalColors(currentVirtualColor);
>>>> +
>>>> + for(set<PhysicalKmerColor>:: iterator sampleIterator =
>>>> samples->begin();
>>>> + sampleIterator != samples->end(); ++sampleIterator) {
>>>> + PhysicalKmerColor value = *sampleIterator;
>>>> + samplesVector[value] = true;
>>>> + // cout << " " << value;
>>>> + }
>>>> +
>>>> + for (std::vector<bool>::iterator it =
>>>> samplesVector.begin();
>>>> + it != samplesVector.end(); ++it) {
>>>> + buffer[bytes] = *it;
>>>> + bytes++;
>>>> + }
>>>> + // buffer[bytes] = '\0';
>>> You can remove commented lines.
>>>
>>>> + }
>>>> +
>>>> +
>>>> + Message message;
>>>> + message.setNumberOfBytes(bytes);
>>>> + message.setBuffer(buffer);
>>>> +
>>>> + // message.setTag(MatrixOwner::PUSH_KMERS_SAMPLES);
>>>> + if(m_hashTableIterator.hasNext()){
>>>> + message.setTag(KmersMatrixOwner::PUSH_KMER_SAMPLES);
>>>> + }else{
>>>> + message.setTag(KmersMatrixOwner::PUSH_KMER_SAMPLES_END);
>>>> + }
>>>> +
>>>> + // Message response;
>>>> + // response.setTag(MatrixOwner::PUSH_PAYLOAD_END);
>>>> + // send(m_matrixOwner, response);
>>>> +
>>>> + send(m_kmersMatrixOwner, message);
>>>> +
>>>> +}
>>>> +
>>>> diff --git a/code/Surveyor/StoreKeeper.h b/code/Surveyor/StoreKeeper.h
>>>> index e44cf98..36adf77 100644
>>>> --- a/code/Surveyor/StoreKeeper.h
>>>> +++ b/code/Surveyor/StoreKeeper.h
>>>> @@ -34,6 +34,10 @@
>>>>
>>>> #include <RayPlatform/actors/Actor.h>
>>>> #include <RayPlatform/structures/MyHashTable.h>
>>>> +#include <RayPlatform/structures/MyHashTableIterator.h>
>>>> +
>>>> +#include <iostream>
>>>> +#include <sstream>
>>>>
>>>> /**
>>>> * Provides genomic storage.
>>>> @@ -55,6 +59,7 @@ private:
>>>>
>>>> int m_mother;
>>>> int m_matrixOwner;
>>>> + int m_kmersMatrixOwner;
>>>>
>>>> bool m_configured;
>>>>
>>>> @@ -64,6 +69,8 @@ private:
>>>> */
>>>> MyHashTable<Kmer,ExperimentVertex> m_hashTable;
>>>>
>>>> + MyHashTableIterator<Kmer,ExperimentVertex> m_hashTableIterator;
>>>> +
>>>> int m_kmerLength;
>>>> bool m_colorSpaceMode;
>>>>
>>>> @@ -79,6 +86,13 @@ private:
>>>> void printLocalGramMatrix();
>>>> void printColorReport();
>>>>
>>>> + /* ostringstream m_currentKmer; */
>>>> + /* ostringstream m_currentSamplesKmers; */
>>>> + int m_sampleSize;
>>>> + string m_outputKmersMatrixPath;
>>>> + void printLocalKmersMatrix(string & m_kmer, string &
>>>> m_samplesKmers);
>>>> + void sendKmersSamples();
>>>> +
>>>> void sendMatrixCell();
>>>>
>>>> public:
>>>> @@ -86,14 +100,19 @@ public:
>>>> StoreKeeper();
>>>> ~StoreKeeper();
>>>>
>>>> + void setOutputKmersMatrixPath(string pathPrefix);
>>>> + void setSamplesSize(int sampleSize);
>>>> +
>>>> void receive(Message & message);
>>>>
>>>> enum {
>>>> FIRST_TAG = 10250,
>>>> PUSH_SAMPLE_VERTEX,
>>>> PUSH_SAMPLE_VERTEX_OK,
>>>> - MERGE,
>>>> - MERGE_OK,
>>>> + MERGE_GRAM_MATRIX,
>>>> + MERGE_GRAM_MATRIX_OK,
>>>> + MERGE_KMERS_MATRIX,
>>>> + MERGE_KMERS_MATRIX_OK,
>>>> LAST_TAG
>>>> };
>>>> };
>>> ------------------------------------------------------------------------------
>>> Managing the Performance of Cloud-Based Applications
>>> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
>>> Read the Whitepaper.
>>> http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Denovoassembler-devel mailing list
>>> Den...@li...
>>> https://lists.sourceforge.net/lists/listinfo/denovoassembler-devel
>>