#15 Transform not producing output chunks

[Justin M Wozniak]

My transform output QMap is always empty. How do I set up Transform.m_OutputData so that I actually get some chunks?

[Guy Jennings]

Have you pushed your current code?

[Justin M Wozniak]

I was running from Swift. I just pushed a change to cctwtcltest-3.tcl that will allow you to transform multiple chunks and observe the "products" number, which is the size of the output QMap. products is stuck at 0.

[Guy Jennings]

Lots of nexus crap reappeared in your latest push!

You need to give the CctwDataChunk constructor a pointer to a CctwChunkedData object when you create it - that is where it gets its coordinate system from. If you do this you
should (hopefully) not need to call setChunkSize and the chunk size should be correctly calculated for the edge cases.

[Justin M Wozniak]

I think the issue is in the output chunk. cctwtransformer.cpp:198 uses 'if (m_OutputData->containsPixel(pixels))' - I think this prevents anything from going into the output QMap. This prevents the call to m_OutputData->chunkContaining(), so no output chunk is registered.

Should I rearrange this code or is there a planned way to handle this?

[Guy Jennings]

It means that you have to set up the output dataset dimensions as well, but this has probably happened automatically:

try:

cctw outputData.dimensions
cctw outputData.chunkSize
cctw outputData.chunkCount

to see if they have been set

[Justin M Wozniak]

Yes, in the transform method I report:

qDebug("input chunk size: %s\n", qPrintable(chSize.toString()));
qDebug("output chunk size: %s\n", qPrintable(m_OutputData->chunkSize().toString()));

resulting in:

Transforming chunk data: 208
input chunk size: [128,128,128]
output chunk size: [128,128,128]

However, some of these pixels are supposed to land in other output chunks, right?

[Guy Jennings]

You also need to look at the chunkStart() property, this is the coordinates
of the start element of the chunk within the wider dataset coordinates

[Guy Jennings]

chunkStart is set by the CctwDataChunk constructor, based on the coordinates of
the data set that you pass in to it.

[Justin M Wozniak]

I set up a ChunkedData with a good chunkStart() and still get no outputs. I am going to sketch an alternate transform method for discussion.

[Guy Jennings]

Let me see if I can rewrite it

[Justin M Wozniak]

I have a sketch- don't do anything drastic...

[Guy Jennings]

OK, but I can see several problems with the code so far...

[Justin M Wozniak]

Ok.

[Justin M Wozniak]

I just pushed the sketch of static methods in [5b42bc] . The key thing to review for discussion is CctwTransformer::transformChunkData2() .

[Guy Jennings]

I'm profoundly skeptical about the way you are going here...

Remember that the input and output dataset dimensions and chunk sizes need not be the same, so how do the static methods tell if they should use the parameters for the input or output datasets...

Also, not all the chunks of a given dataset will be the same size - you are not allowing for this either...

[Guy Jennings]

I think a better approach is to make more direct use of the algorithms already in place for the desktop version, but with slight tweaks to the ownership of data items. I have a revised version of Cctwtcl_Transform_Cmd that just takes a single chunk number and reads that chunk from the input data, transforms it and returns a list of partial output chunks to be merged. In this formulation we don't need Cctwtcl_Input_Cmd at all. It appears to be working for me but I don't want to push it to the repo with all your static stuff in it - I'd suggest reverting that commit if you are amenable

[Justin M Wozniak]

Ok I will revert first thing tomorrow.

[Justin M Wozniak]

Ok I removed the static methods.

[Justin M Wozniak]

I put some notes about this at: https://sourceforge.net/p/cctw/wiki/SwiftFrameworkRequirements

[Guy Jennings]

Pushed a revised version of the tcl commands - this now assumes that the file name and dataset name are setup beforehand for each instance, e.g. by reading a startup script, then you call cctw_transform <chunk_id> which reads one chunk of input data, transforms it and returns a list of transformed chunks, each represented by a list of 4 numbers -
{ chunkid dataptr weightptr length } for each chunk.</chunk_id>

I need to change the code so that the chunks are allocated with malloc so that swift can free them and I need to change the code to reduce the storage size and initialization times

[Guy Jennings]

I've pushed changes so that the chunk buffers are allocated with calloc, so that swift can free them, and I have optimized the memory allocation strategy so that chunk objects are only actually allocated when they are used - this should minimize memory usage.

Revision [afd62c]

[Justin M Wozniak]

This looks good...