Transforms cause NPE.

[Shen Flik]

Hi folks,

It is my first time to use transforms and I think there may be something wrong with my configuration which eventually leads below NullPointerException.
    2014-11-19 15:59:11,572 ERROR [store-001] [DataExtractorService] [store-001-push-1] Failed to extract batch 000-8
    java.lang.NullPointerException
        at org.jumpmind.symmetric.io.data.writer.AbstractProtocolDataWriter.start(AbstractProtocolDataWriter.java:127)
        at org.jumpmind.symmetric.io.data.writer.NestedDataWriter.start(NestedDataWriter.java:60)
        at org.jumpmind.symmetric.model.ProcessInfoDataWriter.start(ProcessInfoDataWriter.java:58)
        at org.jumpmind.symmetric.io.data.writer.TransformWriter.write(TransformWriter.java:215)
        at org.jumpmind.symmetric.io.data.DataProcessor.forEachDataInTable(DataProcessor.java:199)
        at org.jumpmind.symmetric.io.data.DataProcessor.forEachTableInBatch(DataProcessor.java:169)
        at org.jumpmind.symmetric.io.data.DataProcessor.process(DataProcessor.java:115)
        at org.jumpmind.symmetric.service.impl.DataExtractorService.extractOutgoingBatch(DataExtractorService.java:707)
        at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:513)
        at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:422)
        at org.jumpmind.symmetric.service.impl.PushService.pushToNode(PushService.java:193)
        at org.jumpmind.symmetric.service.impl.PushService.execute(PushService.java:157)
        at org.jumpmind.symmetric.service.impl.NodeCommunicationService$2.run(NodeCommunicationService.java:307)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

My configuration is built base on sample as following which try to update item's price when any item is returned.

***# transform for sale return line item***
***INSERT INTO sym_transform_table ***
***(transform_id,transform_point,source_node_group_id,target_node_group_id,source_table_name,target_table_name,update_first,delete_action,column_policy,last_update_time,create_time)***
***VALUES('sale_return_line_item', 'EXTRACT', 'store', 'corp', 'sale_return_line_item', 'sale_return_line_item', 1, 'DEL_ROW', 'IMPLIED', current_timestamp, current_timestamp);***

***# work through the table sale_return_line_item***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,source_column_name,pk,last_update_time,create_time)***
***VALUES('sale_return_line_item', 'I', 'tran_id', 'tran_id', 1, current_timestamp, current_timestamp);***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,source_column_name,pk,last_update_time,create_time)***
***VALUES('sale_return_line_item', 'I', 'item_id', 'item_id', 1, current_timestamp, current_timestamp);***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,source_column_name,pk,last_update_time,create_time)***
***VALUES('sale_return_line_item', 'U', 'tran_id', 'tran_id', 1, current_timestamp, current_timestamp);***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,source_column_name,pk,last_update_time,create_time)***
***VALUES('sale_return_line_item', 'U', 'item_id', 'item_id', 1, current_timestamp, current_timestamp);***

***# transform for item_selling_price***
***INSERT INTO sym_transform_table ***
***(transform_id,transform_point,source_node_group_id,target_node_group_id,source_table_name,target_table_name,update_first,delete_action,column_policy,last_update_time,create_time)***
***VALUES('sale_return_line_item_price', 'EXTRACT', 'store', 'corp', 'sale_return_line_item', 'item_selling_price', 1, 'NONE', 'SPECIFIED', current_timestamp, current_timestamp);***

***# update price according to return line item***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,source_column_name,pk,last_update_time,create_time)***
***VALUES('sale_return_line_item_price', 'I', 'item_id', 'item_id', 1, current_timestamp, current_timestamp);***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,pk,transform_type,transform_expression,last_update_time,create_time)***
***VALUES('sale_return_line_item_price', 'I', 'store_id', 1, 'lookup', 'SELECT store_id FROM sale_transaction WHERE tran_id = :TRAN_ID', current_timestamp, current_timestamp);***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,source_column_name,pk,last_update_time,create_time)***
***VALUES('sale_return_line_item_price', 'I', 'price', 'price', 0, current_timestamp, current_timestamp);***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,source_column_name,pk,last_update_time,create_time)***
***VALUES('sale_return_line_item_price', 'U', 'item_id', 'item_id', 1, current_timestamp, current_timestamp);***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,pk,transform_type,transform_expression,last_update_time,create_time)***
***VALUES('sale_return_line_item_price', 'U', 'store_id', 1, 'lookup', 'SELECT store_id FROM sale_transaction WHERE tran_id = :TRAN_ID', current_timestamp, current_timestamp);***
***INSERT INTO sym_transform_column ***
***(transform_id,include_on,target_column_name,source_column_name,pk,last_update_time,create_time)***
***VALUES('sale_return_line_item_price', 'U', 'price', 'price', 0, current_timestamp, current_timestamp);***

What is wrong? How can I configure the transforms?

Thanks in advance.

Flik

[Shen Flik]

One thing to add on.
The version of SymmetricDS I used is 3.6.11.
Without adding transforms the example works fine for me.

[Shen Flik]

Hi all,

The problem is solved and the root cause is separate sets of transform columns are configured for update and inserting.
Originally I just wanna update price when a record is inserted or updated and do nothing when a record is deleted. So I create columns set for insert and update separately. Although the configuration is accepted, symmetricds doesn't work as expected. When I insert a row into table sale_return_line_item with values as [tran_id=1000, item_id=110000055, price=0.55], following code block inside retrieve method of class TransformedData returns empty map.

protected Map<String, String> retrieve(Map<TransformColumn.IncludeOnType, LinkedHashMap<String, String>> source) {

    Map<String, String> list = new LinkedHashMap<String, String>(source == null ? 0
            : source.size());
    if (source != null) {
        LinkedHashMap<String, String> values = source.get(IncludeOnType.ALL);
        if (values != null) {
            list.putAll(values);
        }

        IncludeOnType type = IncludeOnType.DELETE;
        if (targetDmlType == DataEventType.UPDATE && sourceDmlType != DataEventType.DELETE) {
            type = IncludeOnType.UPDATE;
        } else if (targetDmlType == DataEventType.INSERT) {
            type = IncludeOnType.INSERT;
        }

        values = source.get(type);
        if (values != null) {
            list.putAll(values);
        }
    }
    return list;
}

The variable type eventually is set to UPDATE but the source only contains INSERT entry like INSERT={tran_id=1000, item_id=110000055, price=0.55}. Then it results a NULL table instance passed to start method of class AbstractProtocolDataWriter.

After I merged two transform columns set by setting column [include_on] of table transform_column to '*' as its default and setting column [delete_action] of table transform_table to 'NONE', symmetricds will update price properly and do nothing when a row is deleted.

Regards,
Flik

[Shen Flik]

I also try to set transform column set for only UPDATE or INSERT, it works fine also.

I am curious about how to omit symmetricds taking any action only for INSERT or UPDATE.
But maybe it is not a problem since they are very rare cases.

Flik

[Chris Henson]

Can't you set the delete_action to NONE?

I think the issue might be caused because you have update_first checked.

Could this be related?

http://www.symmetricds.org/issues/view.php?id=2053

[Shen Flik]

Hi Chris,

I know I could set include_on column to "*" and delete_action column to "NONE" for a transform only working for update and insert operation. It is exactly what I do for my configuration.
I'm just curious about how to allow transform only working for update/delete or insert/delete scenarios.

Update_first didn't cause issue you mentioned but it really cause another problem in following commnet when record didn't exist in target database.
https://sourceforge.net/p/symmetricds/discussion/739236/thread/b3d1d9d8/#4ad4

regards,
Flik