I'm developing Replication in Adempiere and i have one question which i think that concern all Developers.
Status of Replication now is:
1) Exoprt of Records into custom defined xml format is considered done.
2) Import of XML Formats is considered working. Here need to finish 1-2 things, but this will be done in next 1-2 days.
But i have one issue.
First let's describe how things work at the moment.
I have developed ModelValidator class which is responsible to generate proper XML Format when record is modified. All this means that class is invoked when type is TYPE_AFTER_CHANGE
if (po.get_TableName().equals(MBPartner.Table_Name) && type == TYPE_AFTER_CHANGE)
{
MBPartner bPartner = (MBPartner)po; // Cast is not necessary. Thisi s just an example.
exportRecord(bPartner);
}
Exportes works as expected.
I have developed a process which imports records and here comes the problem. Importer till now works as expected and manages to parse XML file and update record, but when Importer save the record Exporter is invoked and we get a cycle...
So i think that we need new TYPES in ModelValidator:
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_DELETE_REPLICATION
Also new method in PO:
saveReplication() or save(boolean isReplication)
This method will be something like:
public boolean save (boolean isReplication)
{
setReplication(isReplication));
return save();
} // save
and i have to modify save() method to use new TYPE.
Please let me know your opinion.
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry i made small mistake.
Please read this:
So i think that we need new TYPES in ModelValidator:
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_DELETE_REPLICATION
as:
So i think that we need new TYPES in ModelValidator:
TYPE_AFTER_NEW_REPLICATION
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_DELETE_REPLICATION
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I don't know how the design of replication goes, it's not clear in wiki page.
I see this is a big need on Adempiere, and is a big opportunity to make Adempiere more connectable with other systems.
Currently I see several correlated sponsored developments:
* Replication
* Integration with OSCommerce
* EDI
* Web Service
And some other "projects" talking about similar things like:
* BI
* Jitterbit
And looks like they're being developed in an unconnected way. It's a shame not exploit this situation to manage a "central" design - that must be the a good case for the PMC.
In past I made a lot of research and development in replication and synchronization between two systems, studying top EAI's in the market, and top heterogeneous db replicators.
Talking from my lack of knowledge of current Adempiere replication design, and knowing that I must reread a lot of EAI documentation, I think the current approach is too focused in developing quickly something functional and not in the core architecture needed.
I think replication must be seen in layers:
1 - a flexible way to define the tables involved in replication (for subscriber and for publisher)
2 - define the transformations needed (in subscriber or in publisher)
3 - saving the proper information in a messaging table or system
4 - a way to manage subscribers and publishers
5 - a delivering system for publishers - managing different ways of delivering
6 - a recollection system for subscribers - managing different ways of delivering
Mostly every system involved in a replication environment is a subscriber and a publisher at the same time.
But there are LOTS of needs in replication, i.e.:
- a one-centralized-read-write system replicating to one or many read-only systems (like BI)
- a one-centralized-read-write system replicating both ways with other(s) read-write systems (like POS)
- a one-centralized-read-only receiving information from many decentralized-read-write systems
etc
-----------------------
Again from my lack of knowledge of the design here, exporting/importing to XML is just a little part of the whole architecture needed.
-----------------------
NOTE ASIDE:
Every time I write about any topic, looks like a stopper because I mostly think big.
I know the open source rule of 10% seed, I just think if the 10% seed has a good architecture the 90% evolution is easier.
That's a main issue I think for the PMC, the ability to ensure a wide view of problems, and advising on solutions with a better architecture - it doesn't matter if just the 10% of the solution is done initially.
Regards,
Carlos Ruiz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
thank's for the answer.
Design is similar to EDI.
We will have Export/Import Layer and transport layer.
Administrator of system can define Export format which will be used as Import Format.
Administrator will define Transport mechanism.
For EDI i will implement JMS and Local file initially during development and test period.
If you look at EDI you will see the same structure.
But EDI can't be reused as by design(not design in Adempiere) in EDI we can have duplicate Element names which can't be resolved to which Column to be mapped. Also EDI has three lier structure : EID Format -> EDI Segment -> EDI Field. While in Replication and XML we have two tier system: Export Format(Table) -> Export Format Line(Column) which can point to other Export Format. So we can get unlimited tree structure of XML.
I also think that in Export Format we can have some kind of transformation, but it will be used when exporting to other systems, for Replication i think that it will not be used in most of the cases.
I also would like to point that currently i'm in the Prove Of Concept phase which by plan is two weeks. For completel work i think that i will need 1.5-2 months more.
I would like to make one general note. All available tools are not appropriate as they are not designed to work with AD. Power of Adempiere is AD from where developer can understand structure of model.
My question was more concrete.
I MUST modify core in order to fix issues which i sticked now. So i have to do it, but i would like to have Adempiere developer's opinion regarding the change. Probably there is better way or it could be done by simple way.
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
>I would like to make one general note. All available tools are not appropriate as they are not designed to work with >AD. Power of Adempiere is AD from where developer can understand structure of model.
>My question was more concrete.
One more note regarding previous note.
Every system must have some Export/Import functionality and after that external tools can be used. I do not say that i will not use any external tool or that external tools are not good, but prior using such tool Adempiere must have some way to export/import data. I do not think that using external tool to make integration at DB level is good think for one ERP system, that's why i concentrated to develop Export/Import first. I choose to make export/import in XML as it is most used and can be transformed.
AS you know integration can be done at several levels:
1) DB level. Application A reads/writes data directly to DB of Application B. I do not think that it is good in ERP system.
2) Integration at Application level. I think that here there were several types. But i do not remember them now and i do not have time to search them, sorry :(
Please feel free to add them and comment them!
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If possible, it would be better if we can extend and leverage what is already in Adempiere instead of building everything from the ground up. It should be possible to extend the changelog (ad_changelog) feature in Adempiere and use that to generate the replication message. Also, it might be good to add versioning support for all table which will help in implementing conflict resolution.
For the cycle issue, if the changelog mechanism is use, we can add a way to turn on or off the change log management during the save operation.
Finally, I don't think using ModelValidator to generate the xml export file is a good approach. Replication should be a background process that have little performance impact on normal transaction and the frequency of replication should also be user configurable. I think we should have a background process that read the changelog and replication configuration to generate the replication message ( xml or otherwise ) and send that out.
Regards,
Low
P.S It will encourage more collaboration and input if the spec/design is publish fully in the wiki.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I modidifed wiki page and added some information.
I see 3 ways to 'triger export event', probably there are more...
1. When record is saved in ModelValidator
2. At some predefined schedule & AD_ChangeLog
3. Use Workflow functionality and add new Node which will be responsible for sending XML Message
I agree. But ModelVlaidator is one possible opion.
At the same time Replication MUST be guaranteeed. Which means that record should not be saved if Change Message can't be sent to Message Storage.
JMS guarantee delivery once message is stored into Queue or Topic.
I think to install on each local Adempiere instance one JMS Server which will collect all messages from local clients.
It will be task of JMS Server to transfer message to other Adempiere Instances once Internet connection is online.
Option 2. At some predefined schedule & AD_ChangeLog
I think that it is possible to be done! But..
According to me require some additioanl processing. We must take care to collect all non sent to JMS Server Change Messages which duplicates work of JMS server.
I think that we can implement all options at some moment if we have resources and time.
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I would like once again to thank you for your opinion.
You comment for Ad_ChangeLog made me to research more deeply if i could use it now in Replication.
Unfortunately at this moment i found that AD_ChangeLog do not work in 2 of 3 general uses cases:
** There are uses cases in which AD_ChangeLog do not work.
*** Create new Business Partner. AD_ChanegLog do not contain information that new record was created.
I think that this aplies for all other tables.
*** Delete existing Business Partner. AD_ChanegLog do not contain information that record was deleted.
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
All these projects needs replication or integration.
All these projects are being sponsored/developed in an unconnected way.
Trifon said EDI has one solution and replication will have another solution.
While I think that we can find a common approach for both problems, and embracing the other needs at the same time.
I think we need to structure the PMC for coping with such "projects-integration" needs.
What I'm seeing is we are going to finish with five paid projects (lots of US$) - and five single solutions that work for every single need, forming all of them a "bedspread of snippets" that can't be joined and integrated easily in Adempiere.
We could exploit this situation to join all those small projects and try a bigger generic project "ADempiere replication and integration architecture (and development)" - that can be used for all those things.
More thinking about this:
- Linux has Linus Torvalds to put some order to the architecture
- Compiere has Jorg Janke to put some order to the architecture
- ADempiere??? ADempiere NEEDS an active PMC for this
---------------------
Now to the technicals:
> I see 3 ways to 'triger export event', probably there are more...
> 1. When record is saved in ModelValidator
> 2. At some predefined schedule & AD_ChangeLog
> 3. Use Workflow functionality and add new Node which will be
> responsible for sending XML Message
You're mixing the saving of the message and the delivery of the message.
If a table is marked to replicate - it ALWAYS must save the info for replication.
And transaction must fail if the replication info can't be saved (just like it happens with AD_ChangeLog).
The other part of the problem is how and when to send the message and received the confirmation of the counterpart system.
The transport layer must be flexible / definable in many ways:
- frequency of deploying of messages (it could be immediate)
- transport method (i.e. file saved in a directory, file sent via ftp, calling a web service in a remote system, sending and EDI frame via XXX, sending an e-mail with the message in the body - or attached, inserting directly the message in another database ... etc)
- ways to react to the confirmation of the message (resend? wait? rollback?)
One of the biggest problems in replication is that many times you CANNOT simply send the transactions grouped by table, but you must send the transactions in the same order as they happen (to avoid referential integrity problems in the target system).
So, you have to make sure you can replicate in the same order (AD_ChangeLog.Created?)
---------------------
> Unfortunately at this moment i found that AD_ChangeLog do not work
> in 2 of 3 general uses cases
Of course Trifon, AD_ChangeLog is not complete, that must be enhanced.
Good you point there. We're just talking if AD_ChangeLog can be enhanced (and I think it can be) to support easily more replication scenarios.
---------------------
> But EDI can't be reused as by design(not design in Adempiere) in EDI
> we can have duplicate Element names which can't be resolved to which
> Column to be mapped.
> Also EDI has three lier structure : EID Format -> EDI Segment ->
> EDI Field.
> While in Replication and XML we have two tier system: Export
> Format(Table) -> Export Format Line(Column) which can point to other
> Export Format. So we can get unlimited tree structure of XML.
I didn't get this point.
Looking Adempiere as Table/Column replication is too narrow.
We need to ensure the replication of TRANSACTIONS, not records.
That's the point of EDI Format/Segment/Field.
In Adempiere we need to replicate
* Invoice (1 record)
** InvoiceLine's (many records)
** InvoiceTax'es (many records)
** InvoicePaySchedule's (many records)
as one transaction.
If the replication of one invoiceline fails, the whole transaction must fail.
So you need to add some transaction information to the message, or construct a complete message in XML tree holding the complete trx.
You can have even more complex scenarios with detail of detail (i.e. BP, BPLocation, Contact?), and moreover we must cope the need of replicating customized columns and tables, and an implementation can have a child table of InvoiceLine.
IMHO EDI is just a way to format the message.
---------------------
I think this problem goes beyond the forum capabilities, we need to make brainstorms, meetings, etc. We need to attract community wisdom.
Collected money from the projects could be distributed fairly between all of them participating in the meetings, design, development, etc.
Regards,
Carlos Ruiz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
>So you need to add some transaction information to the message, or construct a complete message in XML >tree holding the complete trx.
>
>You can have even more complex scenarios with detail of detail (i.e. BP, BPLocation, Contact?), and >moreover we must cope the need of replicating customized columns and tables, and an implementation can >have a child table of InvoiceLine.
Yes. Currently Export format can build complex tree XML files.
One small example:
as a consequence of my documentation effort with Victor I am digging into the Code and so I begin to understand more or less when a theme like this is touched.
I noticed that Adempiere first describes what has to be done in AD. Then it has an interacting hierachy of classes which read from AD, match to actual code instantiations and do the logic.
Sticking to this policy if we should implement replication, the first thing whe had to do is to describe it in the Application Dictionary: are there different types of replication, which elements are involved, what has to be done in either case etc. Then classes and code should follw.
Replication is on the other hand very complex as I infer from Trifon's and other posts here. As I understand Trifon, I suppose, too, that a "thorough" implementation will not be done for different reasons in the project.
Here my suggestion to Trifon in order to reach a compromise between a "general" and a "one-way" implementation:
- General idea: build your code supposing there is an AD description of Replication.
- build abstract classses, interfaces which would be filled from AD, even if they are only stubs at the beginning. The only thing you would do is to implement their constructors.
- write the actual replication as it is done in many main() methods I found in Adempiere, where functionalyt is shown in a compact way.
- this way, the possibility of implementing replication description in AD still exists and also the mechanism of handling it in code. Developers can then enhance the functionality. Trifon: perhaps it is easier said than done, but I wanted to mention it to you ;-) .
The issue of a person/committee in charge of Software Architecture is very actual.
Best regards,
Mario Calderon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Doesn't replication imply that you need to keep two or more databases exactly synchronized?
'Defining the replication in the AD' seems like a huge risk. Any small detail missed in a manual setup could corrupt the other databases.
The requirement would seem to be much more of a forced synchonization. The deltas from both instances need to be extracted and reconciled. No doubt there are oracle or postgres tools that do that in a db specific way. Are there generic tools for db replication?
If a specific tool must be built to deal with the uniqueness of Adempiere keys, wouldn't it be preferable that the tool automatically replicated the entire db, rather than needing to manually define what to replicate?
Just wonderin...
Regards,
Joel Stangeland
www.idalica.com
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
There is a hidden field "Replication Type" in AD_Table.
There is code to manage different sequences in different installations.
There is code to replicate in ReplicationLocal.java -> looks like it manages the replication set based on the Updated column of replicated tables.
There is some code (looks unused) in ReplicationRemote.java - I suppose this code is intended to be used in the remote installations.
I'm not saying this is a complete solution, it must be enhanced:
i.e. Updated column must be guaranteed
or changed to manage AD_ChangeLog instead of
AD_ChangeLog is just another seed that needs to be enhanced. There are known flaw points.
Even I'm not saying this is a good solution if we want to cover EDI and integration with OSCommerce, looks like JJ seed just cover integration between Adempiere's
But as well as Heng Sin, I'm wondering also why starting from zero?
And I'm asking why every development is seen as independent from the others?
Why not joining all those correlated projects?
Why not making a good enough architecture - it doesn't matter if the developed solution covers just 10% of the architecture?
Do we know the later cost of fixing a wrong architecture? A design flaw?
What's the difference in core concept between EDI and replication? Just the message format?
IMHO EDI is just another type of replication - and integration with OSCommerce is another type.
Even I see Import functionality a way to replicate, but it can be enhanced A LOT!
i.e.
- managing record types (currently just one record type can be managed)
- managing groups of records (this allow us to manage transactionality)
I made a complete solution for Oracle (unhappily PL/SQL & Forms based) to manage import/export of complex files (record types, transactionality), even this solution could manage the situation of receiving messages not from files.
------------------------
Joel:
> Doesn't replication imply that you need to keep two or more databases
> exactly synchronized?
There are lots of needs on replication:
- like BI - replicate summarized information to a read-only system
- for reporting purposes - replicate exact information to a read-only system
- like integration with OSCommerce - it can be seen as a replication because the final objective is to have the same info in both systems although different
- like integration with other businesses via EDI - same as previous
- like integration of POS - is a two-way replication but every table can be seen as one-way
- there is another more complex scenario (merging) where the same table is updated in both systems - it needs management of collisions (manual or automatic)
- here the list continues with a long etcetera
> 'Defining the replication in the AD' seems like a huge risk. Any small
> detail missed in a manual setup could corrupt the other databases.
I think if you need flexibility the only way is adding some level of complexity like this.
Integration between systems is a consistent headache :-)
For such needs the industry invented: interfaces - EDI - EAI - WebServices - SOA ... another etcetera here
------------------------
In other words, what I'm trying to say is -> replication is a big thing, is structural, architectural.
It must not be seen as a solution for every need, that's the old way of constructing interfaces between systems.
It will work, but none solution will serve for a new need.
Maybe "sponsored development" is not the best way to develop a GOOD SOLUTION for ADempiere.
Possibly it can be a good solution for a single need, but I doubt it can be a good solution for the whole problem.
AS ALWAYS, I CAN'T AND I DON'T WANT TO STOP ANY WAY OF WORK HERE, ANY TRIAL OF INNOVATION.
AS ALWAYS, I'M JUST WARNING AND ADVISING.
Regards,
Carlos Ruiz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
>Doesn't replication imply that you need to keep two or more databases exactly
>synchronized?
Not necessary exactly. In some cases user wants only sales data or Products data synchronized. Depending on what they want to achieve. When you have multiple stores in differente locations you need to have products replication from central instance to local instances and sales data replicated from local to central. You do not need all sales data replicated to all instances.
>But as well as Heng Sin, I'm wondering also why starting from zero?
I think that this what i have done is just begginig part. I do not think that it is from zero. I do not see any Export/Import functionlaity in Admepiere which can work for all tables or which is configuirable or export functionlait which is triggered on each new record creation. This is what i have done till now na di think that it gives some new functionality which can be good for other integration projects, like oScommerce. Exporting records on some trigger is functionality which other systems have and which makes system capable to talk with externals systems and whihc i think is good and valuable.
Till now i made posibility to Export/Import records from Adempiere, moreorver i made posibility user to define structure of this exported record.
Now i have to define interface which could have different implementations for different transport mechanisms (JMS, http, file storage,..)
I think that this is the first REQUIRED CONDITION for any development in replication area. I do not know how to make replication without to be able to Export/Import. So from now on there are many possibilities.
Just one thing. Very important one. I MUST deliver working version of EDI/Replication/what ever i do...
Joel had reall client who needs EDI. Victor has reall client too. I do not have 6 months to spend in architectrure of big system lso noone can guarantee that such big thing would work there are risks in any development project. I preffer to make working application and after this on second stage to improve. Please do not push me to make big desing. Also i asked concrete question, unfortunately i do not get suggestion or help, just proposals till now. They are very good, but at the end of the day i have to make the change for which i asked. I feel a bit sorry as i have to spend lot of time to read forum.
Also mixing EDI and Replication at early stage is dangerous. EDI had to be delivered prior replication. EDI must work for Joel client and Replication must work for Victor. I can't put EDI on risk because of Replication. If i try to make big design now i think i will fail. That's why i preffer to make working version of EDI and working version of Replication independent. Also please note that there are companies that build applications for transforming XML to EDI and EDI to XML and this are big applications, which for me means that it is more work if i want to dovelop general design or if i want ot intergate such application with Adempiere.
I completely agree with you Calros, but there are specific conditions under which we work and which force us to do what is necessary to do. Nothing more or nothing less.
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
> I do not know how to make replication without to be able to Export/Import.
There are many other approaches i.e. WebServices, RPC
The Export/Import approach is just one type of, and what I found incorrect in the Export/Import approach is that is mixing the data, message and transport to only one solution.
That's why I'm saying it must be seen as layers:
- Layer of data saving for replication (AD_ChangeLog?)
- Layer of construction of the message (read the change log and construct the message)
*** Types of messages: XML file, EDI frame ... etcetera
- Layer of transport of the message
*** transfer a file via ftp? calling a Webservice? sending an e-mail?
If you focus just on export/import approach you're mixing the layers and constructing a solution that can't work easily with other needs.
That's exactly what's happening, replication can't exploit EDI development.
> I preffer to make working application and after this on second stage to improve.
> Please do not push me to make big desing.
I can't push you to make a big design. I just can advice contractors to ask for that.
And as you pointed mostly there are dependencies on time to deliver.
But in my long experience working for many companies - is a mistake to deliver a badly-designed working solution.
Obviously is a decision of the contractor - just pointing AGAIN that if contractor put the wrong conditions it will be difficult the development reach the trunk - because it will finish being a very narrow solution - almost like a customization.
> Also i asked concrete question, unfortunately
> i do not get suggestion or help, just proposals till now.
I think the answer for your concrete question is precisely what we're discussing here.
Developing a solution just thinking on export/import an XML file is too narrow, I'll recommend to avoid such thinking when changing the core (i.e. PO.java)
If this is being developed as a specific customization, then I agree to make it in ModelValidator as you proposed initially. And we must face reality, the design will be narrow because we need speed of development, and possibly it won't reach trunk.
To change PO.java (very important core class) I think a better design is needed.
CAUTION: THIS IS JUST MY OPINION - I'M NOT PUSHING NOTHING HERE - AGAIN I'M JUST ADVISING.
If others in this project think different we can discuss and open spaces for policies and rules.
> If i try to make big design now i think i will fail.
I don't think so, my advice is you always need to think on big design.
Imagine what would be Compiere/Adempiere is JJ don't think big design every time?
What I admire from JJ is precisely he thinks big design even when he's contracted to make small things.
>There are many other approaches i.e. WebServices, RPC
>The Export/Import approach is just one type of, and what I found incorrect in the Export/Import approach >is that is mixing the data, message and transport to only one solution.
>
>That's why I'm saying it must be seen as layers:
>- Layer of data saving for replication (AD_ChangeLog?)
>- Layer of construction of the message (read the change log and construct the message)
>*** Types of messages: XML file, EDI frame ... etcetera
>- Layer of transport of the message
>*** transfer a file via ftp? calling a Webservice? sending an e-mail?
I would like to point that i can construct XML message from AD_ChangeLog i do not say that i will not do it. AD_ChangeLog has information for contstructing XML message. I just say that for initial Prof of Concept it will be overkill to fix AD_ChangeLog functionality.
>The Export/Import approach is just one type of, and what I found incorrect in the Export/Import approach >is that is mixing the data, message and transport to only one solution.
Why do you think so?
I do not say that i will make all in one solution.
For EDI i made liers. I think that i can make liers for Replication too.
>Developing a solution just thinking on export/import an XML file is too narrow,
>I'll recommend to avoid such thinking when changing the core (i.e. PO.java)
I think that even with AD_ChangeLog PO have to be changed as it will trigger new event that NEW record is created in the system. This new event will create new record in AD_ChangeLog which will lead to export and we will get cycle.
OK. We can disable making new record in AD_CahngeLog, but what happens if someone listen for new records in the system and send SMS message for exmaple. I think that this listener it must distinguish between record created by user adn record created by Replication process. It is just additional notification type. I think that it is addition to the system not change.
Please advice how to avoid changing PO and i will implement it.
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
> For EDI i made liers. I think that i can make liers for Replication too.
I would recommend instead of creating new tiers, try to reuse the same.
In fact instead of creating new tables to describe the EDI format (C_EDIFormat/ C_EDIFormat_Line/ C_EDIFormat_LineElement) I would preferred to extend AD_ImpFormat/ AD_ImpFormatRow functionality.
-------------
NOTE ASIDE:
Maybe we're finding a possible problem in "Sponsored Development" approach.
Asking to Sponsors: Can we make some sort of "Sponsored Design" also?
We could make brainstorms, meetings, forum discussions, wiki discussions, etc?
And share the "sponsored design" money between people participating in design according to the participation?
Please don't take it wrong.
I'm just throwing ideas trying to get more community wisdom to sponsored work.
-------------
>> The Export/Import approach is just one type of,
>> and what I found incorrect in the Export/Import
>> approach is that is mixing the data, message
>> and transport to only one solution.
> Why do you think so?
Because the Export/Import functionality you're proposing is pointing to "trigger" the export from PO (data saving) and it implies just one format of the message (XML format of ADCK) and it implies just one way of transport (sending or replicating a XML file).
I would prefer a solution where:
* the data saving is generic (like AD_ChangeLog or AD_ReplicationLog).
* and the definition of the tables being "saved" are flexible (like AD_ReplicationTable?)
* and the definition of the message format is centralized and flexible (like extending AD_ImpFormat/ AD_ImpFormatRow?)
* and the definition of the transport of the message is centralized and flexible (like extending AD_Replication or extending C_EDIProcessorType)
-------------
> I think that even with AD_ChangeLog PO have to be changed as
> it will trigger new event that NEW record is created in the
> system. This new event will create new record in AD_ChangeLog
> which will lead to export and we will get cycle.
> Please advice how to avoid changing PO and i will implement it.
It's hard Trifon, I think solution must be thought looking at the complete picture.
For your specific issue related to a possible loop in ModelValidator, I have two opinions:
1 - There is no problem, first time the ModelValidator will be called for the table (i.e. C_BPartner), second time the ModelValidator will be called for AD_ChangeLog, no loop in this case.
2 - If you find an scenario (very possible) where the ModelValidator is called in a loop, the easiest solution could be to implement an approach like the isCalloutActive/setCalloutActive.
Regards,
Carlos Ruiz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
i would like once again to thank you for participating in this disussion as it clarify things.
>I would recommend instead of creating new tiers, try to reuse the same.
>In fact instead of creating new tables to describe the EDI format (C_EDIFormat/
>C_EDIFormat_Line/ C_EDIFormat_LineElement) I would preferred to extend
> AD_ImpFormat/ AD_ImpFormatRow functionality.
I think that this is not possible easely. I do not say not completely possible but...
Let me explain.
--AD_ImportFormat and AD_ImportFormatRow are intended to allow user to define import format, but can't help in export.
--Also they are found to I_Xxxx tables.
--Existing Import in Adempiere is two step process. For EDI we need one step process.
EDI really needs 3 tables. Initial desing was with 2 but i had to add 3-rd one.
>Because the Export/Import functionality you're proposing is pointing to "trigger"
>the export from PO (data saving) and it implies just one format of the message
>(XML format of ADCK) and it implies just one way of transport (sending or
> replicating a XML file).
Let's forget about ADCK for Replication(Even if it is better tool :) ). Replication will completely replace it.
Format of XML message can be defined by user(EXP_Format and EXP_FormatLine in Replication project. It can contain primary keys or whatever XML sturcture user needs). I choosed XMl format as it can be transformed.
As i pointed in wiki i see three ways:
ModelValidator
AD_CahngeLog
Workflow
>I would prefer a solution where:
>
>* the data saving is generic (like AD_ChangeLog or AD_ReplicationLog).
>* and the definition of the tables being "saved" are flexible (like AD_ReplicationTable?)
>* and the definition of the message format is centralized and flexible (like extending AD_ImpFormat/ >AD_ImpFormatRow?)
>* and the definition of the transport of the message is centralized and flexible
>(like extending AD_Replication or extending C_EDIProcessorType)
Yes, i like this plan(but do not see AD_CahngeLog as the only way). And i think that it is possible to be done.
As i pointed till now i made just way to define Format of message.
>* and the definition of the message format is centralized and flexible (like extending AD_ImpFormat/ >AD_ImpFormatRow?)
Sorry, but i find AD_ImpFormat and AD_ImpFormatRow not good for this case.
1. They can't describe tree structure.
2. Also they are intended to import data from plain text files which are not tree oriented also.
3. Also if i change them i will broke existing functionality. I think that existing Import functionality is good and flexible for plain files that's why i do not want to change it.
I would like to point that i created new tables EXP_format and EXP_FormatRow.
If we find way to merge AD_ImpFormat and EXP_Format than - no porlbem i agree.
>For your specific issue related to a possible loop in ModelValidator, I have two opinions:
>
>1 - There is no problem, first time the ModelValidator will be called for the table (i.e. C_BPartner), >second time the ModelValidator will be called for AD_ChangeLog, no loop in this case.
I do not agree.
1. First time CODE(ModelValidator or whaterver) will be called when user create record.
2. AD_CahngeLog is Created.
3. Process read AD_ChangeLog and send message to external system that new record is created.
4. External system read message and creates new record into remote instance.
5. External system save message into AD_ChangeLog.
6. External system construct export mesage from AD_ChangeLog.
7. Local system get message from remote system that new record is created... And we get cycle.
How to avoid it?
>
>2 - If you find an scenario (very possible) where the ModelValidator is called in a loop, the easiest >solution could be to implement an approach like the isCalloutActive/setCalloutActive.
What i made is to add new property into PO and set it to true when saving Replicated record. But i also need new ModelValidator Event as i would like to distinguish if record is from replication or not. I think that such addiotna is good one as allos to build system which can notify upon creation of replicated record.
Once again thank's for the discussion.
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Trifon, I didn't say that AD_ImpFormat and AD_ImpFormatRow are ready to be used for export or for formatting messages.
What I'm trying to say is that we must try to use the same objects.
Now we have AD_Imp... objects, C_EDI... objects, and now AD_Exp... objects, all of the three used to describe formats of records/messages/frames/whatever.
What I mean is we must not create new tables for every need but trying to extend current functionality.
Look the original design of Adempiere, the same tables are used for Sales and Purchase orders, the same tables are used for Customer and Vendor Invoices, the same tables are used for Receipts and Payments.
I'm worried if we start creating new tables for every need even when some similar functionality exists.
What we must ask ourselves is: how AD_ImpFormat and AD_ImpFormatRow can be extended to manage better formatting of messages (not just files):
- possibly we will need a third table
- possibly we will need to independize the AD_ImpFormat from I_ table and make it more flexible (preserving compatibility with old import processes)
----------------------------
> I do not agree.
> 1. First time CODE(ModelValidator or whaterver) will be called when user create
> record.
> 2. AD_CahngeLog is Created.
> 3. Process read AD_ChangeLog and send message to external system that new record
> is created.
> 4. External system read message and creates new record into remote instance.
> 5. External system save message into AD_ChangeLog.
> 6. External system construct export mesage from AD_ChangeLog.
> 7. Local system get message from remote system that new record is created...
> And we get cycle.
Trifon, I'm still seeing you're mixing the saving of the data with the delivery of the message.
You must not deliver a message within an incomplete transaction (it can be rolled back).
You need to see the saving of the data independent of the sending of the message.
Data Saving:
Saving of the data -> possibly AD_ChangeLog or AD_ReplicationLog or whatever (not writing XML files).
Saving of the data must occurs ALWAYS and if the data can't be saved the process must be rolled back.
---------------------------------------
Message Formatting and Message Sending:
---------------------------------------The message formatting and sending must occurs in a later step (preferrably defined in a flexible way).
It can be at commit or it can be in a process running every n minutes.
Message Formatting:
The format of the message can be defined in a flexible Message/Record/Field definition (like EDI, Import or export formatting).
In AD_Imp there is a trial to manage transformation via Callouts.
We could make a generic formatting solution based on those tables.
Message Sending:
The sending of the message(s) is different - we need to define the frequency, the transport, and the way to manage confirmation receipt
Frequency:
- It can be immediate and mandatory, meaning the transaction must be rolled back if the message can't be sent and confirmation received. This implies a big problem in replication systems - some databases solved via two-phase commit protocol - http://en.wikipedia.org/wiki/Two-phase_commit.
But possibly this is out of scope of initial solution.
So I think implementing immediate and mandatory messaging must be avoided at this moment, but we can design the structures to support it in future.
- Frequency can be also every n minutes, or hours, or days, or maybe every n messages.
And we could have different frequencies for different targets.
- It can also be manual pushing a button - as you made in EDI
Transport:
- Here we can have several ways of transport and the data needed for the transport, i.e:
* ftp - we will need target host, directory, user and password
* e-mail - we will need just a target e-mail
* file - we will need the directory and some naming convention
* database - a possibility is the message format is the plain SQL, then the transport will need the target database connection definition
* etc
I'm not saying you must implement all these transport methods, just define them and allow a simple way to extend them.
And implement just the way currently needed and agreed with Joel or Victor.
---------------------
Confirmation Receipt:
---------------------
This is harder, we need to define how to act on confirmation of the target system.
If we must wait, if we must resend, if we must ignore. There are many possible scenarios also.
i.e.
* we can ignore a failed replication that replicates the whole table.
* we CAN'T ignore a failed replication that represents transactional data, in fact the rest of the replication must be stopped.
The target system must implement the sending of the confirmation. This can be like defining a replication from in the opposite way.
-------------------------
JOKING: I'm thinking how much to charge for this design brainstorm ;-)
Regards,
Carlos Ruiz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I would like to write more on what i think regarding replication and other integratin projects which are comming.
For replication and generaly for intergation every application needs Import / Export functionality.
In Admepiere we have some Import / Export functionality but it is not generally applicable to all tables in the system. That's why my first task in Replication project was to make general Import / Export functionality in Adempiere. It is not tied to Replication or any other project. It is just functionality which allows user of Adempiere to define format of XML document(it can can tree structure). Please refer to example in the Replication project: http://www.adempiere.com/wiki/index.php/Sponsored_Development:_Replication#Example
I would like to say that i'm glad that i managed to do such general Import / Export functionality as it will allow further integration with external system possible task. Also i would like to poin that i managed to do this only because we have Application dictionary( it is my big friend which helps me in all tasks).
Alos i would like to say that this Export format is completely managed by AD(Just a point to Mario that we have to define all in AD - I completely agree Mario)
Strange thing in the case is that i found that i can use Export format as Import format too. Probably it is not so strange but it was a good surpise for me. I was prepared to create two tables.
Currently i can say that this genral Import / Export functionality works. I have to finish one thing in Import but it is just matter of some hours to be done.
Now to get back to concrete project - Replication. After we have Export / Import functionality we must find way to transfer CHANGE message to external system. My plan is to use JMS as proven technology which guarantee transfer of messages. I have to show to Victor a proof of concept that such replication can be done with the choosen techonolgy. For this task by plan i had 2 weeks. After that we have to continue or not with further development.
Till now i had to toch core system only once when i had to change ModelValidator and PO in order to avoid cycle issue i faced. I think that this is general issue. For any system where listeners/triggers can be defined a special care must be taken in order to avoid cycles. The same cycle will hapen if i define triger which creates new record in the same tables where triger is attached.
So this is status of Replication -> i must make JMS sending and recept.
At second stage we will have to go deeply into details:
-- Existing infrasructure
-- Different Replication strategies.
-- ...
-- all other detials pointed by Calros, Hengsin, Mario, Joel and Armen
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for this clarification. Interestingly, it helps to clarify the separation that Carlos outlines here (https://sourceforge.net/forum/message.php?msg_id=4450603). Since they were posted a couple minutes apart it doesn't seem yours was a response... Great minds do think alike!
It appears you are already separating the transport piece.
- Layer of data saving for replication (AD_ChangeLog?)
- Layer of construction of the message (read the change log and construct the
message)
- Layer of transport of the message
There are some other issues that need to be separated in order to understand the big picture, namely Replication and Integration.
Replication implies the matching and synching of two ADempiere Systems. The requirements are unique here, because you are not creating ADempiere documents, but sharing existing ones.
Critical issues to manage:
* ID generation and resolution.
* Sharing changes to AD
* Business rules were applied in the creation of the data, so it is a data synch.
* Replication would presumably be 'automoted'
* Again I would assert you can't pick and choose items to replicate. It is not 'loosely-coupled'. Each system has to have the same AD, the same BP's, the same users, etc etc.
Integration implies sharing data between ADempiere and a non-adempiere system. The requirements are different, because an incoming records create a new ADempiere document.
Critical issues to manage:
* Incoming data needs to apply the business rules, therefore, must be processed through the model.
* The model can handle the id generations naturally.
* Data transformation belongs outside ADempiere, because it will be custom depending on the source system and whether it is customized. Ultimately, it would be nice to have some 'connectors' pre-configured for things like OSCommerce. That's what makes that a different project.
* Integrations are not necessarily automatic.
Time may prove this wrong, but I'm not convinced that the EDI requirements are or could have been reusable for either Replication or Integration. EDI mapping is a complicated process in itself, and the EDI tools in ADempiere are designed to simplify that from the AD user standpoint.
Maybe what trifon has begun with his replication Import/Export is the start of a generic Interface that can be used for the integrations as well?
Some things that would make this useful:
* If the data is ETL'd from the integrated system into this interface, it can be run through the model and thus put into ADempiere.
* For outgoing integrations, a standard interface that serves as the starting point for ETL's to other systems.
Then plug-ins could be developed for systems like OSCommerce or QuickBooks. These could be re-usable plug-ins, and could be used for initial data imports as well.
Hope there was something useful in those musings...
Regards,
Joel Stangeland
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
>Thanks for this clarification. Interestingly, it helps to clarify the separation that Carlos outlines here
You welcome. I would like to thank you for entering this discusison as Implementation Providers and End users are reall users of the functionality developers make.
>Time may prove this wrong, but I'm not convinced that the EDI requirements are or could have been >reusable for either Replication or Integration. EDI mapping is a complicated process in itself, and the >EDI tools in ADempiere are designed to simplify that from the AD user standpoint.
I hope that EDI mapping is simple from user point of view. You will be the first company that will implement it for customer. I still must develop some parts of EDI, and i hope to do it this week.
>Maybe what trifon has begun with his replication Import/Export is the start of a generic Interface that >can be used for the integrations as well?
Yes, i hope so. Atleast this could allow Administrator to setup triger/notification mechanism which exports custom defined XML message upon record creation. I think something which would be good is custom creation upon Document creation. I hope that this Import/Exoprt functionality will be the starting point and will enable Adempiere to talk with external system.
>Some things that would make this useful:
>* If the data is ETL'd from the integrated system into this interface, it can be run through the model >and thus put into ADempiere.
>* For outgoing integrations, a standard interface that serves as the starting point for ETL's to other >systems.
Yes. When XML(now, or any other in future) message is received importer must find proper model and use it for saving of the record. I do not plan to use plain SQL Updates or Inserts. I even do not hink for such thing.
As a starting point for outgoing intergation i'm not sure what to write. But i think that EAI tools can be configuired to react on message receipt and perfrom actions.
Kind regards,
Trifon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The idea of using the ESB was to handle the conversion to the message format externally to ADempiere. The interface in and out of ADempiere could be consistent, and post to a JMS queue.
But then the choice of tools and formats for transport and messaging would not have to be re-coded.
Anyway, just to keep the creative juices flowing....
Joel Stangeland
www.idalica.com
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear Developers,
I'm developing Replication in Adempiere and i have one question which i think that concern all Developers.
Status of Replication now is:
1) Exoprt of Records into custom defined xml format is considered done.
2) Import of XML Formats is considered working. Here need to finish 1-2 things, but this will be done in next 1-2 days.
But i have one issue.
First let's describe how things work at the moment.
I have developed ModelValidator class which is responsible to generate proper XML Format when record is modified. All this means that class is invoked when type is TYPE_AFTER_CHANGE
if (po.get_TableName().equals(MBPartner.Table_Name) && type == TYPE_AFTER_CHANGE)
{
MBPartner bPartner = (MBPartner)po; // Cast is not necessary. Thisi s just an example.
exportRecord(bPartner);
}
Exportes works as expected.
I have developed a process which imports records and here comes the problem. Importer till now works as expected and manages to parse XML file and update record, but when Importer save the record Exporter is invoked and we get a cycle...
So i think that we need new TYPES in ModelValidator:
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_DELETE_REPLICATION
Also new method in PO:
saveReplication() or save(boolean isReplication)
This method will be something like:
public boolean save (boolean isReplication)
{
setReplication(isReplication));
return save();
} // save
and i have to modify save() method to use new TYPE.
Please let me know your opinion.
Kind regards,
Trifon
Sorry i made small mistake.
Please read this:
So i think that we need new TYPES in ModelValidator:
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_DELETE_REPLICATION
as:
So i think that we need new TYPES in ModelValidator:
TYPE_AFTER_NEW_REPLICATION
TYPE_AFTER_CHANGE_REPLICATION
TYPE_AFTER_DELETE_REPLICATION
Kind regards,
Trifon
Hi Trifon, thanks for asking.
I don't know how the design of replication goes, it's not clear in wiki page.
I see this is a big need on Adempiere, and is a big opportunity to make Adempiere more connectable with other systems.
Currently I see several correlated sponsored developments:
* Replication
* Integration with OSCommerce
* EDI
* Web Service
And some other "projects" talking about similar things like:
* BI
* Jitterbit
And looks like they're being developed in an unconnected way. It's a shame not exploit this situation to manage a "central" design - that must be the a good case for the PMC.
In past I made a lot of research and development in replication and synchronization between two systems, studying top EAI's in the market, and top heterogeneous db replicators.
Talking from my lack of knowledge of current Adempiere replication design, and knowing that I must reread a lot of EAI documentation, I think the current approach is too focused in developing quickly something functional and not in the core architecture needed.
I think replication must be seen in layers:
1 - a flexible way to define the tables involved in replication (for subscriber and for publisher)
2 - define the transformations needed (in subscriber or in publisher)
3 - saving the proper information in a messaging table or system
4 - a way to manage subscribers and publishers
5 - a delivering system for publishers - managing different ways of delivering
6 - a recollection system for subscribers - managing different ways of delivering
Mostly every system involved in a replication environment is a subscriber and a publisher at the same time.
But there are LOTS of needs in replication, i.e.:
- a one-centralized-read-write system replicating to one or many read-only systems (like BI)
- a one-centralized-read-write system replicating both ways with other(s) read-write systems (like POS)
- a one-centralized-read-only receiving information from many decentralized-read-write systems
etc
PD: A good summary of EAI in wikipedia article
http://en.wikipedia.org/wiki/Enterprise_application_integration
This could be seen as very big (looking like ETL tool or EAI tool), and indeed is a BIG thing.
Maybe instead of reinventing wheel we could research a little on the possibility of integrating into Adempiere one of the open source eai java tools, like:
http://www.manageability.org/blog/stuff/open-source-messaging-integration-transformation-routing-java/view
-----------------------
Again from my lack of knowledge of the design here, exporting/importing to XML is just a little part of the whole architecture needed.
-----------------------
NOTE ASIDE:
Every time I write about any topic, looks like a stopper because I mostly think big.
I know the open source rule of 10% seed, I just think if the 10% seed has a good architecture the 90% evolution is easier.
That's a main issue I think for the PMC, the ability to ensure a wide view of problems, and advising on solutions with a better architecture - it doesn't matter if just the 10% of the solution is done initially.
Regards,
Carlos Ruiz
Hi Carlos,
thank's for the answer.
Design is similar to EDI.
We will have Export/Import Layer and transport layer.
Administrator of system can define Export format which will be used as Import Format.
Administrator will define Transport mechanism.
For EDI i will implement JMS and Local file initially during development and test period.
If you look at EDI you will see the same structure.
But EDI can't be reused as by design(not design in Adempiere) in EDI we can have duplicate Element names which can't be resolved to which Column to be mapped. Also EDI has three lier structure : EID Format -> EDI Segment -> EDI Field. While in Replication and XML we have two tier system: Export Format(Table) -> Export Format Line(Column) which can point to other Export Format. So we can get unlimited tree structure of XML.
I also think that in Export Format we can have some kind of transformation, but it will be used when exporting to other systems, for Replication i think that it will not be used in most of the cases.
I also would like to point that currently i'm in the Prove Of Concept phase which by plan is two weeks. For completel work i think that i will need 1.5-2 months more.
I would like to make one general note. All available tools are not appropriate as they are not designed to work with AD. Power of Adempiere is AD from where developer can understand structure of model.
My question was more concrete.
I MUST modify core in order to fix issues which i sticked now. So i have to do it, but i would like to have Adempiere developer's opinion regarding the change. Probably there is better way or it could be done by simple way.
Kind regards,
Trifon
>I would like to make one general note. All available tools are not appropriate as they are not designed to work with >AD. Power of Adempiere is AD from where developer can understand structure of model.
>My question was more concrete.
One more note regarding previous note.
Every system must have some Export/Import functionality and after that external tools can be used. I do not say that i will not use any external tool or that external tools are not good, but prior using such tool Adempiere must have some way to export/import data. I do not think that using external tool to make integration at DB level is good think for one ERP system, that's why i concentrated to develop Export/Import first. I choose to make export/import in XML as it is most used and can be transformed.
AS you know integration can be done at several levels:
1) DB level. Application A reads/writes data directly to DB of Application B. I do not think that it is good in ERP system.
2) Integration at Application level. I think that here there were several types. But i do not remember them now and i do not have time to search them, sorry :(
Please feel free to add them and comment them!
Kind regards,
Trifon
Hi Trifon,
If possible, it would be better if we can extend and leverage what is already in Adempiere instead of building everything from the ground up. It should be possible to extend the changelog (ad_changelog) feature in Adempiere and use that to generate the replication message. Also, it might be good to add versioning support for all table which will help in implementing conflict resolution.
For the cycle issue, if the changelog mechanism is use, we can add a way to turn on or off the change log management during the save operation.
Finally, I don't think using ModelValidator to generate the xml export file is a good approach. Replication should be a background process that have little performance impact on normal transaction and the frequency of replication should also be user configurable. I think we should have a background process that read the changelog and replication configuration to generate the replication message ( xml or otherwise ) and send that out.
Regards,
Low
P.S It will encourage more collaboration and input if the spec/design is publish fully in the wiki.
Hi Low,
thank's for the opinion.
I modidifed wiki page and added some information.
I see 3 ways to 'triger export event', probably there are more...
1. When record is saved in ModelValidator
2. At some predefined schedule & AD_ChangeLog
3. Use Workflow functionality and add new Node which will be responsible for sending XML Message
I agree. But ModelVlaidator is one possible opion.
At the same time Replication MUST be guaranteeed. Which means that record should not be saved if Change Message can't be sent to Message Storage.
JMS guarantee delivery once message is stored into Queue or Topic.
I think to install on each local Adempiere instance one JMS Server which will collect all messages from local clients.
It will be task of JMS Server to transfer message to other Adempiere Instances once Internet connection is online.
Option 2. At some predefined schedule & AD_ChangeLog
I think that it is possible to be done! But..
According to me require some additioanl processing. We must take care to collect all non sent to JMS Server Change Messages which duplicates work of JMS server.
I think that we can implement all options at some moment if we have resources and time.
Kind regards,
Trifon
Hi Trifon,
>I see 3 ways to 'triger export event', probably there are more...
Also you need to take care many UPDATE sql statements.
If you look at the wiki, I wrote some known issues I met in the past.
Best regards,
Armen
Hi Low,
I would like once again to thank you for your opinion.
You comment for Ad_ChangeLog made me to research more deeply if i could use it now in Replication.
Unfortunately at this moment i found that AD_ChangeLog do not work in 2 of 3 general uses cases:
** There are uses cases in which AD_ChangeLog do not work.
*** Create new Business Partner. AD_ChanegLog do not contain information that new record was created.
I think that this aplies for all other tables.
*** Delete existing Business Partner. AD_ChanegLog do not contain information that record was deleted.
Kind regards,
Trifon
What I'm trying to say is:
There are several projects here pointing to replication/integration:
http://adempiere.com/wiki/index.php/Sponsored_Development:_Replication
http://adempiere.com/wiki/index.php/Sponsored_Development:_Interface_with_OSCommerce_%28urgent%29
http://adempiere.com/wiki/index.php/Sponsored_Development:_EDI_Import/Export
http://adempiere.com/wiki/index.php/Sponsored_Development:_WebService
* Jitterbit
And some other needs for replication/integration like:
* BI
* https://sourceforge.net/forum/message.php?msg_id=4448453
All these projects needs replication or integration.
All these projects are being sponsored/developed in an unconnected way.
Trifon said EDI has one solution and replication will have another solution.
While I think that we can find a common approach for both problems, and embracing the other needs at the same time.
I think we need to structure the PMC for coping with such "projects-integration" needs.
What I'm seeing is we are going to finish with five paid projects (lots of US$) - and five single solutions that work for every single need, forming all of them a "bedspread of snippets" that can't be joined and integrated easily in Adempiere.
We could exploit this situation to join all those small projects and try a bigger generic project "ADempiere replication and integration architecture (and development)" - that can be used for all those things.
More thinking about this:
- Linux has Linus Torvalds to put some order to the architecture
- Compiere has Jorg Janke to put some order to the architecture
- ADempiere??? ADempiere NEEDS an active PMC for this
---------------------
Now to the technicals:
> I see 3 ways to 'triger export event', probably there are more...
> 1. When record is saved in ModelValidator
> 2. At some predefined schedule & AD_ChangeLog
> 3. Use Workflow functionality and add new Node which will be
> responsible for sending XML Message
You're mixing the saving of the message and the delivery of the message.
If a table is marked to replicate - it ALWAYS must save the info for replication.
And transaction must fail if the replication info can't be saved (just like it happens with AD_ChangeLog).
The other part of the problem is how and when to send the message and received the confirmation of the counterpart system.
The transport layer must be flexible / definable in many ways:
- frequency of deploying of messages (it could be immediate)
- transport method (i.e. file saved in a directory, file sent via ftp, calling a web service in a remote system, sending and EDI frame via XXX, sending an e-mail with the message in the body - or attached, inserting directly the message in another database ... etc)
- ways to react to the confirmation of the message (resend? wait? rollback?)
One of the biggest problems in replication is that many times you CANNOT simply send the transactions grouped by table, but you must send the transactions in the same order as they happen (to avoid referential integrity problems in the target system).
So, you have to make sure you can replicate in the same order (AD_ChangeLog.Created?)
---------------------
> Unfortunately at this moment i found that AD_ChangeLog do not work
> in 2 of 3 general uses cases
Of course Trifon, AD_ChangeLog is not complete, that must be enhanced.
Good you point there. We're just talking if AD_ChangeLog can be enhanced (and I think it can be) to support easily more replication scenarios.
---------------------
> But EDI can't be reused as by design(not design in Adempiere) in EDI
> we can have duplicate Element names which can't be resolved to which
> Column to be mapped.
> Also EDI has three lier structure : EID Format -> EDI Segment ->
> EDI Field.
> While in Replication and XML we have two tier system: Export
> Format(Table) -> Export Format Line(Column) which can point to other
> Export Format. So we can get unlimited tree structure of XML.
I didn't get this point.
Looking Adempiere as Table/Column replication is too narrow.
We need to ensure the replication of TRANSACTIONS, not records.
That's the point of EDI Format/Segment/Field.
In Adempiere we need to replicate
* Invoice (1 record)
** InvoiceLine's (many records)
** InvoiceTax'es (many records)
** InvoicePaySchedule's (many records)
as one transaction.
If the replication of one invoiceline fails, the whole transaction must fail.
So you need to add some transaction information to the message, or construct a complete message in XML tree holding the complete trx.
You can have even more complex scenarios with detail of detail (i.e. BP, BPLocation, Contact?), and moreover we must cope the need of replicating customized columns and tables, and an implementation can have a child table of InvoiceLine.
IMHO EDI is just a way to format the message.
---------------------
I think this problem goes beyond the forum capabilities, we need to make brainstorms, meetings, etc. We need to attract community wisdom.
Collected money from the projects could be distributed fairly between all of them participating in the meetings, design, development, etc.
Regards,
Carlos Ruiz
Hi Carlos,
>So you need to add some transaction information to the message, or construct a complete message in XML >tree holding the complete trx.
>
>You can have even more complex scenarios with detail of detail (i.e. BP, BPLocation, Contact?), and >moreover we must cope the need of replicating customized columns and tables, and an implementation can >have a child table of InvoiceLine.
Yes. Currently Export format can build complex tree XML files.
One small example:
<C_BPartner AD_Client_Value="SYSTEM">
<AD_Client>
<AD_Client_Value>GardenWorld</AD_Client_Value>
</AD_Client>
<AD_Org>
<Value>0</Value>
<AD_Client_Value>
<AD_Client_Value>SYSTEM</AD_Client_Value>
</AD_Client_Value>
</AD_Org>
<Value>GardenAdmin-8</Value>
<Name>GardenAdmin BP-9</Name>
<DUNS>Duns-1</DUNS>
</C_BPartner>
This is generated by Exporter.
I thik that it could even export complete Client tree. All data which belongs to given AD_Client, but administrator must build Export Format firts...
Kind regards,
Trifon
This is example with more data:
<C_BPartner AD_Client_Value="GardenWorld">
<AD_Client>
<AD_Client_Value>GardenWorld</AD_Client_Value>
</AD_Client>
<AD_Org>
<Value>HQ</Value>
<AD_Client_Value>
<AD_Client_Value>GardenWorld</AD_Client_Value>
</AD_Client_Value>
</AD_Org>
<Value>Test-Replication-BP-3</Value>
<Name>Test-Replication-BP-3</Name>
<DUNS>Duns-3 </DUNS>
<Created>2007-08-05 21:21:37.0</Created>
<CreatedBy>
<Name>SuperUser</Name>
<AD_Client_Value>
<AD_Client_Value>SYSTEM</AD_Client_Value>
</AD_Client_Value>
</CreatedBy>
<Updated>2007-08-05 21:21:37.0</Updated>
<UpdatedBy>
<Name>SuperUser</Name>
<AD_Client_Value>
<AD_Client_Value>SYSTEM</AD_Client_Value>
</AD_Client_Value>
</UpdatedBy>
</C_BPartner>
Kind regards,
Trifon
Hello there,
as a consequence of my documentation effort with Victor I am digging into the Code and so I begin to understand more or less when a theme like this is touched.
I noticed that Adempiere first describes what has to be done in AD. Then it has an interacting hierachy of classes which read from AD, match to actual code instantiations and do the logic.
Sticking to this policy if we should implement replication, the first thing whe had to do is to describe it in the Application Dictionary: are there different types of replication, which elements are involved, what has to be done in either case etc. Then classes and code should follw.
Replication is on the other hand very complex as I infer from Trifon's and other posts here. As I understand Trifon, I suppose, too, that a "thorough" implementation will not be done for different reasons in the project.
Here my suggestion to Trifon in order to reach a compromise between a "general" and a "one-way" implementation:
- General idea: build your code supposing there is an AD description of Replication.
- build abstract classses, interfaces which would be filled from AD, even if they are only stubs at the beginning. The only thing you would do is to implement their constructors.
- write the actual replication as it is done in many main() methods I found in Adempiere, where functionalyt is shown in a compact way.
- this way, the possibility of implementing replication description in AD still exists and also the mechanism of handling it in code. Developers can then enhance the functionality. Trifon: perhaps it is easier said than done, but I wanted to mention it to you ;-) .
The issue of a person/committee in charge of Software Architecture is very actual.
Best regards,
Mario Calderon
Hello All,
Maybe I'm out to lunch on this, but...
Doesn't replication imply that you need to keep two or more databases exactly synchronized?
'Defining the replication in the AD' seems like a huge risk. Any small detail missed in a manual setup could corrupt the other databases.
The requirement would seem to be much more of a forced synchonization. The deltas from both instances need to be extracted and reconciled. No doubt there are oracle or postgres tools that do that in a db specific way. Are there generic tools for db replication?
If a specific tool must be built to deal with the uniqueness of Adempiere keys, wouldn't it be preferable that the tool automatically replicated the entire db, rather than needing to manually define what to replicate?
Just wonderin...
Regards,
Joel Stangeland
www.idalica.com
Good points Mario.
There is a seed from JJ currently in windows "Replication Strategy" and "Replication"
Configuration: AD_ReplicationStrategy + AD_ReplicationTable + AD_Table
Execution: AD_Replication + AD_ReplicationLog + AD_ReplicationRun
There is a hidden field "Replication Type" in AD_Table.
There is code to manage different sequences in different installations.
There is code to replicate in ReplicationLocal.java -> looks like it manages the replication set based on the Updated column of replicated tables.
There is some code (looks unused) in ReplicationRemote.java - I suppose this code is intended to be used in the remote installations.
I'm not saying this is a complete solution, it must be enhanced:
i.e. Updated column must be guaranteed
or changed to manage AD_ChangeLog instead of
AD_ChangeLog is just another seed that needs to be enhanced. There are known flaw points.
Even I'm not saying this is a good solution if we want to cover EDI and integration with OSCommerce, looks like JJ seed just cover integration between Adempiere's
But as well as Heng Sin, I'm wondering also why starting from zero?
And I'm asking why every development is seen as independent from the others?
Why not joining all those correlated projects?
Why not making a good enough architecture - it doesn't matter if the developed solution covers just 10% of the architecture?
Do we know the later cost of fixing a wrong architecture? A design flaw?
What's the difference in core concept between EDI and replication? Just the message format?
IMHO EDI is just another type of replication - and integration with OSCommerce is another type.
Even I see Import functionality a way to replicate, but it can be enhanced A LOT!
i.e.
- managing record types (currently just one record type can be managed)
- managing groups of records (this allow us to manage transactionality)
I made a complete solution for Oracle (unhappily PL/SQL & Forms based) to manage import/export of complex files (record types, transactionality), even this solution could manage the situation of receiving messages not from files.
------------------------
Joel:
> Doesn't replication imply that you need to keep two or more databases
> exactly synchronized?
There are lots of needs on replication:
- like BI - replicate summarized information to a read-only system
- for reporting purposes - replicate exact information to a read-only system
- like integration with OSCommerce - it can be seen as a replication because the final objective is to have the same info in both systems although different
- like integration with other businesses via EDI - same as previous
- like integration of POS - is a two-way replication but every table can be seen as one-way
- there is another more complex scenario (merging) where the same table is updated in both systems - it needs management of collisions (manual or automatic)
- here the list continues with a long etcetera
> 'Defining the replication in the AD' seems like a huge risk. Any small
> detail missed in a manual setup could corrupt the other databases.
I think if you need flexibility the only way is adding some level of complexity like this.
Integration between systems is a consistent headache :-)
For such needs the industry invented: interfaces - EDI - EAI - WebServices - SOA ... another etcetera here
------------------------
In other words, what I'm trying to say is -> replication is a big thing, is structural, architectural.
It must not be seen as a solution for every need, that's the old way of constructing interfaces between systems.
It will work, but none solution will serve for a new need.
Maybe "sponsored development" is not the best way to develop a GOOD SOLUTION for ADempiere.
Possibly it can be a good solution for a single need, but I doubt it can be a good solution for the whole problem.
AS ALWAYS, I CAN'T AND I DON'T WANT TO STOP ANY WAY OF WORK HERE, ANY TRIAL OF INNOVATION.
AS ALWAYS, I'M JUST WARNING AND ADVISING.
Regards,
Carlos Ruiz
Hi Carlos, Joel, Mario,
>Doesn't replication imply that you need to keep two or more databases exactly
>synchronized?
Not necessary exactly. In some cases user wants only sales data or Products data synchronized. Depending on what they want to achieve. When you have multiple stores in differente locations you need to have products replication from central instance to local instances and sales data replicated from local to central. You do not need all sales data replicated to all instances.
>But as well as Heng Sin, I'm wondering also why starting from zero?
I think that this what i have done is just begginig part. I do not think that it is from zero. I do not see any Export/Import functionlaity in Admepiere which can work for all tables or which is configuirable or export functionlait which is triggered on each new record creation. This is what i have done till now na di think that it gives some new functionality which can be good for other integration projects, like oScommerce. Exporting records on some trigger is functionality which other systems have and which makes system capable to talk with externals systems and whihc i think is good and valuable.
Till now i made posibility to Export/Import records from Adempiere, moreorver i made posibility user to define structure of this exported record.
Now i have to define interface which could have different implementations for different transport mechanisms (JMS, http, file storage,..)
I think that this is the first REQUIRED CONDITION for any development in replication area. I do not know how to make replication without to be able to Export/Import. So from now on there are many possibilities.
Just one thing. Very important one. I MUST deliver working version of EDI/Replication/what ever i do...
Joel had reall client who needs EDI. Victor has reall client too. I do not have 6 months to spend in architectrure of big system lso noone can guarantee that such big thing would work there are risks in any development project. I preffer to make working application and after this on second stage to improve. Please do not push me to make big desing. Also i asked concrete question, unfortunately i do not get suggestion or help, just proposals till now. They are very good, but at the end of the day i have to make the change for which i asked. I feel a bit sorry as i have to spend lot of time to read forum.
Also mixing EDI and Replication at early stage is dangerous. EDI had to be delivered prior replication. EDI must work for Joel client and Replication must work for Victor. I can't put EDI on risk because of Replication. If i try to make big design now i think i will fail. That's why i preffer to make working version of EDI and working version of Replication independent. Also please note that there are companies that build applications for transforming XML to EDI and EDI to XML and this are big applications, which for me means that it is more work if i want to dovelop general design or if i want ot intergate such application with Adempiere.
I completely agree with you Calros, but there are specific conditions under which we work and which force us to do what is necessary to do. Nothing more or nothing less.
Kind regards,
Trifon
> I do not know how to make replication without to be able to Export/Import.
There are many other approaches i.e. WebServices, RPC
The Export/Import approach is just one type of, and what I found incorrect in the Export/Import approach is that is mixing the data, message and transport to only one solution.
That's why I'm saying it must be seen as layers:
- Layer of data saving for replication (AD_ChangeLog?)
- Layer of construction of the message (read the change log and construct the message)
*** Types of messages: XML file, EDI frame ... etcetera
- Layer of transport of the message
*** transfer a file via ftp? calling a Webservice? sending an e-mail?
If you focus just on export/import approach you're mixing the layers and constructing a solution that can't work easily with other needs.
That's exactly what's happening, replication can't exploit EDI development.
> I preffer to make working application and after this on second stage to improve.
> Please do not push me to make big desing.
I can't push you to make a big design. I just can advice contractors to ask for that.
And as you pointed mostly there are dependencies on time to deliver.
But in my long experience working for many companies - is a mistake to deliver a badly-designed working solution.
Obviously is a decision of the contractor - just pointing AGAIN that if contractor put the wrong conditions it will be difficult the development reach the trunk - because it will finish being a very narrow solution - almost like a customization.
> Also i asked concrete question, unfortunately
> i do not get suggestion or help, just proposals till now.
I think the answer for your concrete question is precisely what we're discussing here.
Developing a solution just thinking on export/import an XML file is too narrow, I'll recommend to avoid such thinking when changing the core (i.e. PO.java)
If this is being developed as a specific customization, then I agree to make it in ModelValidator as you proposed initially. And we must face reality, the design will be narrow because we need speed of development, and possibly it won't reach trunk.
To change PO.java (very important core class) I think a better design is needed.
CAUTION: THIS IS JUST MY OPINION - I'M NOT PUSHING NOTHING HERE - AGAIN I'M JUST ADVISING.
If others in this project think different we can discuss and open spaces for policies and rules.
> If i try to make big design now i think i will fail.
I don't think so, my advice is you always need to think on big design.
Imagine what would be Compiere/Adempiere is JJ don't think big design every time?
What I admire from JJ is precisely he thinks big design even when he's contracted to make small things.
Collaboration best practices: Think big to act small
http://www.silicon.com/research/specialreports/collaboration/0,3800003020,39121010,00.htm
Regards,
Carlos Ruiz
Hi Carlos,
>There are many other approaches i.e. WebServices, RPC
>The Export/Import approach is just one type of, and what I found incorrect in the Export/Import approach >is that is mixing the data, message and transport to only one solution.
>
>That's why I'm saying it must be seen as layers:
>- Layer of data saving for replication (AD_ChangeLog?)
>- Layer of construction of the message (read the change log and construct the message)
>*** Types of messages: XML file, EDI frame ... etcetera
>- Layer of transport of the message
>*** transfer a file via ftp? calling a Webservice? sending an e-mail?
I would like to point that i can construct XML message from AD_ChangeLog i do not say that i will not do it. AD_ChangeLog has information for contstructing XML message. I just say that for initial Prof of Concept it will be overkill to fix AD_ChangeLog functionality.
>The Export/Import approach is just one type of, and what I found incorrect in the Export/Import approach >is that is mixing the data, message and transport to only one solution.
Why do you think so?
I do not say that i will make all in one solution.
For EDI i made liers. I think that i can make liers for Replication too.
>Developing a solution just thinking on export/import an XML file is too narrow,
>I'll recommend to avoid such thinking when changing the core (i.e. PO.java)
I think that even with AD_ChangeLog PO have to be changed as it will trigger new event that NEW record is created in the system. This new event will create new record in AD_ChangeLog which will lead to export and we will get cycle.
OK. We can disable making new record in AD_CahngeLog, but what happens if someone listen for new records in the system and send SMS message for exmaple. I think that this listener it must distinguish between record created by user adn record created by Replication process. It is just additional notification type. I think that it is addition to the system not change.
Please advice how to avoid changing PO and i will implement it.
Kind regards,
Trifon
Hi Trifon,
> For EDI i made liers. I think that i can make liers for Replication too.
I would recommend instead of creating new tiers, try to reuse the same.
In fact instead of creating new tables to describe the EDI format (C_EDIFormat/ C_EDIFormat_Line/ C_EDIFormat_LineElement) I would preferred to extend AD_ImpFormat/ AD_ImpFormatRow functionality.
-------------
NOTE ASIDE:
Maybe we're finding a possible problem in "Sponsored Development" approach.
Asking to Sponsors: Can we make some sort of "Sponsored Design" also?
We could make brainstorms, meetings, forum discussions, wiki discussions, etc?
And share the "sponsored design" money between people participating in design according to the participation?
Please don't take it wrong.
I'm just throwing ideas trying to get more community wisdom to sponsored work.
-------------
>> The Export/Import approach is just one type of,
>> and what I found incorrect in the Export/Import
>> approach is that is mixing the data, message
>> and transport to only one solution.
> Why do you think so?
Because the Export/Import functionality you're proposing is pointing to "trigger" the export from PO (data saving) and it implies just one format of the message (XML format of ADCK) and it implies just one way of transport (sending or replicating a XML file).
I would prefer a solution where:
* the data saving is generic (like AD_ChangeLog or AD_ReplicationLog).
* and the definition of the tables being "saved" are flexible (like AD_ReplicationTable?)
* and the definition of the message format is centralized and flexible (like extending AD_ImpFormat/ AD_ImpFormatRow?)
* and the definition of the transport of the message is centralized and flexible (like extending AD_Replication or extending C_EDIProcessorType)
-------------
> I think that even with AD_ChangeLog PO have to be changed as
> it will trigger new event that NEW record is created in the
> system. This new event will create new record in AD_ChangeLog
> which will lead to export and we will get cycle.
> Please advice how to avoid changing PO and i will implement it.
It's hard Trifon, I think solution must be thought looking at the complete picture.
For your specific issue related to a possible loop in ModelValidator, I have two opinions:
1 - There is no problem, first time the ModelValidator will be called for the table (i.e. C_BPartner), second time the ModelValidator will be called for AD_ChangeLog, no loop in this case.
2 - If you find an scenario (very possible) where the ModelValidator is called in a loop, the easiest solution could be to implement an approach like the isCalloutActive/setCalloutActive.
Regards,
Carlos Ruiz
Hi Carlos,
i would like once again to thank you for participating in this disussion as it clarify things.
>I would recommend instead of creating new tiers, try to reuse the same.
>In fact instead of creating new tables to describe the EDI format (C_EDIFormat/
>C_EDIFormat_Line/ C_EDIFormat_LineElement) I would preferred to extend
> AD_ImpFormat/ AD_ImpFormatRow functionality.
I think that this is not possible easely. I do not say not completely possible but...
Let me explain.
--AD_ImportFormat and AD_ImportFormatRow are intended to allow user to define import format, but can't help in export.
--Also they are found to I_Xxxx tables.
--Existing Import in Adempiere is two step process. For EDI we need one step process.
EDI really needs 3 tables. Initial desing was with 2 but i had to add 3-rd one.
>Because the Export/Import functionality you're proposing is pointing to "trigger"
>the export from PO (data saving) and it implies just one format of the message
>(XML format of ADCK) and it implies just one way of transport (sending or
> replicating a XML file).
Let's forget about ADCK for Replication(Even if it is better tool :) ). Replication will completely replace it.
Format of XML message can be defined by user(EXP_Format and EXP_FormatLine in Replication project. It can contain primary keys or whatever XML sturcture user needs). I choosed XMl format as it can be transformed.
As i pointed in wiki i see three ways:
ModelValidator
AD_CahngeLog
Workflow
>I would prefer a solution where:
>
>* the data saving is generic (like AD_ChangeLog or AD_ReplicationLog).
>* and the definition of the tables being "saved" are flexible (like AD_ReplicationTable?)
>* and the definition of the message format is centralized and flexible (like extending AD_ImpFormat/ >AD_ImpFormatRow?)
>* and the definition of the transport of the message is centralized and flexible
>(like extending AD_Replication or extending C_EDIProcessorType)
Yes, i like this plan(but do not see AD_CahngeLog as the only way). And i think that it is possible to be done.
As i pointed till now i made just way to define Format of message.
>* and the definition of the message format is centralized and flexible (like extending AD_ImpFormat/ >AD_ImpFormatRow?)
Sorry, but i find AD_ImpFormat and AD_ImpFormatRow not good for this case.
1. They can't describe tree structure.
2. Also they are intended to import data from plain text files which are not tree oriented also.
3. Also if i change them i will broke existing functionality. I think that existing Import functionality is good and flexible for plain files that's why i do not want to change it.
I would like to point that i created new tables EXP_format and EXP_FormatRow.
If we find way to merge AD_ImpFormat and EXP_Format than - no porlbem i agree.
>For your specific issue related to a possible loop in ModelValidator, I have two opinions:
>
>1 - There is no problem, first time the ModelValidator will be called for the table (i.e. C_BPartner), >second time the ModelValidator will be called for AD_ChangeLog, no loop in this case.
I do not agree.
1. First time CODE(ModelValidator or whaterver) will be called when user create record.
2. AD_CahngeLog is Created.
3. Process read AD_ChangeLog and send message to external system that new record is created.
4. External system read message and creates new record into remote instance.
5. External system save message into AD_ChangeLog.
6. External system construct export mesage from AD_ChangeLog.
7. Local system get message from remote system that new record is created... And we get cycle.
How to avoid it?
>
>2 - If you find an scenario (very possible) where the ModelValidator is called in a loop, the easiest >solution could be to implement an approach like the isCalloutActive/setCalloutActive.
What i made is to add new property into PO and set it to true when saving Replicated record. But i also need new ModelValidator Event as i would like to distinguish if record is from replication or not. I think that such addiotna is good one as allos to build system which can notify upon creation of replicated record.
Once again thank's for the discussion.
Kind regards,
Trifon
Trifon, I didn't say that AD_ImpFormat and AD_ImpFormatRow are ready to be used for export or for formatting messages.
What I'm trying to say is that we must try to use the same objects.
Now we have AD_Imp... objects, C_EDI... objects, and now AD_Exp... objects, all of the three used to describe formats of records/messages/frames/whatever.
What I mean is we must not create new tables for every need but trying to extend current functionality.
Look the original design of Adempiere, the same tables are used for Sales and Purchase orders, the same tables are used for Customer and Vendor Invoices, the same tables are used for Receipts and Payments.
I'm worried if we start creating new tables for every need even when some similar functionality exists.
What we must ask ourselves is: how AD_ImpFormat and AD_ImpFormatRow can be extended to manage better formatting of messages (not just files):
- possibly we will need a third table
- possibly we will need to independize the AD_ImpFormat from I_ table and make it more flexible (preserving compatibility with old import processes)
----------------------------
> I do not agree.
> 1. First time CODE(ModelValidator or whaterver) will be called when user create
> record.
> 2. AD_CahngeLog is Created.
> 3. Process read AD_ChangeLog and send message to external system that new record
> is created.
> 4. External system read message and creates new record into remote instance.
> 5. External system save message into AD_ChangeLog.
> 6. External system construct export mesage from AD_ChangeLog.
> 7. Local system get message from remote system that new record is created...
> And we get cycle.
Trifon, I'm still seeing you're mixing the saving of the data with the delivery of the message.
You must not deliver a message within an incomplete transaction (it can be rolled back).
You need to see the saving of the data independent of the sending of the message.
Data Saving:
Saving of the data -> possibly AD_ChangeLog or AD_ReplicationLog or whatever (not writing XML files).
Saving of the data must occurs ALWAYS and if the data can't be saved the process must be rolled back.
---------------------------------------
Message Formatting and Message Sending:
---------------------------------------The message formatting and sending must occurs in a later step (preferrably defined in a flexible way).
It can be at commit or it can be in a process running every n minutes.
Message Formatting:
The format of the message can be defined in a flexible Message/Record/Field definition (like EDI, Import or export formatting).
In AD_Imp there is a trial to manage transformation via Callouts.
We could make a generic formatting solution based on those tables.
Message Sending:
The sending of the message(s) is different - we need to define the frequency, the transport, and the way to manage confirmation receipt
Frequency:
- It can be immediate and mandatory, meaning the transaction must be rolled back if the message can't be sent and confirmation received. This implies a big problem in replication systems - some databases solved via two-phase commit protocol - http://en.wikipedia.org/wiki/Two-phase_commit.
But possibly this is out of scope of initial solution.
So I think implementing immediate and mandatory messaging must be avoided at this moment, but we can design the structures to support it in future.
- Frequency can be also every n minutes, or hours, or days, or maybe every n messages.
And we could have different frequencies for different targets.
- It can also be manual pushing a button - as you made in EDI
Transport:
- Here we can have several ways of transport and the data needed for the transport, i.e:
* ftp - we will need target host, directory, user and password
* e-mail - we will need just a target e-mail
* file - we will need the directory and some naming convention
* database - a possibility is the message format is the plain SQL, then the transport will need the target database connection definition
* etc
I'm not saying you must implement all these transport methods, just define them and allow a simple way to extend them.
And implement just the way currently needed and agreed with Joel or Victor.
---------------------
Confirmation Receipt:
---------------------
This is harder, we need to define how to act on confirmation of the target system.
If we must wait, if we must resend, if we must ignore. There are many possible scenarios also.
i.e.
* we can ignore a failed replication that replicates the whole table.
* we CAN'T ignore a failed replication that represents transactional data, in fact the rest of the replication must be stopped.
The target system must implement the sending of the confirmation. This can be like defining a replication from in the opposite way.
-------------------------
JOKING: I'm thinking how much to charge for this design brainstorm ;-)
Regards,
Carlos Ruiz
Hi everyone,
I would like to write more on what i think regarding replication and other integratin projects which are comming.
For replication and generaly for intergation every application needs Import / Export functionality.
In Admepiere we have some Import / Export functionality but it is not generally applicable to all tables in the system. That's why my first task in Replication project was to make general Import / Export functionality in Adempiere. It is not tied to Replication or any other project. It is just functionality which allows user of Adempiere to define format of XML document(it can can tree structure). Please refer to example in the Replication project:
http://www.adempiere.com/wiki/index.php/Sponsored_Development:_Replication#Example
I would like to say that i'm glad that i managed to do such general Import / Export functionality as it will allow further integration with external system possible task. Also i would like to poin that i managed to do this only because we have Application dictionary( it is my big friend which helps me in all tasks).
Alos i would like to say that this Export format is completely managed by AD(Just a point to Mario that we have to define all in AD - I completely agree Mario)
Strange thing in the case is that i found that i can use Export format as Import format too. Probably it is not so strange but it was a good surpise for me. I was prepared to create two tables.
Currently i can say that this genral Import / Export functionality works. I have to finish one thing in Import but it is just matter of some hours to be done.
Now to get back to concrete project - Replication. After we have Export / Import functionality we must find way to transfer CHANGE message to external system. My plan is to use JMS as proven technology which guarantee transfer of messages. I have to show to Victor a proof of concept that such replication can be done with the choosen techonolgy. For this task by plan i had 2 weeks. After that we have to continue or not with further development.
Till now i had to toch core system only once when i had to change ModelValidator and PO in order to avoid cycle issue i faced. I think that this is general issue. For any system where listeners/triggers can be defined a special care must be taken in order to avoid cycles. The same cycle will hapen if i define triger which creates new record in the same tables where triger is attached.
So this is status of Replication -> i must make JMS sending and recept.
At second stage we will have to go deeply into details:
-- Existing infrasructure
-- Different Replication strategies.
-- ...
-- all other detials pointed by Calros, Hengsin, Mario, Joel and Armen
Kind regards,
Trifon
Hello Trifon,
Thanks for this clarification. Interestingly, it helps to clarify the separation that Carlos outlines here (https://sourceforge.net/forum/message.php?msg_id=4450603). Since they were posted a couple minutes apart it doesn't seem yours was a response... Great minds do think alike!
It appears you are already separating the transport piece.
- Layer of data saving for replication (AD_ChangeLog?)
- Layer of construction of the message (read the change log and construct the
message)
- Layer of transport of the message
There are some other issues that need to be separated in order to understand the big picture, namely Replication and Integration.
Replication implies the matching and synching of two ADempiere Systems. The requirements are unique here, because you are not creating ADempiere documents, but sharing existing ones.
Critical issues to manage:
* ID generation and resolution.
* Sharing changes to AD
* Business rules were applied in the creation of the data, so it is a data synch.
* Replication would presumably be 'automoted'
* Again I would assert you can't pick and choose items to replicate. It is not 'loosely-coupled'. Each system has to have the same AD, the same BP's, the same users, etc etc.
Integration implies sharing data between ADempiere and a non-adempiere system. The requirements are different, because an incoming records create a new ADempiere document.
Critical issues to manage:
* Incoming data needs to apply the business rules, therefore, must be processed through the model.
* The model can handle the id generations naturally.
* Data transformation belongs outside ADempiere, because it will be custom depending on the source system and whether it is customized. Ultimately, it would be nice to have some 'connectors' pre-configured for things like OSCommerce. That's what makes that a different project.
* Integrations are not necessarily automatic.
Time may prove this wrong, but I'm not convinced that the EDI requirements are or could have been reusable for either Replication or Integration. EDI mapping is a complicated process in itself, and the EDI tools in ADempiere are designed to simplify that from the AD user standpoint.
Maybe what trifon has begun with his replication Import/Export is the start of a generic Interface that can be used for the integrations as well?
Some things that would make this useful:
* If the data is ETL'd from the integrated system into this interface, it can be run through the model and thus put into ADempiere.
* For outgoing integrations, a standard interface that serves as the starting point for ETL's to other systems.
Then plug-ins could be developed for systems like OSCommerce or QuickBooks. These could be re-usable plug-ins, and could be used for initial data imports as well.
Hope there was something useful in those musings...
Regards,
Joel Stangeland
Hi Joel,
>Thanks for this clarification. Interestingly, it helps to clarify the separation that Carlos outlines here
You welcome. I would like to thank you for entering this discusison as Implementation Providers and End users are reall users of the functionality developers make.
>Time may prove this wrong, but I'm not convinced that the EDI requirements are or could have been >reusable for either Replication or Integration. EDI mapping is a complicated process in itself, and the >EDI tools in ADempiere are designed to simplify that from the AD user standpoint.
I hope that EDI mapping is simple from user point of view. You will be the first company that will implement it for customer. I still must develop some parts of EDI, and i hope to do it this week.
>Maybe what trifon has begun with his replication Import/Export is the start of a generic Interface that >can be used for the integrations as well?
Yes, i hope so. Atleast this could allow Administrator to setup triger/notification mechanism which exports custom defined XML message upon record creation. I think something which would be good is custom creation upon Document creation. I hope that this Import/Exoprt functionality will be the starting point and will enable Adempiere to talk with external system.
>Some things that would make this useful:
>* If the data is ETL'd from the integrated system into this interface, it can be run through the model >and thus put into ADempiere.
>* For outgoing integrations, a standard interface that serves as the starting point for ETL's to other >systems.
Yes. When XML(now, or any other in future) message is received importer must find proper model and use it for saving of the record. I do not plan to use plain SQL Updates or Inserts. I even do not hink for such thing.
As a starting point for outgoing intergation i'm not sure what to write. But i think that EAI tools can be configuired to react on message receipt and perfrom actions.
Kind regards,
Trifon
Hi Carlos, Trifon,
It might be worthwhile to review Jeff Davis concepts for integration here:
http://jeff-davis.blogspot.com/2007/05/adempiere-integration-framework-part-1.html
The idea of using the ESB was to handle the conversion to the message format externally to ADempiere. The interface in and out of ADempiere could be consistent, and post to a JMS queue.
But then the choice of tools and formats for transport and messaging would not have to be re-coded.
Anyway, just to keep the creative juices flowing....
Joel Stangeland
www.idalica.com