This message is for all of the sourceforge users how have downloaded the DataJuncture DataImporter source files
Please send me your commits! In order to improve the product I need to know:
1) How you are using the product. (What type of jobs, etc.)
2) Likes and dislikes.
3) Functionality requests.
4) Other helpful commits.
Thanks
drolem at users.sourceforge.net
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If I were to use this, it would be for flat file input and output. Since output doesn't appear to be a part of the project, that could be a problem ;-)
I'm not sure how far you want to go down this path. At the high end there a numerous products that do what you are doing plus lots more. Biztalk, Merant etc.. On the other hand, I have been working on something similar in some ways. Here are some comments that hopefully you will find helpful.
Your code is too tied to the user interface. My goal has been that the tool can be used from a gui like yours, from the web for simpler mappings and from code where I need to do more complicated mappings not possible via the ui. This is something none of the vendor products can do now.
I found that separating the idea of mapping data from a file to a target source such as a dataset, is a completely separate concern than parsing the source data. While addressing some of the points given later, this became very important.
Defaulting data is a fairly important and challenging task. you have formulas in your api but I found that having one shot to correct or adjust missing or badly formatted data was not enough. Since I would like an api that non-developers could use, formulas are especially troublesome to me.
There are many flat file formats that are not tabular data. Cobol data files, EDI, many financial transactions, legacy systems to name just a few. Many of the flat files I deal with have multiple record types, identified by data in the first column or by an external definintion. In addition, many of these files use multiple record types to put together a single "record". So for user joe you might have an address record, an name record, a transaction record etc.. All these map to a single datarow. Not sure how this project could be extended to support such a thing.
Good luck and I'll check back from time to time to see how it's going.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
> If I were to use this, it would be for flat file input and output. Since output doesn't appear to be a part of the project, that could be a problem ;-)
The first version did not have text output, but the current version has CDataTargetTextFileDelimited.vb. This plugin can be used to generate CSV and Fixed Length flat files. Please try it and let me know what you think.
> Your code is too tied to the user interface. My goal has been that the tool can be used from a gui like yours, from the web for simpler mappings and from code where I need to do more complicated mappings not possible via the ui. This is something none of the vendor products can do now.
> I found that separating the idea of mapping data from a file to a target source such as a dataset, is a completely separate concern than parsing the source data. While addressing some of the points given later, this became very important.
Ya, tying business logic to the Windows UI was one of the early design decisions. The idea is the class could be created as an object without the form being loaded or load the form to view the default settings. The plugin/class would still support advanced features the form would not be able to display. (The form is tied to the default settings only.) Also, a web service could create a new UI with extended features. I think what the classes need is: 1) Use more class events, like New and not Form_Load. 2) Do not use the form controls to store data. I need another code pass to ensure these two rules.
> Defaulting data is a fairly important and challenging task. you have formulas in your api but I found that having one shot to correct or adjust missing or badly formatted data was not enough. Since I would like an api that non-developers could use, formulas are especially troublesome to me.
Agreed, I would like to provide more formatting/default options so basic operations will not require a formula. I dont know if you have tested formulas yet but they are very slooooow.
> There are many flat file formats that are not tabular data. Cobol data files, EDI, many financial transactions, legacy systems to name just a few. Many of the flat files I deal with have multiple record types, identified by data in the first column or by an external definintion. In addition, many of these files use multiple record types to put together a single "record". So for user joe you might have an address record, an name record, a transaction record etc.. All these map to a single datarow. Not sure how this project could be extended to support such a thing.
I worked with Cobol in college and I know exactly what you are talking about. Row headers like 00, 10, 15, 20, etc. I am working on a multiple table insert from one flat file. It would be easy to create a clone of this plugin and add row identifiers as a method of choosing which table gets the row. EDI on the other had, aaahhh, why can we all just use XML ;-) Seriously EDI is a long-term goal, but I will need help from other open source developers.
> Good luck and I'll check back from time to time to see how it's going.
Thanks
Cya
Robert
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hey. I was surprised to find this project because it is very similar to a project I just registered also. sourceforge.net/projects/datamonkey This project is also written in VB and is about 4 years old. Though I haven't really worked on it over the last 3 years. One difference is the data I wrote DataMonkey to deal with was not simple flat file data, but text data in complex formats. Though I do offer limited support for database/sql/custom objects whatever.
There are other products that do this yes, but I am only aware of one that is well known and powerful (DataJunction) and it is pretty damned expensive. I mean, why have Linux when windows is already available? (Not that I am a Linux bigot) Kind of a dumb question I think.
Datamonkey is also tied to the UI, but loosely. I have had plans to decouple the engine and interface and think it could be done in a day or two. The UI really only defines the project files and kicks things off.
Anyway, I am looking for a project admin if anyone is interested.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So you guys aren't using VB6? What are .VB files anyway?
I recently added some documentation for DataMonkey which you will probably need. The sad thing is that when I got to the end of the example I realized DataMonkey wouldn't produce the output that I stated it would at the beginning of the example :(
I guess I forgot that I hadn't done much (anything) with the text output options...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2002-11-10
Great project.
I have worked on many projects that do this kind of thing. I would like to see transaction base processing on the target side, meaning when a insert fails the entire job fails. I would also like to see a MS SQL .sql or Oracle .sql file target export. I found that having the raw sql output when your target is a database is very very helpful, especially if you have novice users trying to put text data into a number database filed.
I also agree with some of the other comments in regarding non-tabular file formats. It would also be nice to link conversions together... like converting a weird format to tabular data then to a database.
I'm interested in lending a hand; I have .net experience and many many years of c++, com, ado, web, and vb.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello All,
This message is for all of the sourceforge users how have downloaded the DataJuncture DataImporter source files
Please send me your commits! In order to improve the product I need to know:
1) How you are using the product. (What type of jobs, etc.)
2) Likes and dislikes.
3) Functionality requests.
4) Other helpful commits.
Thanks
drolem at users.sourceforge.net
If I were to use this, it would be for flat file input and output. Since output doesn't appear to be a part of the project, that could be a problem ;-)
I'm not sure how far you want to go down this path. At the high end there a numerous products that do what you are doing plus lots more. Biztalk, Merant etc.. On the other hand, I have been working on something similar in some ways. Here are some comments that hopefully you will find helpful.
Your code is too tied to the user interface. My goal has been that the tool can be used from a gui like yours, from the web for simpler mappings and from code where I need to do more complicated mappings not possible via the ui. This is something none of the vendor products can do now.
I found that separating the idea of mapping data from a file to a target source such as a dataset, is a completely separate concern than parsing the source data. While addressing some of the points given later, this became very important.
Defaulting data is a fairly important and challenging task. you have formulas in your api but I found that having one shot to correct or adjust missing or badly formatted data was not enough. Since I would like an api that non-developers could use, formulas are especially troublesome to me.
There are many flat file formats that are not tabular data. Cobol data files, EDI, many financial transactions, legacy systems to name just a few. Many of the flat files I deal with have multiple record types, identified by data in the first column or by an external definintion. In addition, many of these files use multiple record types to put together a single "record". So for user joe you might have an address record, an name record, a transaction record etc.. All these map to a single datarow. Not sure how this project could be extended to support such a thing.
Good luck and I'll check back from time to time to see how it's going.
Thanks for the feedback :)
> If I were to use this, it would be for flat file input and output. Since output doesn't appear to be a part of the project, that could be a problem ;-)
The first version did not have text output, but the current version has CDataTargetTextFileDelimited.vb. This plugin can be used to generate CSV and Fixed Length flat files. Please try it and let me know what you think.
> Your code is too tied to the user interface. My goal has been that the tool can be used from a gui like yours, from the web for simpler mappings and from code where I need to do more complicated mappings not possible via the ui. This is something none of the vendor products can do now.
> I found that separating the idea of mapping data from a file to a target source such as a dataset, is a completely separate concern than parsing the source data. While addressing some of the points given later, this became very important.
Ya, tying business logic to the Windows UI was one of the early design decisions. The idea is the class could be created as an object without the form being loaded or load the form to view the default settings. The plugin/class would still support advanced features the form would not be able to display. (The form is tied to the default settings only.) Also, a web service could create a new UI with extended features. I think what the classes need is: 1) Use more class events, like New and not Form_Load. 2) Do not use the form controls to store data. I need another code pass to ensure these two rules.
> Defaulting data is a fairly important and challenging task. you have formulas in your api but I found that having one shot to correct or adjust missing or badly formatted data was not enough. Since I would like an api that non-developers could use, formulas are especially troublesome to me.
Agreed, I would like to provide more formatting/default options so basic operations will not require a formula. I dont know if you have tested formulas yet but they are very slooooow.
> There are many flat file formats that are not tabular data. Cobol data files, EDI, many financial transactions, legacy systems to name just a few. Many of the flat files I deal with have multiple record types, identified by data in the first column or by an external definintion. In addition, many of these files use multiple record types to put together a single "record". So for user joe you might have an address record, an name record, a transaction record etc.. All these map to a single datarow. Not sure how this project could be extended to support such a thing.
I worked with Cobol in college and I know exactly what you are talking about. Row headers like 00, 10, 15, 20, etc. I am working on a multiple table insert from one flat file. It would be easy to create a clone of this plugin and add row identifiers as a method of choosing which table gets the row. EDI on the other had, aaahhh, why can we all just use XML ;-) Seriously EDI is a long-term goal, but I will need help from other open source developers.
> Good luck and I'll check back from time to time to see how it's going.
Thanks
Cya
Robert
Hey. I was surprised to find this project because it is very similar to a project I just registered also. sourceforge.net/projects/datamonkey This project is also written in VB and is about 4 years old. Though I haven't really worked on it over the last 3 years. One difference is the data I wrote DataMonkey to deal with was not simple flat file data, but text data in complex formats. Though I do offer limited support for database/sql/custom objects whatever.
There are other products that do this yes, but I am only aware of one that is well known and powerful (DataJunction) and it is pretty damned expensive. I mean, why have Linux when windows is already available? (Not that I am a Linux bigot) Kind of a dumb question I think.
Datamonkey is also tied to the UI, but loosely. I have had plans to decouple the engine and interface and think it could be done in a day or two. The UI really only defines the project files and kicks things off.
Anyway, I am looking for a project admin if anyone is interested.
Thanks for your post. I looked at DataMonkey and it looks coolI need to reinstall VB6 so I can play-around with the source. :)
So you guys aren't using VB6? What are .VB files anyway?
I recently added some documentation for DataMonkey which you will probably need. The sad thing is that when I got to the end of the example I realized DataMonkey wouldn't produce the output that I stated it would at the beginning of the example :(
I guess I forgot that I hadn't done much (anything) with the text output options...
Great project.
I have worked on many projects that do this kind of thing. I would like to see transaction base processing on the target side, meaning when a insert fails the entire job fails. I would also like to see a MS SQL .sql or Oracle .sql file target export. I found that having the raw sql output when your target is a database is very very helpful, especially if you have novice users trying to put text data into a number database filed.
I also agree with some of the other comments in regarding non-tabular file formats. It would also be nice to link conversions together... like converting a weird format to tabular data then to a database.
I'm interested in lending a hand; I have .net experience and many many years of c++, com, ado, web, and vb.