Re: [Karmatool-users] EXT :Re: Adding workflow

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Michael,

The timestep in Karma refers to the logical timestamp in distributed
system. It's optional.

There is ingest schema and query schema available in the service-core
package,
config/karma_ingest_type_schema.xsd
config/karma_query_schema.xsd

I see several problems in the notification xml files:
1) The workflowID in all notification should be the same.
2) The serviceID in your case should be Process1's ID. So all the
serviceIDs in serviceInvoked1.xml, dataProduced1.xml, and dataConsumed1.xml
should match.

After modifying and ingesting the notification, could you please check the
raw_notification and exe_entity tables if the new entries are actually
stored? You may want to give a new workflowID for clean ingest.

Could you send me the karma log file if the problem remains?

Thanks,

Yuan

On Fri, Apr 19, 2013 at 5:19 PM, Black, Michael (IS) <Mic...@ng...
> wrote:

>  One thing that would help would be a simple example with accompanying
> explanation of what is what.
>
> There are so many IDs flying around inside the XML I don't know who refers
> to what.  And what does "timestep" do?
>
> And the only behavior I get is no successful entry in Karma (or at least
> no retrievable OPM).
>
> I was able to do some simple renaming but I'm at a loss to make a trivlal
> example.
>
> Actor1 runs Process 1 which Consumes File1 and produces File2
>
> bin/sendNotification.sh config/karma.properties
> $CurDir/workflowInvoked.xml
> bin/sendNotification.sh config/karma.properties
> $CurDir/serviceInvoked1.xml
> bin/sendNotification.sh config/karma.properties  $CurDir/dataProduced1.xml
> bin/sendNotification.sh config/karma.properties  $CurDir/dataConsumed1.xml
>
>
> workflowInvoked:
> <ns:workflowInvoked xmlns:ns="http://www.dataandsearch.org/karma/2010/08/"
> xmlns
> :soapenv="http://schemas.xmlsoap.org/soap/envelope/">
>   <ns:invoker>
>     <ns:userInformation>
>       <ns:userDN>Michael_Black</ns:userDN>
>       <ns:type>PERSON</ns:type>
>       <ns:email>Mic...@ng...</ns:email>
>     </ns:userInformation>
>     <ns:type>USER</ns:type>
>   </ns:invoker>
>   <ns:invokee>
>     <ns:workflowInformation>
>       <ns:workflowID>NGC_20130419_003</ns:workflowID>
>       <ns:workflowNodeID>WFNode:WFNode1</ns:workflowNodeID>
>       <ns:timestep>1476332968</ns:timestep>
>     </ns:workflowInformation>
>     <ns:type>WORKFLOW</ns:type>
>   </ns:invokee>
>   <ns:invocationTime>2013-04-18T16:08:00.000-05:00</ns:invocationTime>
> </ns:workflowInvoked>
>
> serviceInvoked1
> <ns:serviceInvoked xmlns:ns="http://www.dataandsearch.org/karma/2010/08/"
> xmlns:
> soapenv="http://schemas.xmlsoap.org/soap/envelope/">
>   <ns:invoker>
>     <ns:workflowInformation>
>       <ns:workflowID>NGC_20130419_002</ns:workflowID>
>       <ns:workflowNodeID>WFNode:WFNode1</ns:workflowNodeID>
>       <ns:timestep>1476332968</ns:timestep>
>     </ns:workflowInformation>
>     <ns:type>WORKFLOW</ns:type>
>   </ns:invoker>
>   <ns:invokee>
>     <ns:serviceInformation>
>       <ns:workflowID>NGC_20130419_002</ns:workflowID>
>       <ns:workflowNodeID>WFNode:WFNode2</ns:workflowNodeID>
>       <ns:timestep>-356643003</ns:timestep>
>       <ns:serviceID>svc:TestService_-356643003</ns:serviceID>
>     </ns:serviceInformation>
>     <ns:type>SERVICE</ns:type>
>   </ns:invokee>
>   <ns:invocationTime>2013-04-18T16:10:00.0
>
> dataConsumed1
> <ns:dataConsumed xmlns:ns="http://www.dataandsearch.org/karma/2010/08/"
> xmlns:so
> apenv="http://schemas.xmlsoap.org/soap/envelope/">
>   <ns:action>CONSUME</ns:action>
>   <ns:actor>
>     <ns:serviceInformation>
>       <ns:workflowID>NGC_20130419_002</ns:workflowID>
>
> <ns:workflowNodeID>WFNode:KarmaProcessor-180041393</ns:workflowNodeID>
>       <ns:timestep>-180041393</ns:timestep>
>       <ns:serviceID>svc:TestService_-180041393</ns:serviceID>
>     </ns:serviceInformation>
>     <ns:type>SERVICE</ns:type>
>   </ns:actor>
>   <ns:dataObject>
>     <ns:file>
>       <ns:fileURI>file:File_-644433975</ns:fileURI>
>       <ns:ownerDN>-644433975</ns:ownerDN>
>       <ns:createDate>2013-02-01T14:49:23.550-05:00</ns:createDate>
>       <ns:size>868013951</ns:size>
>       <ns:objectName>testKarmaFile_-644433975.txt</ns:objectName>
>     </ns:file>
>   </ns:dataObject>
>   <ns:timestamp>2013-02-01T14:49:23.621-05:00</ns:timestamp>
>   <ns:dataRole>File Consumed</ns:dataRole>
>   <ns:annotations>
>     <ns:property>execution_environment</ns:property>
>     <ns:value>
>       <value>POSIX</value>
>     </ns:value>
>   </ns:annotations>
>
> <ns:dataProduced xmlns:ns="http://www.dataandsearch.org/karma/2010/08/"
> xmlns:so
> apenv="http://schemas.xmlsoap.org/soap/envelope/">
>   <ns:action>PRODUCE</ns:action>
>   <ns:actor>
>     <ns:serviceInformation>
>       <ns:workflowID>NGC_20130419_002</ns:workflowID>
>       <ns:workflowNodeID>WFNode:WFNode2</ns:workflowNodeID>
>       <ns:timestep>-356643003</ns:timestep>
>       <ns:serviceID>svc:TestService_-356643003</ns:serviceID>
>     </ns:serviceInformation>
>     <ns:type>SERVICE</ns:type>
>   </ns:actor>
>   <ns:dataObject>
>     <ns:file>
>       <ns:fileURI>file:File_-644433975</ns:fileURI>
>       <ns:ownerDN>-644433975</ns:ownerDN>
>       <ns:createDate>2013-02-01T14:49:23.550-05:00</ns:createDate>
>       <ns:size>868013951</ns:size>
>       <ns:objectName>testKarmaFile_-644433975.txt</ns:objectName>
>     </ns:file>
>   </ns:dataObject>
>   <ns:timestamp>2013-02-01T14:49:23.551-05:00</ns:timestamp>
>   <ns:annotations>
>     <ns:property>execution_environment</ns:property>
>     <ns:value>
>       <value>POSIX</value>
>     </ns:value>
>   </ns:annotations>
> </ns:dataProduced>
>
>
>
>
>  Michael D. Black
> Senior Scientist
> Analytics, Production and Services
> Advanced GEOINT Systems
> Northrop Grumman Information Systems
>    ------------------------------
> *From:* yu...@um... [yu...@um...] on behalf of Yuan Luo
> [yu...@in...]
> *Sent:* Friday, April 19, 2013 3:52 PM
> *To:* Black, Michael (IS)
> *Subject:* EXT :Re: [Karmatool-users] Adding workflow
>
>   Hi Michael,
>
>  Sure. Let me know if you still get error.
>
>  Yes. We don't have that kind of docs at this moment, but will bring this
> up in our internal meetings. For now, please follow that sample and let me
> know if you have any problem writing notifications.
>
>  Thanks,
>
>  Yuan
>
>
> On Fri, Apr 19, 2013 at 4:10 PM, Black, Michael (IS) <
> Mic...@ng...> wrote:
>
>>  Yes..you the path that I'm trying to modify.
>>
>> I think the error may have been on my part...I'm going to experiment here
>> and if I have problems I'll send the sample files and perhaps you can help
>> me figure out what I'm messing up if it doesn't work.
>>
>> One problem is I don't see any docs on how to actually enter a workflow
>> in the system...what minimums are expected or such.
>>
>>  Michael D. Black
>> Senior Scientist
>> Analytics, Production and Services
>> Advanced GEOINT Systems
>> Northrop Grumman Information Systems
>>    ------------------------------
>> *From:* Yuan Luo [yu...@in...]
>> *Sent:* Friday, April 19, 2013 1:38 PM
>> *To:* Michael Black
>> *Cc:* karmatool-users
>> *Subject:* EXT :Re: [Karmatool-users] Adding workflow
>>
>>   Hi Michael,
>>
>>  Does sample1 come from
>> client-core/samples/sampleWorkflow1/notification? Please see my inline
>> reply. I might have missed your points, so correct me if I misunderstood
>> anything.
>>
>>  Thanks,
>>
>>  Yuan
>>
>>
>> On Fri, Apr 19, 2013 at 1:01 PM, Michael Black <mdb...@ya...>wrote:
>>
>>>  I am able to load a workflow using the sample1 workflow recently added.
>>> ****
>>>
>>> ** **
>>>
>>> But…when I try and make changes to that workflow to experiment with
>>> different names and such the OPM file produced will contain the old names.
>>> ****
>>>
>>> I was trying just to slowly migrate over to what I want to put in the
>>> system by changing user name for example with a new workFlowID.  But the
>>> old user name still shows in the OPM file.****
>>>
>>> **
>>>
>> In Karma system, we only have add/append modes. A workflow will be added
>> into the system when ingesting the first notification that contains a new
>> workflowID. The following notifications with that workflowID will be
>> appended to that workflow. There's no deletion or modification for a
>> workflow. If you change a workflowID (I assume you changed all the
>> notifications with new workflowID in the sampleWorkflow1 directory), it
>> will be considered as a new workflow. Both the old and new workflow will
>> exist.
>>
>>
>>>  **
>>>
>>> And if I zap the tables and try to insert my modified workflow no OPM
>>> file gets produced at all.  So there appears to be some dependency that I
>>> don’t see.****
>>>
>>> **
>>>
>> There should be no problem to get new workflow ingested and queried after
>> zapping tables. Could you please provide more details?
>>
>>
>>>  **
>>>
>>> So I’m wondering if there’s some document somewhere I should read that
>>> explains this process as I’m completely befuddled.****
>>>
>>> **
>>>
>> The latest document is
>> http://iweb.dl.sourceforge.net/project/karmatool/v3.2.3/KarmaUserGuide.pdf
>>
>>  Some pointers of Karma and Provenance:
>> http://www.cs.indiana.edu/~plale/papers/Plale-FGCS-preprintJan2011.pdf
>>  http://www.cs.indiana.edu/~plale/papers/KarmaInfoModel_SWF09.pdf
>>  http://link.springer.com/chapter/10.1007%2F978-3-642-29931-5_3
>>
>>
>>>  **
>>>
>>> Is there some tool to generate a generic workflow?
>>>
>> Are you referring to a tool that can generate notifications? If yes,
>> KarmaAdaptor is one solution for that. The KarmaAdaptor takes a customized
>> rule file and apply to the workflow log files to get the provenance
>> information and ingest to Karma Server with notifications. The adaptor and
>> its doc is here http://sourceforge.net/projects/karmatool/files/v3.2.3.
>>  Or are you asking for these workflow tools such as Kepler
>> https://kepler-project.org/?
>> If it's something else, please let me know.
>>
>>>  ****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> Michael Black****
>>>
>>> ** **
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Precog is a next-generation analytics platform capable of advanced
>>> analytics on semi-structured data. The platform includes APIs for
>>> building
>>> apps and a phenomenal toolset for data science. Developers can use
>>> our toolset for easy data analysis & visualization. Get a free account!
>>> http://www2.precog.com/precogplatform/slashdotnewsletter
>>> _______________________________________________
>>> Karmatool-users mailing list
>>> Kar...@li...
>>> https://lists.sourceforge.net/lists/listinfo/karmatool-users
>>>
>>>
>>
>>
>>  --
>>
>> Yuan Luo
>> School of Informatics and Computing, Indiana University
>> Data to Insight Center, Indiana University
>> Cell: (812)272-0208   http://www.yuanluo.net
>>
>>
>
>
>  --
>
> Yuan Luo
> School of Informatics and Computing, Indiana University
> Data to Insight Center, Indiana University
> Cell: (812)272-0208   http://www.yuanluo.net
>
>

-- 

Yuan Luo
School of Informatics and Computing, Indiana University
Data to Insight Center, Indiana University
Cell: (812)272-0208   http://www.yuanluo.net