Documentation on usage of JavaFBP

Developers
Pani
2009-03-16
2013-05-23
  • Pani

    Pani - 2009-03-16

    Hi Paul,

    I am looking for documentation/samples on how to use JavaFBP APIs. I came across your project when I was reading about Java based DataFlow programming languages.
    I looked at pervasive's datarush API and looking for alternatives to them.

    Does JavaFBP API make use of Multi-core CPUs and help us do parallel programming in Java? If so, please point me to the documentation.

    Thanks in advance,
    Pani

     
    • J. Paul Morrison

      Yes, JavaFBP uses multiple Java Threads, so, depending on the JVM implementation, it should use multiple cores.  The book is on http://www.jpaulmorrison.com/fbp (about half-way down) - and the documentation for JavaFBP is at http://www.jpaulmorrison.com/fbp/jsyntax.htm

      Feel free to email me if you have questions.

      Regards,

      Paul M.
      paul dot morrison at rogers dot com

       
      • Pani

        Pani - 2009-03-17

        Thank you very much Paul. I will keep you posted on my progress.

        Thanks,
        Pani

         
        • Pani

          Pani - 2009-07-01

          Hi Paul,

          I finally got a chance to use JavaFBP to do few experiments - to see if architecting the solution using FBP would help me process data faster. It is amazing how a 182KB jar file could so much. I have few quick questions.

          How do we control the tracing feature ?
          Was wondering how we could switch off tracing and if there are any tracing-levels defined somewhere.

          I am using JavaFBP-2.3.jar  and DrawFBP-2.3.jar.
          Would greatly appreciate your help here.

          Thanks in advance,
          Pani

           
          • J. Paul Morrison

            Hi Pani,

            JavaFBP is up to 2.4 - I'm not sure when the tracing was upgraded, but 2.4 generates a single trace file for the network, and a separate one for each subnet  (I've forgotten if 2.3 does too).  Feel free to suggest improvements - I'm not much of a tracing expert!  To switch off tracing altogether, set the boolean variable called <b> tracing</b> to false in Network in package com.jpmorrsn.fbp.engine .   I know how to set properties in C#, but I didn't know how to set them in Java!   Switching off tracing certainly speeds things up considerably!  You could also read the chapter on performance - http://www.jpaulmorrison.com/fbp/perform.htm - if you haven't done so already.

            Hope this helps - let me know how it goes!  Paul

             
            • Pani

              Pani - 2009-07-02

              Thanks Paul. I turned off the trace per your email. The program used to run for about 32  minutes earlier - with tracing. Now it took 21 minutes to do the same job.

              BTW - I am comparing JavaFBP with just plain simple Java-Class implementation of the same logic. It has been a good exercise so far. I will publish all the results once I am done. Should be done by eod today. I will try to get some documentation ready too - that would probably help the visitors on your site. I will probably come back with more questions on JavaFBP too :).

              Thanks,
              Pani

               
              • Pani

                Pani - 2009-07-02

                Hi Paul,

                Here is what I was trying to do. I have a text file with data in the following format. This text file has 560K lines.

                ID, DATA
                1001, "1A-3A, 20B-25B"
                1002, "2AA-4AA, 5BB-8BB"

                I need to read the above file and create another file in the following format. As you can see the target file will have more lines. Data with error conditions need to be written into another file.

                ID, DATA
                1001, 1A
                1001, 2A
                1001, 3A
                1001, 20B
                1001, 21B
                1001, 22B
                1001, 23B
                1001, 24B
                1001, 25B
                1002, 2AA
                1002, 3AA
                1002, 4AA
                1002, 5BB
                1002, 6BB
                1002, 7BB
                1002, 8BB

                I implemented the above in 3 different ways (all three programs shared the same core logic - with few system.out.println statements of course).

                1)With JavaFBP - Single Process way - refer to Single_Process_Example.png
                2)With JavaFBP - Using Multi-plexing - refer to Multi_Process_Example.png
                3)With Plain Java Class reading,processing and writing from the same Java Class

                Here are my findings - (swtiched off the tracing). The following programs were run
                on a Intel Xeon at 3.0 GHZ - 4 CPU Windows 2003 Server OS - with 6 GB RAM)

                1)With JavaFBP - Single Thread Processing (21 minutes)
                2)With JavaFBP - Using Multi-plexing (30 minutes)
                3)With Plain Java Class reading,processing and writing from the same Java Class (17 minutes)

                I was expecting a performance improvement when I did the multiplexing but it degraded as you can see. My goal was to use multiplexing so that we have different threads working on different lines coming in from load balancer - thereby decreasing the elapsed time to process the source file. The ultimate goal was to see shorter processing time than the Single plain java-class by using multi-threading.

                Is multi-plexing not helping us here because there are some seconds lost in load_balancing upstream and consolidating the packets downstream? I was trying to see if JavaFBP can help me process data faster using multiplexing since we have more CPU power available from the new multi-core, mutli-CPU servers. Please let me know if there is a different way to approach the above problem.

                Thanks,
                Pani

                 
                • J. Paul Morrison

                  Hi Pani,

                  That's very interesting!  I can't see the png files, but I assume the multithreaded version uses a load balancer and multiple instances of your "splitter".  Given the simplicity of the task, it is not too surprising that JavaFBP takes a bit longer than a single Java class.  However,  the multithreaded version should indeed be able to take advantage of the 4 processors - but you imply that you had to "consolidate" the packets downstream, which would end up costing quite a bit.  I see in your problem description that you want to preserve the original sequence in your output, but I am wondering if you really need to do that.  In my experience, we only need to maintain sequence if humans are going to read it - or you need a fixed sequence so you can compare the file with another one.  In my book I advise against trying to recombine "split" files as it is usually not worth the overhead!

                  Regards,

                  Paul

                   

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks