Menu ▾ ▴

MySql Journal problems

Help
maxdaros
2004-12-23
2013-04-16

  • maxdaros

    maxdaros - 2004-12-23

    Hi all,
    can anyone help me with this problem?

    I'm trying journaling with MySql DB.

    When babeldoc try to log journal into DB I've the following output log:

    D:\DRG\Scanners\DRGScanner>babeldoc scanner -s ./config.properties
    <2004-11-23 23:59:59,156> INFO  [main] :  Initializing workers:
    <2004-11-23 23:59:59,326> INFO  [main] :  scanner (ftp) configured...
    <2004-11-23 23:59:59,336> INFO  [main] :  Starting workers...
    <2004-11-24 00:00:00,458> INFO  [main] :  scanner started...
    <2004-11-24 00:00:10,572> INFO  [scanner] :  Document MTL1216.zip enqueued for pipeline Dbloader
    <2004-11-24 00:00:10,713> INFO  [Thread-0] :  Processing document...
    <2004-11-24 00:00:11,133> INFO  [Thread-0] :  PipelineStage name: decompress
    <2004-11-24 00:00:11,233> INFO  [Thread-0] :  PipelineStage name: addAttrib
    <2004-11-24 00:00:11,303> INFO  [Thread-0] :  PipelineStage name: fWriter0
    <2004-11-24 00:00:11,524> INFO  [Thread-0] :  PipelineStage name: router
    <2004-11-24 00:00:11,624> INFO  [Thread-0] :  PipelineStage name: convert
    <2004-11-24 00:00:13,577> INFO  [Thread-0] :  PipelineStage name: fWriter
    <2004-11-24 00:00:14,328> INFO  [Thread-0] :  PipelineStage name: transform
    <2004-11-24 00:00:18,183> INFO  [Thread-0] :  PipelineStage name: fWriter2
    <2004-11-24 00:00:19,285> ERROR [Thread-0] :  [GenericSqlJournal.log]
    java.sql.SQLException: java.lang.IllegalArgumentException: Packet is larger than max_allowed_packet from server configur
    ation of 1048576 bytes
            at com.babeldoc.sql.journal.MySqlJournal.writeDelta(Unknown Source)
            at com.babeldoc.sql.journal.GenericSqlJournal.log(Unknown Source)
            at com.babeldoc.core.journal.Journal.updateDocument(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.trackDocument(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
            at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
            at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
            at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
            at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source)
            at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Thread.java:595)
    <2004-11-24 00:00:19,295> INFO  [Thread-0] :  PipelineStage name: transform2
    <2004-11-24 00:00:19,635> ERROR [Thread-0] :  [GenericSqlJournal.log]
    java.sql.SQLException: java.lang.IllegalArgumentException: Packet is larger than max_allowed_packet from server configur
    ation of 1048576 bytes
            at com.babeldoc.sql.journal.MySqlJournal.writeDelta(Unknown Source)
            at com.babeldoc.sql.journal.GenericSqlJournal.log(Unknown Source)
            at com.babeldoc.core.journal.Journal.updateDocument(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.trackDocument(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
            at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
            at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
            at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
            at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
            at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
            at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source)
            at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Thread.java:595)
    <2004-11-24 00:00:24,713> INFO  [Thread-0] :  PipelineStage name: fWriter3
    <2004-11-24 00:00:24,883> INFO  [Thread-0] :  PipelineStage name: dbwriter

    I've a FTP Scanner that scans a ftp site and feed the documents into a pipeline with the following stages  unzip --> flat file to xml --> transfor to a set of SQL Insert statements -->  load data into DB.
    The data is stored correctly into DB, but the journaling isn't working correctly.

    My pipeline config file is:
    entryStage                       = decompress

    decompress.stageType             = Decompress
    decompress.compressType          = zip
    decompress.nextStage             = addAttrib
    decompress.errorStage            = errorHandler
    decompress.failOnError           = false
    decompress.tracked               = true

    addAttrib.stageType              = Enrich
    addAttrib.enrichScript.unzipped  = ${document.name.substring(0,7)}.txt
    addAttrib.nextStage              = fWriter0
    addAttrib.errorStage             = errorHandler
    addAttrib.failOnError            = false
    addAttrib.tracked                = true

    fWriter0.stageType               = FileWriter
    fWriter0.nextStage               = router
    fWriter0.outputFile              = ../../Unzipped/${document.get("unzipped")}
    fWriter0.errorStage              = errorHandler
    fWriter0.failOnError             = false
    fWriter0.tracked                 = true

    # continua il dbload solo per gli MTL* (per ATI* e OPC* non  ancora stata definita la pipeline)
    # In futuro definire 3 diversi tronconi di pipeline con 3 if differenti e con else (il nextSTage senza test) null
    router.stageType                 = Router
    router.nextStage.convert         = #if ("MTL" == (${document.name.substring(0,3)}))true#end
    router.nextStage                 = null
    router.errorStage                = errorHandler
    router.failOnError               = false
    router.tracked                   = true

    convert.stageType                = FlatToXml
    convert.nextStage                = fWriter
    convert.flatToXmlFile            = ../../Pipelines/Dbloader/conversion/file-convert.xml
    convert.errorStage               = errorHandler
    convert.failOnError              = false
    convert.tracked                  = true

    # Disabilitato
    fWriter.stageType                = FileWriter
    fWriter.ignored                 = true
    fWriter.nextStage                = transform
    fWriter.outputFile               = ../../Pipelines/Dbloader/outputs/XmlOutput.xml
    fWriter.errorStage               = errorHandler
    fWriter.failOnError              = false
    fWriter.tracked                  = true

    transform.stageType              = XslTransform
    transform.nextStage              = fWriter2
    transform.transformationFile     = ../../Pipelines/Dbloader/conversion/transform.xsl
    transform.errorStage             = errorHandler
    transform.failOnError            = false
    transform.tracked                = true

    # Disabilitato
    fWriter2.stageType               = FileWriter
    fWriter2.ignored                 = true
    fWriter2.nextStage               = transform2
    fWriter2.outputFile              = ../../Pipelines/Dbloader/outputs/XmlOutput2.xml
    fWriter2.errorStage              = errorHandler
    fWriter2.failOnError             = false
    fWriter2.tracked                 = true

    transform2.stageType             = XslTransform
    transform2.nextStage             = fWriter3
    transform2.transformationFile    = ../../Pipelines/Dbloader/conversion/transform2.xsl
    transform2.errorStage            = errorHandler
    transform2.failOnError           = false
    transform2.tracked               = true

    # Disabilitato
    fWriter3.stageType               = FileWriter
    fWriter3.ignored                 = true
    fWriter3.nextStage               = dbwriter
    fWriter3.outputFile              = ../../Pipelines/Dbloader/outputs/SqlInserts.sql
    fWriter3.errorStage              = errorHandler
    fWriter3.failOnError             = false
    fWriter3.tracked                 = true

    dbwriter.stageType               = SqlWriter
    dbwriter.nextStage               = null
    dbwriter.messageTag              = messageTagErrorSql
    dbwriter.resourceName            = jdbcConnection
    dbwriter.sql                     = $document.contents
    dbwriter.failOnFirst             = false
    dbwriter.useBatch                = true
    dbwriter.errorStage              = errorHandler
    dbwriter.failOnError             = false
    dbwriter.tracked                 = true

    errorHandler.stageType           = SmtpWriter
    errorHandler.smtpHost            = mySmtpHost
    errorHandler.smtpFrom            = myAddress@mySite
    errorHandler.smtpTo              = myEmail
    errorHandler.smtpSubject         = Document: Ticket: ${ticket.Value}
    errorHandler.nextStage           = null
    errorHandler.tracked             = true

    Thanks in advance.

    Massimo

     

    • vanstalle

      vanstalle - 2004-12-27

      Hey,

      I don't know the database, but as you have tracked=true, babeldoc tries to store the generated document into the db; this can be interesting for debugging purposes, but is generally not a good idea for a production environment; try the set tracked=false to further diagnose the problem.

      Also take account of the fact that storing large binary objects is database dependent; I thought there is an Oracle implementation in babeldoc, but In don't know if there is and implementation for your database.

      Jan

       

    • maxdaros

      maxdaros - 2005-01-07

      Hi,

      the tracked=false in many stages of my pipeline solves the problem.

      Thank you very much.

      Massimo

       

Log in to post a comment.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.