Babeldoc: Universal Document Processor / Discussion / Help: Problems with Decompression stage type

maxdaros - 2004-12-11

Hi all,

babeldoc it's very exciting for me....

I need to download a zip file from a ftp server, then decompress it and load into a db.
So I'm using babeldoc FtpScanner to download the file and then to feed it into a pipeline that I called MTLdbloader.

The scanner process starts and download the file, but at this point I've a problem with the pipeline at first stage of type Decompression.

Here is the log with the error:
D:\test\FTPscanner>babeldoc scanner -s ./config.properties
<2004-11-11 18:56:29,052> INFO [main] : Initializing workers:
<2004-11-11 18:56:29,202> INFO [main] : scanner (ftp) configured...
<2004-11-11 18:56:29,212> INFO [main] : Starting workers...
<2004-11-11 18:56:29,613> INFO [main] : scanner started...
<2004-11-11 18:57:29,819> INFO [scanner] : Document MTL1203.zip enqueued for pipeline MTLdbloader
<2004-11-11 18:57:30,280> INFO [Thread-0] : Processing document...
<2004-11-11 18:57:30,630> INFO [Thread-0] : PipelineStage name: decompress
<2004-11-11 18:57:30,660> ERROR [Thread-0] : [AsynchronousFeeder$1.run]
com.babeldoc.core.pipeline.PipelineException: Invalid pipeline stage type: Decompression
        at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.getPipelineStage(Unknown Source)
        at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
        at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
        at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
        at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
        at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
        at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source)
        at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:595)

My MTLdbloader pipeline configuration file is:
entryStage                    = decompress

decompress.stageType          = Decompression
decompress.compressionType    = gzip, zip
decompress.nextStage          = fWriter0
decompress.tracked            = true

fWriter0.stageType            = FileWriter
fWriter0.nextStage            = convert
fWriter0.outputFile           = data/inputData.txt
fWriter0.tracked              = true

convert.stageType             = FlatToXml
convert.nextStage             = fWriter
convert.flatToXmlFile         = conversion/file-convert.xml
convert.tracked               = true

fWriter.stageType             = FileWriter
fWriter.nextStage             = transform
fWriter.outputFile            = outputs/XmlOutput.xml
fWriter.tracked               = true

transform.stageType           = XslTransform
transform.nextStage           = fWriter2
transform.transformationFile = conversion/transform.xsl
transform.tracked             = true

fWriter2.stageType            = FileWriter
fWriter2.nextStage            = transform2
fWriter2.outputFile           = outputs/XmlOutput2.xml
fWriter2.tracked              = true

transform2.stageType          = XslTransform
transform2.nextStage          = fWriter3
transform2.transformationFile = conversion/transform2.xsl
transform2.tracked            = true

fWriter3.stageType            = FileWriter
fWriter3.nextStage            = dbwriter
fWriter3.outputFile           = outputs/SqlInserts.sql
fWriter3.tracked              = true

dbwriter.stageType            = SqlWriter
dbwriter.nextStage            = null
dbwriter.messageTag           = messageTagErrorSql
dbwriter.resourceName         = jdbcConnection
dbwriter.sql                  = $document.contents
dbwriter.failOnFirst          = false
dbwriter.useBatch             = true
dbwriter.tracked              = true

Can anyone tell me where is something incorrect in my pipeline configuration? There exist a pipeline stage of type "Decompression" (I looked in the source code and there exist a class com\babeldoc\core\pipeline\stage\DecompressionPipelineStage).

I think is useful for the community if there is an example of this stage type...

Many thank in advance.

Massimo

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Sherman Wood - 2004-12-11
  
  The pipeline type name for the Decompression stage is "Decompress". The naming is in a "query.properties" file in the core module.
  
  Sherman
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- maxdaros - 2004-12-11
  
  Thank you,
  
  now the pipeline stage type is correct
  
  any suggestion for the attribute compressionType in this stage?
  
  in my pipeline configuration file I have:
  decompress.stageType = Decompression
  decompress.compressionType = gzip, zip
  decompress.nextStage = fWriter0
  decompress.tracked = true
  
  but I think decompress.compressionType = gzip, zip isn't a good idea, and in the user guide there isn't a list of correct values.
  
  The pipeline give me another error:
  D:\test\FTPscanner>babeldoc scanner -s ./config.properties
  <2004-11-11 20:10:32,337> INFO [main] : Initializing workers:
  <2004-11-11 20:10:32,477> INFO [main] : scanner (ftp) configured...
  <2004-11-11 20:10:32,477> INFO [main] : Starting workers...
  <2004-11-11 20:10:32,768> INFO [main] : scanner started...
  <2004-11-11 20:10:42,882> INFO [scanner] : Document MTL1203.zip enqueued for pipeline MTLdbloader
  <2004-11-11 20:10:43,353> INFO [Thread-0] : Processing document...
  <2004-11-11 20:10:43,523> INFO [Thread-0] : PipelineStage name: decompress
  <2004-11-11 20:10:43,633> ERROR [Thread-0] : [DefaultPipelineStageErrorHandler.handlePipelineStageError] PipelineStage
  name: decompress Error: com.babeldoc.core.pipeline.PipelineException: Invalid compression type: {0}: null
  <2004-11-11 20:10:43,633> ERROR [Thread-0] : [AsynchronousFeeder$1.run]
  com.babeldoc.core.pipeline.PipelineException: Invalid compression type: {0}: null
          at com.babeldoc.core.pipeline.stage.DecompressionPipelineStage.decompressData(Unknown Source)
          at com.babeldoc.core.pipeline.stage.DecompressionPipelineStage.process(Unknown Source)
          at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source)
          at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
          at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
          at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
          at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
          at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
          at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
          at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source)
          at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
          at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Thread.java:595)
  
  Thank you in advance
  
  Massimo
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - Sherman Wood - 2004-12-11
    
    Just try:
    
    decompress.compressionType=zip
    
    Sherman
    
    If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- maxdaros - 2004-12-11
  
  I've tryed decompress.compressionType=zip
  
  but it doesn't work.
  I've the same error.
  
  Massimo
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- maxdaros - 2004-12-12
  
  Hi all,
  
  the right configuration for the decompression stage is:
  decompress.stageType          = Decompress
  decompress.compressType     = zip
  decompress.nextStage          = myNextStage
  decompress.tracked              = true
  
  Massimo
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Problems with Decompression stage type

Forums

Help

Problems with Decompression stage type document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

Problems with Decompression stage type