Menu ▾ ▴

Thread pool

Open Discussion
Biggun972
2011-10-02
2012-09-04

  • Biggun972

    Biggun972 - 2011-10-02

    Hi,

    Is there any support (future processors in next webharvest version or
    plugin(s) available) for parallel execution of processors in WebHarvest ?

    Parallel execution would be really helpful when fetching data from multiples
    urls for instance.

    I draw my thoughts here :

    <config>

    <threadpool count="5">

    <body>

    // Here i would call some processors for workng on the urls

    // The processors here are run in parallel

    </body>

    <on-success>

    // Optionnal tag containing processors to be executed each time one thread has
    successfully finished

    </on-success>

    <on-error>

    // Optionnal tag containing processors to be executed each time one thread has
    finished abruptly

    </on-error>

    </threadpool>

    </config>

    Stephan

     

  • Alex Wajda

    Alex Wajda - 2011-10-02

    Nice question, Stephan.

    Currently there is no concurrency support in WH. neither in v.2.0 or 2.1.

    I also think it would be a good feature. I have raised this idea earlier, but
    nobody seemed to be excited about it.

    Before the concurrency support can be added the WH processors needs to be
    refactored first to ensure their immutability.

    For me the biggest issue now is time. If anybody could help with that, that
    would be perfect.

     

  • Biggun972

    Biggun972 - 2011-10-02

    If BaseProcessor is immutable, is it enough ?

     

Log in to post a comment.