Database Crawler

  • Richard SINELLE

    Richard SINELLE - 2010-11-18


    I have just create a database Crawler.

    How is it possible to launch it for example every 30 minutes ?



  • Emmanuel Keller

    Emmanuel Keller - 2010-11-19

    Hi Richard,

    There is an HTTP API for that. You can launch any database crawl by using this


    Replace indexname and dbcrawlername with your values.

    If you have activated the privileges, you have to add the login and the (api)
    key parameters:


    If you are on a Unix/Linux environment, you can then add a crontab entry to
    automate the job:

    */30 * * * * wget [url]http://localhost:8080/database?use=indexname&name=dbcrawlername[/url]



  • Richard SINELLE

    Richard SINELLE - 2010-11-19

    Thanks Emmanuel for your quickly answer.

    I'm on Windows plateform.

    If I want to add a scheduler in OSS (like for the web crawler), how can I
    start to study this add on ?

  • Richard SINELLE

    Richard SINELLE - 2010-11-22

    Thanks for your answer.

    For my project, it will not possible to use an external scheduler..... no I
    don't want to use an external scheduler because I will have 25 index with for
    each a Database crawler. So I prefer to have the possibility to launch or stop
    the crawler with a web interface.

    For information, I just to finish to update the source code in order to add
    this crawler. Are you interested for a patch file ?


  • Emmanuel Keller

    Emmanuel Keller - 2010-11-24

    We are always happy to integrate valuables contributions !

    If you want to do that, you can create a related artifact in the new features
    tracker and attach your patch.




Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks