#17 PING from workers / queue / poller enhancements

None
moved-to-github
nobody
Feature (2)
2014-08-02
2013-10-07
No

Perhaps a good idea to show admins a little icon in the navbar that workers are online and doing/waiting-for jobs.

Discussion

  • Oliver Gorwits

    Oliver Gorwits - 2013-10-09
    • Status: open --> new
     
  • Eric A. Miller

    Eric A. Miller - 2014-01-09

    Expanding the ticket based upon conversations about the queue and distributed pollers.

    Some thoughts:

    1) It is desirable to have distributed pollers to address connectivity issues to managed devices and allow for splitting load among pollers if needed for scalability. However, having the configuration distributed could prevent the application from being able to use those configuration details if needed. It will also make maintaining configuration files more difficult as they will not be managed in one place.

    2) The queue(s) shouldn't be the performance constraint.

    3) A pub-sub like system might be desirable as the pollers could subscribe to a queue from which jobs would be published.

    4) Desirable not to add additional complex dependencies (Ex: Full blown message queue, etc.)

    5) I would like to find a solution which would be compatible with a HTTPS transport.

    Some links I find of interest on the topic:

    queue_classic Ruby project, but shouldn't be that difficult to port workers to Perl. Requires Postgresql 9+ to to use enhancements to LISTEN/NOTIFY which is basically a Postgresql pub-sub.

    Dancer websockets

    AnyMQ and websockets

    Thought would be to implement queue with multiple channels and have worker nodes communicate with the “manager” over websocket. Keep all configuration on the main application server and have the worker nodes subscribe to a configuration queue/channel on initial start then subscribe to the designated work queue/channel.

     
  • Eric A. Miller

    Eric A. Miller - 2014-01-09
    • summary: PING from workers --> PING from workers / queue / poller enhancements
     
  • Oliver Gorwits

    Oliver Gorwits - 2014-02-23

    Also had http://zeromq.org/ recommended to me.

     
  • Eric A. Miller

    Eric A. Miller - 2014-02-24

    In consideration of #27 and #2 is the concept of a queue per "Domain"?

     
  • Oliver Gorwits

    Oliver Gorwits - 2014-08-02

    Ticket moved from /p/netdisco/netdisco2/26/

     
  • Oliver Gorwits

    Oliver Gorwits - 2017-02-02
    • Status: new --> moved-to-github