#17 PING from workers / queue / poller enhancements

None
new
nobody
Feature (2)
2014-08-02
2013-10-07
Oliver Gorwits
No

Perhaps a good idea to show admins a little icon in the navbar that workers are online and doing/waiting-for jobs.

Discussion

  • Oliver Gorwits
    Oliver Gorwits
    2013-10-09

    • Status: open --> new
     
  • Eric A. Miller
    Eric A. Miller
    2014-01-09

    Expanding the ticket based upon conversations about the queue and distributed pollers.

    Some thoughts:

    1) It is desirable to have distributed pollers to address connectivity issues to managed devices and allow for splitting load among pollers if needed for scalability. However, having the configuration distributed could prevent the application from being able to use those configuration details if needed. It will also make maintaining configuration files more difficult as they will not be managed in one place.

    2) The queue(s) shouldn't be the performance constraint.

    3) A pub-sub like system might be desirable as the pollers could subscribe to a queue from which jobs would be published.

    4) Desirable not to add additional complex dependencies (Ex: Full blown message queue, etc.)

    5) I would like to find a solution which would be compatible with a HTTPS transport.

    Some links I find of interest on the topic:

    queue_classic Ruby project, but shouldn't be that difficult to port workers to Perl. Requires Postgresql 9+ to to use enhancements to LISTEN/NOTIFY which is basically a Postgresql pub-sub.

    Dancer websockets

    AnyMQ and websockets

    Thought would be to implement queue with multiple channels and have worker nodes communicate with the “manager” over websocket. Keep all configuration on the main application server and have the worker nodes subscribe to a configuration queue/channel on initial start then subscribe to the designated work queue/channel.

     
  • Eric A. Miller
    Eric A. Miller
    2014-01-09

    • summary: PING from workers --> PING from workers / queue / poller enhancements
     
  • Oliver Gorwits
    Oliver Gorwits
    2014-02-23

    Also had http://zeromq.org/ recommended to me.

     
  • Eric A. Miller
    Eric A. Miller
    2014-02-24

    In consideration of #27 and #2 is the concept of a queue per "Domain"?

     
  • Oliver Gorwits
    Oliver Gorwits
    2014-08-02

    Ticket moved from /p/netdisco/netdisco2/26/