How to use Signals2 to pass data through tasks that filter and augment that data ?

[elpidio valdez]

This question is looking for design ideas and opinions for a system that can pass data to processing functions in a loosely coupled way. I believe this forum is more appropriate for such musings than, say stack overflow.

I will explain my use case and give my initial ideas in pseudo C++:

I have a sensor which provides rgb video frames and depth images at about 30Hz. I want to pass these frames to a variety of image processing functions in order to extract information about the environment. I have successfully used boost::signals2 to decouple the sensor from the processing. i.e. the sensor signals new frames and processing classes connect to those signals and listen for new frames. Using a shared pointer means that expensive image copies are avoided and the images will be automatically deleted when all the processing has been done.

signal<void (shared_ptr<RGBImage>)> OnNewFrameRGB;
signal<void (shared_ptr<DepthImage>)> OnNewFrameDepth;

I have realised that many processing functions create data which would be of use to other processing functions. They need to chain into a pipeline (but I want this to emerge from the system, not be explicitly programmed). As an example pipeline:

rgbImage -> grayscaleimage -> keypoint detection -> object recognition -> navigation decisions

The results of these processing steps may be of interest to more than one function e.g.

• object recognition -> speech response
• grayscaleimage -> edge detection

So it would be nice to aggregate the results as processing proceeds and broadcast the new data to the processing functions. All of this is on a per-frame basis.

I am considering using a Frame class to aggregate the results. Since data is passed by shared pointer the results should be available to all objects that recieve new frame signals. I also implement an overloaded function, add to aggregate the new data, and a signal to inform listeners when new data is available in the frame.

signal<void (shared_ptr<Frame> ) OnNewFrame;

class Frame {
  shared_ptr<RGBImage> rgb;
  shared_ptr<DepthImage> depth;
  shared_ptr<GrayImage> gray;
  shared_ptr<Keypoints> keyPoints;
  shared_ptr<GrayImage> edges;
  etc...

  enum DataKey {RGB, DEPTH, GRAY, KEYPOINTS, EDGES};

  signal<void (DataKey, shared_ptr<Frame>)> onDataAdded;

  void add(DataKey key, shared_ptr<RGBImage> rgbData) {
    rgb = rgbData;
    onDataAdded(RGB, this);
  }

    void add(DataKey key, shared_ptr<GrayImage> grayData) {
      switch (key) {
        case GRAY:  {gray  = grayData; onDataAdded(GRAY,  this); break;}
        case EDGES: {edges = grayData; onDataAdded(EDGES, this); break;}
      } 
  }

  lots of other overloads for 'add'...

};

The processing classes contain code like:

class Grayscale {

  Grayscale() {
    OnNewFrame.connect(bind(this, &Grayscale::register, _1));
  }

  void register(shared_ptr<Frame> frame) {
    frame.onDataAdded.connect(bind(this, &Grayscale::processFrame, _1, _2));
  }

  void processFrame(DataKey key, shared_ptr<Frame> frame) {
    if (key==RGB) {
      shared_ptr<GrayImage> gray = frame->rgb->ConvertToGrayscale();
      frame->add(GRAY, gray);
    }
  }

};

Thats where I have got to. I think the design can be improved in so many ways. I am looking for opinions and ideas - although a fully coded and tested solution would not be frowned upon :)

1. I hate having to hard code the types of aggregated data - it would be much nicer to have some extensible collection, but how to handle diverse datatypes ? boost.tuple ? boost::fusion::map ?
2. I hate listeners being pestered every time data is added and having to test if it is relevant to them. Probably want to use a function signature here.
3. There is a fair bit of boilerplate in the listener classes. I'd like that to be generated more automatically.

The solution is probably gratuitous amounts of TEMPLATE MAGIC. There is an excellent article, which is giving me ideas here:
https://thehermeticvault.com/software-development/making-boost-signals2-more-oop-friendly

[elpidio valdez]

I came up with a solution that works for me and inspired by:
https://thehermeticvault.com/software-development/making-boost-signals2-more-oop-friendly

I wrote a WorkflowContainer template class which I inherit in other classes to set up automatic execution of functions which add data to the container. Using this I can define a video frame as follows:

namespace field {
  struct rgb;
  struct depth;
  struct gray;
  struct keypoints;
  struct edges;
  struct objects;
}

typedef map<
  Field<field::rgb,       Image>,
  Field<field::depth,     DepthImage>,
  Field<field::gray,      Image>,
  Field<field::keypoints, Keypoints>,
  Field<field::edges,     Image>,
  Field<field::objects,   RecognizedObjects>
> FrameFields;

class Frame : public WorkflowContainer<FrameFields> {
public:

  Frame(int id) {
    frameNum = id;
  }

  ~Frame() {
    cout << "Deleting frame: " << frameNum << endl;
  }

  int frameNum;

};

Generator classes do the processing on the image frame and aggregate their results. An example definition is:

// Implement a step of workflow: rgb --> gray

class GrayImageGenerator {
public:

  GrayImageGenerator() {
    OnNewFrame.connect(boost::bind(&GrayImageGenerator::registerInterest, this, _1));
  }

  void registerInterest(const shared_ptr<Frame> &frame) {
    frame->connect<field::rgb>(boost::bind(&GrayImageGenerator::processFrame, this, frame));
  }

  void processFrame(const shared_ptr<Frame> &frame) {
    cout << "GrayImageGenerator" << " processing frame ";
    cout << frame->frameNum;
    cout << endl;

    frame->setData<field::gray>(new Image());
  }

} TheGrayImageGenerator;

I have attached the code and a simple example of usage.

[Smith jhone]

[zamirmewelldy]

Using Signals2 to pass data through tasks that filter and augment that data involves creating a chain of signal handlers that process the data at each stage. Signals2 is a flexible C++ library that allows for the implementation of the observer pattern, enabling communication between objects using signals for Smartcric live and slots.

To accomplish this task, you would start by defining the data structure to be passed through the tasks. Then, you would create signal objects to represent the flow of data. Each task would be implemented as a signal handler, with its own filtering and augmentation logic.