Menu ▾ ▴

Adding a new event filter

Tobi Delbruck Luca Longinotti Sim Bamford BjoernBeyer

How can I write my own custom event processor?

Event processing is at the heart of jAER's purpose. Although viewing, logging, and displaying AER from systems is very useful, the true power of this representation is explored by doing things with the events.

Adding a new event filter is the way you can process events from your system (retina, cochlea, multineuron learning chip, etc) in jAER. You can develop your own event processing filter by following these steps:

  1. Subclass net.sf.jear.eventprocessing.EventFilter2D
  2. Since you are extending the EventFilter2D which is extending the EventFilter you need to Override the following methods:
    • filterPacket(EventPacket) (to extend abstract EventFilter2D)
    • resetFilter() (to extend abstract EventFilter)
    • initFilter() (to extend abstract EventFilter)
  3. Add properties that will be automatically displayed in the GUI to the filter by implementing javabean getter/setter methods.
  4. Document your filter properties by using the EventFilter instance method setPropertyTooltip(String groupName, String parameterName, String tip). The first parameter is optional and will cause the properties to be grouped together if entered. This tooltip will be shown to the user when they hover over the property label. See below for explicit instructions on adding this netbeans macro.
  5. Run the project to show the AEViewer window, and then add the filter to the processing chain as follows:
    from the menu choose "View" -> "Event filtering" -> "Filters"; this brings up the "Filters" window;
    then choose "View" -> "Select filters"; this brings up the "Class Chooser";
    select your filter in the list to the left, and use "Add" to move it to the right;
    The "Class list" to the right defines the filters that will be applied and the order in which they will be applied. Your Filter will be added to the left in the left automatically if it subclasses EventFilter2D either directly or as a child of a subclassing class.
    There is no need to add your new filter class to any static array of classes - you can add arbitrary filters for each Chip class as just described. Once you return from the "Class chooser", the "Filters" window is populated with the filters you have chosen. Activate any combination of these that you like by ticking the tick box to the left of each filter to enable it.
    Each time new events arrive -- either from hardware, from a network socket, or from a logged data file -- your filterPacket method will be called automatically and you can do whatever you like with the events.

If you're interested in processing event streams coming in from two or more sensors, see the "Multi-Modal Processing" page.

Annotating the rendered graphics

You can annotate the ChipCanvas window by implementing the FrameAnnotator interface and then filling in the _annotate(GLDrawable)'' method if you want to draw something on top of the rendered events. There are many examples in the existing code base, browse the online javadoc for jAER or see for examples SimpleOrientationFilter, or RectangularClusterTracker for highly evolved event processing classes.

Adding properties

You can add properties (parameters) to your filter that are adjustable via the GUI by implementing javabean get/set methods for a field.
This can be done for boolean, integer and float variables and will automatically add the methodname (the common name after 'get' and 'set') and a textbox to enter values to the filter UI.
For example, the following, taken from the complete example below, adds a property for an float value:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
private float floatProperty = getFloat("floatProperty",0.1f);

/** getter for FloatProperty
 * @return value of floatProperty */
public float getFloatProperty(){
  return floatProperty;
}

/** setter for FloatProperty; updates GUI and saves as preference 
 * @param NewFloat float value to set */
public void setFloatProperty(final float NewFloat) {
  putFloat("floatProperty",NewFloat);
  float OldValue = this.floatProperty;
  this.floatProperty = NewFloat;
  support.firePropertyChange("floatProperty",OldValue,NewFloat);
}

Adding Sliders

As well as adding simple properties it is possible to add sliders to the filter UI by implementing javabean get/set/min/max methods for a field.
This will add the name, a textbox to enter values and a slider that can be dragged with the mouse to the filter UI. The Slider can not be dragged to lower/higher values than set by the min/max methods, but lower/higher values can be entered in the textbox.
The following code fragment adds a float slider:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
private float floatSlider = getFloat("floatSlider",0.789f);

/** getter for FloatSlider
 * @return value of floatSlider */
public float getFloatSlider(){
  return floatSlider;
}

/** setter for FloatSlider; updates GUI and saves as preference 
 * @param NewFloat float value to set */
public void setFloatSlider(float NewFloat) {
  putFloat("floatSlider",NewFloat);
  float OldValue = this.floatSlider;
  this.floatSlider = NewFloat;
  support.firePropertyChange("floatSlider",OldValue,NewFloat);
}

/** getter for minimum value of floatSlider
 * @return minimum value of slider for floatSlider */
public float getMinFloatSlider() {
  return MinFloatSlider;
}

/** getter for maximum value of floatSlider
 * @return maximum value of slider for floatSlider */
public float getMaxFloatSlider() {
  return MaxFloatSlider;
}

Adding Actions

In addition to adding properties and sliders on can add actions, that will show automatically in the filter UI as pressable buttons by implementing the javabean 'do' method for a function.
The label of the button is the name of the method after the 'do' keyword.
The following code fragment adds a action:

1
2
3
4
5
/** Implements a button in the filter UI 'yourActionHere' */
public void doYourActionHere() {
  //Add your code here that is exectued
  // upon button press.
}

Processing events

Iterating over the events in a packet

Events are passed into filterPacket as a EventPacket. You can iterate over these events, casting each one to the expected type of event, e.g. PolarityEvent in the case of on/off retina events. PolarityEvent extends BasicEvent to add a spike polarity. BasicEvent's have a timestamp and x and y event locations. The timestamp is in microseconds.

Here's a code example taken from below

1
2
for(Object e:in){ // iterate over the input packet
BasicEvent i=(BasicEvent)e; // cast the object to basic event to get timestamp, x and y

Note how the iterator iterates over the input packet in (passed in the filterPacket call). The events are then cast from Object to BasicEvent. If you know your input events are a deeper subclass of BasicEvent (e.g. PolarityEvent), then you can cast to this class directly.

Generating new output events

To generate output events that are different than the input events (e.g. in the case of a cleaing-filter that discards events, or an annotating-filter that annotates events with additional information, e.g. orientation), you need to use EventFitler's built-in output packet out. Here's an example. First, before the packet iterator, make sure the built in output packet has the correct type of output event. In the example below, the output packet is checked to have the same event type as the input packet. But you could also call checkOutputEventType with a class, e.g. PolarityEvent.class, and checkOutputEventType would allocate these events for output.

1
checkOutputPacketEventType(in); // always do this check to make sure you have a valid output packet, unless you are just returning back the input events

Next, also before the packet iterator, get the built-in output iterator outItr. This iterator lets you access the next output event. Calling out.outputIterator initializes the output packet (effectively clearing it).

1
OutputEventIterator outItr=out.outputIterator(); // initialize the obtain the output event iterator

If you want to actually generate an output event, then Inside the packet iterator make a call something like the following example.

1
2
BasicEvent o=(BasicEvent)outItr.nextOutput();  // make an output event
o.copyFrom(i); // copy the BasicEvent fields from the input event

Here the outItr is called to get a reference to the next output, and then the fields such as timestamp, x_, and _y from the event i are copied to o.

Finally, if you want to modify the output events, or annotate them, you can now directly set the output event's fields. For example

o.x=o.x-10; // this will shift the events 10 pixels to the left

Here is where you also annotate the events with new information, e.g. orientation or direction and speed.

Parameter persistence

You can add persistence of your filter properties using the [EventFilter2D.getPrefs()] method to obtain a Java Preferences key. In your property setter, use getPrefs() to put your property.

You can see examples of using preferences in the example below.

The methods getInt, putInt, getFloat, getString, etc are built-in and prepend the class name to the key.
These methods are available for the data types: 'long', 'int', 'float', 'double', 'byteArray', 'floatArray', 'boolean', 'String'.

Updating your GUI property values automatically

Each EventFilter has a built-in PropertyChangeSupport named "support". You can use this to fire property changes that will update the GUI property values if they are changed using the properties set method. For example, in the fragment below, the preference value is set for property "dt", it's value is saved, then the value is updated, then the property change event is fired passing the old and new values of dt. Adding the property change support is only necessary if the method is called by threads other than the GUI user interface thread. For example, if your filter dynamically changes dt and you want the GUI to show the actual value to the user.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
// the built-in method getInt gets the value from Preferences using the key "dt"
protected int dt=getInt("dt",30000);
// adds a tooltip for "dt" property
setPropertyTooltip("Timing","dt", "max delay allowed for passing events");

// this getter/setter pair defines a property for which a control is automaticaly built in the GUI**
public int getDt() { // note how "dt" becomes "Dt"
  return this.dt;
}

public void setDt(final int dt) {
  putInt("dt",dt);
  int olddt=this.dt;
  this.dt = dt;

  // updates the GUI if this method is called from code
  support.firePropertyChange("dt",olddt,dt); 
}

Adding tooltips and grouping properties

As shown above, you can add tooltips for your properties using the built-in method setPropertyTooltip, as shown in the following fragment taken from SimpleOrientationFilter's constructor:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
/** Creates a new instance of SimpleOrientationFilter */
public SimpleOrientationFilter(AEChip chip) {
  super(chip);
  chip.addObserver(this);
  // properties, tips and groups
  final String size="Size", tim="Timing", disp="Display";

  setPropertyTooltip(disp,"showGlobalEnabled", "shows line of average orientation");
  setPropertyTooltip(tim,"minDtThreshold", "Coincidence time, events that pass this coincidence test are considerd for orientation output");
  setPropertyTooltip(tim,"dtRejectMultiplier", "reject delta times more than this factor times minDtThreshold to reduce noise");
  setPropertyTooltip(tim,"dtRejectThreshold", "reject delta times more than this time in us to reduce effect of very old events");
  setPropertyTooltip("multiOriOutputEnabled", "Enables multiple event output for all events that pass test");
  setPropertyTooltip(tim,"useAverageDtEnabled", "Use averarge delta time instead of minimum");
  setPropertyTooltip(disp,"passAllEvents", "Passes all events, even those that do not get labled with orientation");
  setPropertyTooltip(size,"subSampleShift", "Shift subsampled timestamp map stores by this many bits");
  setPropertyTooltip(size,"width", "width of RF, total is 2*width+1");
  setPropertyTooltip(size,"length", "length of half of RF, total length is length*2+1");
  setPropertyTooltip(tim,"oriHistoryEnabled", "enable use of prior orientation values to filter out events not consistent with history");
  setPropertyTooltip(disp,"showVectorsEnabled", "shows local orientation segments");
  setPropertyTooltip(tim,"oriHistoryMixingFactor", "mixing factor for history of local orientation, increase to learn new orientations more quickly");
  setPropertyTooltip(tim,"oriDiffThreshold", "orientation must be within this value of historical value to pass");
}

This creates the following GUI:

Adding class description

The annotation Description adds a description that is used to build the GUIs. Use this to add a tooltip describing your class, as shown below:

1
2
3
// this jAER annotation type is used in GUIs
@Description("Removes uncorrelated background activity") 
public class BackgroundActivityFilter extends EventFilter2D implements Observer  {

An example of a complete EventFilter

Below is commented and simplified code for BackgroundActivityFilter. This filter removes background activity from uncorrelated AEs. It only pass through events that have recieved some "support" from the spatio-temporal neighborhood.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
package ch.unizh.ini.caviar.eventprocessing.filter;

import ch.unizh.ini.caviar.chip.*;
import ch.unizh.ini.caviar.event.*;
import ch.unizh.ini.caviar.event.EventPacket;
import ch.unizh.ini.caviar.eventprocessing.EventFilter2D;
import java.util.*;

/**
 * An AE background that filters slow background activity by only passing 
 * inPacket that are supported by another event in the past 
 * {@link #setDt dt} in the immediate spatial neighborhood, defined
 * by a subsampling bit shift.
 * @author tobi */
// this jAER annotation type is used in GUIs
@Description("Removes uncorrelated background activity") 
public class BackgroundActivityFilter extends EventFilter2D implements Observer{
  public boolean isGeneratingFilter(){ return false;}
  final int DEFAULT_TIMESTAMP=Integer.MIN_VALUE;

  /** the time in timestamp ticks (1us at present) that a spike
   * needs to be supported by a prior event in the 
   * neighborhood by to pass through */
  // the built-in method getInt gets the value from Preferences
  protected int dt=getInt("dt",30000);

  /** the amount to subsample x and y event location by in bit shifts 
   * when writing to past event times map. This effectively increases 
   * the range of support. E.g. setting subSamplingShift to 1 quadruples 
   * range because both x and y are shifted right by one bit */
  private int subsampleBy=getPrefs().getInt("BackgroundActivityFilter.subsampleBy",0);


  int[][] lastTimestamps;

  public BackgroundActivityFilter(AEChip chip){
    super(chip); // always call the super method!
    // do this if you want the filter to be informed of 
    //  changes in the chip (e.g. size)
    chip.addObserver(this); 
    initFilter();
    resetFilter();
    // this adds tooltips in GUI
    setPropertyTooltip("dt","Events with less than this delta time to neighbors pass through");
    setPropertyTooltip("subsampleBy","Past events are subsampled by this many bits");
  }

  void allocateMaps(AEChip chip){
    lastTimestamps=new int[chip.getSizeX()][chip.getSizeY()];
  }

  int ts=0; // used to reset filter

  /**filters in to out. If filtering is enabled, the number of out may be less
   * than the number put in
   * @param in input events can be null or empty.
   * @return the processed events, may be fewer in number. 
   *  filtering may occur in place in the in packet. */
  synchronized public EventPacket filterPacket(EventPacket in) {
    if(!filterEnabled) return in; // do this check to avoid always running filter
    if(enclosedFilter!=null) {
      in=enclosedFilter.filterPacket(in); // you can enclose a filter in a filter
    }

    // always do this check to make sure you have a valid output packet, 
    //  unless you are just returning back the input events
    checkOutputPacketEventType(in); 
    if(lastTimestamps==null) allocateMaps(chip);


    // The general idea for this filter is for each event only write it to the 
    //  out buffers if it is within dt of the last time an event happened 
    //  in neighborhood

    // initialize the obtain the output event iterator
    OutputEventIterator outItr=out.outputIterator(); 
    int sx=chip.getSizeX()-1;
    int sy=chip.getSizeY()-1;
    for(Object e:in) { // iterate over the input packet
      // cast the object to basic event to get timestamp, x and y
      BasicEvent i=(BasicEvent)e; 
      ts=i.timestamp;
      short x=(short)(i.x>>>subsampleBy), y=(short)(i.y>>>subsampleBy);
      int lastt=lastTimestamps[x][y];
      int deltat=(ts-lastt);
      if(deltat<dt && lastt!=DEFAULT_TIMESTAMP){
        BasicEvent o=(BasicEvent)outItr.nextOutput();  // make an output event
        o.copyFrom(i); // copy the BasicEvent fields from the input event
      }

      try{
        // for each event stuff the event's timestamp into the lastTimestamps 
        //  array at neighboring locations
        // lastTimestamps[x][y][type]=ts; // don't write to ourselves, we need 
        //  support from neighbor for next event
        // bounds checking here to avoid throwing expensive exceptions, 
        //  even though we duplicate java's bound checking...
        if(x>0) lastTimestamps[x-1][y]=ts;
        if(x<sx) lastTimestamps[x+1][y]=ts;
        if(y>0) lastTimestamps[x][y-1]=ts;
        if(y<sy) lastTimestamps[x][y+1]=ts;
        if(x>0 && y>0) lastTimestamps[x-1][y-1]=ts;
        if(x<sx && y<sy) lastTimestamps[x+1][y+1]=ts;
        if(x>0 && y<sy) lastTimestamps[x-1][y+1]=ts;
        if(x<sx && y>0) lastTimestamps[x+1][y-1]=ts;
      }catch(ArrayIndexOutOfBoundsException eoob){
        allocateMaps(chip);  // the chip size may have changed
      }  // boundaries
    }
    // you must return the output packet (or the original input packet if you 
    //  are not modifying the events
    return out;  
  }

  // this getter/setter pair defines a property for which a control is 
  //  automatically built in the GUI
  /**gets the background allowed delay in us
   * @return delay allowed for spike since last in neighborhood to pass (us) */
  public int getDt() {
    return this.dt;
  }

  /**sets the background delay in us
   * @see #getDt
   * @param dt delay in us */
  public void setDt(final int dt) {
    // the putInt method is built-in, along with putFloat, putString, etc. 
    //  It stores the value using the key "dt"
    putInt("dt",dt); 
    // calling support with the same name updates the GUI
    support.firePropertyChange("dt",this.dt,dt); 
    this.dt = dt;
  }

  public Object getFilterState() {
    return lastTimestamps;
  }

  void resetLastTimestamps(){
    for(int i=0;i<lastTimestamps.length;i++)
      Arrays.fill(lastTimestamps[i],DEFAULT_TIMESTAMP);
  }

  synchronized public void resetFilter() {
    // set all lastTimestamps to max value so that any event is soon enough, 
    //  guarenteed to be less than it
    resetLastTimestamps();
  }


  public void update(Observable o, Object arg) {
    initFilter();
  }

  public void initFilter() {
    allocateMaps(chip);
  }

  public int getSubsampleBy() {
    return subsampleBy;
  }

  /** Sets the number of bits to subsample by when storing events into 
   * the map of past events.
   * Increasing this value will increase the number of events that pass 
   * through and will also allow passing events from small sources that 
   * do not stimulate every pixel.
   * @param subsampleBy the number of bits, 0 means no subsampling, 
   *   1 means cut event time map resolution by a factor of two in x and in y */
  public void setSubsampleBy(int subsampleBy) {
    if(subsampleBy<0) subsampleBy=0; else if(subsampleBy>4) subsampleBy=4;
    this.subsampleBy = subsampleBy;
    putInt("BackgroundActivityFilter.subsampleBy",subsampleBy);
  }

}

Using a filter

To use this filter, build the project (remember to build the Project jar, F11 in netbeans, not just compile the class), then open the filter window and Select View/Customize...

You should get the following dialog:

Select your filter on the left side (you can filter on part of it's name using the Filter box) and double click it or use the Add button to add it to the list on the right. Edit the list of active filters as you like. Click OK. Now your filter will be in the list of filters in the FilterFrame:

Controlling a filter

Clicking on the R button resets the filter; the P button collapses the list and opens the GUI for controlling the filter

Adding macro to netbeans to ease filter tooltip making

In netbeans, open the tab Tools/Options/Editor/Macro and create a new macro with the following macro string

select-word copy-to-clipboard caret-begin-line caret-down "{setPropertyTooltip(\"" paste-from-clipboard "\",\"\");}" insert-break caret-up caret-end-line caret-backward caret-backward caret-backward caret-backward

To use this macro, click inside the property and hit Ctrl+9.

Related

Wiki: Home
Wiki: MultiModalProcessing