I'm trying to get the TCP Input to work with Dynap-se for a multiuser setup, possibly using a new version of pyNCS.
I see that, by default, the ENABLE_NETWORK_INPUT enables a TCP client that connects to the output TCP server. Is this intentional? If so, does this mean that I can send events to that server so that they are sent to the Dynap-se device?
Thanks,
Emre
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
it is good to know that you are working on a new pyNCS version for the Dynap-se.
In cAER we have NETWORK_INPUT and OUTPUT modules for event streaming. In addition, you can remotely control the status of the modules and the biases when connecting to the port 4040 port using "./utils/caerctl/caer-ctl" or the "javaFX2" gui. Note that server settings are always stored in the caer-config.xml file that is created automatically (with default values) once you close cAER (control-c) after the first run.
1. Control Server:
SSH control server setting, it is always enable and does not require any flag during compilation.
/// from caer-config.html , by default the sshs server is listening only for local connections
<node name="server" path="/server/">
<attr key="backlogSize" type="short">5</attr>
<attr key="ipAddress" type="string">127.0.0.1</attr>
<attr key="portNumber" type="int">4040</attr>
</node>
// just change the ip to the local ip of the machine. This will allow any local machine to connect to the control server. Note that you can use the "TAB" to navigate the XML tree of options. Once you have modified this caer-config.xml file save it and start cAER. The options will be automatically loaded at start-up. In here we also have a python client for interacting with the caer-control server https://svn.code.sf.net/p/jaer/code/scripts/python/cAER_utils/imagers_characterization/caer_communication.py (we used it for controlling cAER during image sensors characterization, not sure weather this is updated now)
2. Network streaming of events
For streaming event data via network you can use the modules (from main_dynapse.c)
TCP output can be enabled by using the flag -DENABLE_NETWORK_OUTPUT=1 when running cmake. UDP output will need to be inserted manually at this stage.
These modules will stream events from one instance of cAER to another one. In other words, you can have one cAER program running on a machine that is directly connected via USB to the Dynap-se board and stream events to another instance if cAER running somewhere else (this works for all event types: spikes, polarity, frame, special events, point1d/2d etc..), and for all supported devices (Dynap-se, DVS128, DAVIS240C etc..). Network modules can use TCP or UDP protocols.
The receiver cAER program will run in the mainloop the functions container = caerInputNetTCP(moduleID) or container = caerInputNetUDP(moduleID). This will allow the container to be filled with event data as if they would come from the usb (container = caerInputDYNAPSEFX2(1)).
Again, the settings for these modules can be changed via the sshs control interface and they will be stored in the caer-config.xml file.
// change IP to a local ip if you wish to stream to another machine in the lan
// the default port for data stream is 777
<node name="5-NetTCPServerOutput" path="/1/5-NetTCPServerOutput/">
<attr key="backlogSize" type="short">5</attr>
<attr key="concurrentConnections" type="short">5</attr>
<attr key="excludeHeader" type="bool">false</attr>
<attr key="ipAddress" type="string">127.0.0.1</attr>
<attr key="maxBytesPerPacket" type="int">0</attr>
<attr key="portNumber" type="short">7777</attr>
<attr key="shutdown" type="bool">true</attr>
<attr key="validEventsOnly" type="bool">false</attr>
</node>
At this stage it is a bit odd the way in which we have to declare modules in the main_dynapse.c program, however we are working a new release of cAER that is addressing this issue with plug-and-play and graphical interface capabilities.
Let us know if you have troubles, and keep us posted with your pyNCS integration. Definitely an interesting development!
all the best,
Federico
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks Federico. This is helpful information. I may have misunderstood the meaning of input server. I was hoping that I could stream events to the dynapse from the network. Is there any way I can stream events to Dynapse through the network, i.e. how we used to do it with the pyAEX server?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
one way of sending stimulations via network would be to use a module that is able to inject spikes to the Dynap-se board. For example, the spikegenerator module or a modified version of that. Via network, you can change the sshs property of the spikegenerator module (it is by default enabled with the dynapse compilation). https://inilabs.com/support/hardware/user-guide-dynap-se/#h.96mbx221ja77
So that the spikegenerator will inject spikes accordingly to your settings.
However, what you would want to do here is a new cAER module that will stream to the USB what is receive via network. You can have a look at this link on how to write your own filter/application https://inilabs.com/support/software/caer/#h.lk31wtmkvdha (this docs need improvement). You could start by modifying the spike_gen.c into your pyAex Server.
sorry for the short answer, if you have more questions we could organize a quick call tomorrow or so.
best,
Federico
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
After looking into the code I find that the easiest way is to extend the output server. The server can read data sent by clients throught the libuv read callbacks and process the data to be sent to the chip in a way similar to gen_spikes.c.
I'm assuming AEDAT 3.1 data at the client. Is there a function for easily converting AEDAT 3.1 to dynapse format?
Also, is there a buffer on the dynapse board?
Would you be available for sync at 16:00 or 17:00 ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
In the meantime, the right way would be to use libuv to read the callbacks and process the data to be sent to the chip (you are right). Use a thread as it is done in the gen_spike.c, in that way it will be faster. Of course, you will be limited by the USB 2.0 bandwith. On the Dynap-se board we have fifos in the logic for in and out spikes.
In libcaer there are few functions that you can use to send data to the usb. I would suggest you use the function:
as this uses packet to send to the usb and it is faster.
The input data format is described in the manual (https://lh5.googleusercontent.com/Fw4hO1u_Il4DfoqMLQsbyEllGIgecUMbFiNUlFar41bWgs8brCrmU5pp17nuv4Jp95t0StXcTZZRV2qZOpc5DkhbYMIN7jKsnGwNCJzfVFC0-Pis4Z1Ho2j0vfrH1ZS3EA) and libcaer exposes few simple functions:
Please don't change the input/output modules. These are intended for generic event I/O, and should never know anything about devices or specific applications/formats.
The clean way to do what you want is the following:
- your application sends Spike event packes, in AEDAT 3.1 format, to any of cAER's network input modules
- the network input module parses this as any other event packet data, and makes it available to the rest of cAER (other modules)
- then you have one module that takes this Spike data and does what you want with it, which in this case would basically be wait for the ISI to pass and send Spikes, very similar to what the SpikeGenerator module does
This kind of generic implementation is nice because then you can use this with any kind of Spike events that cAER is processing, for example you could have one Dynap-SE send its spikes to another, or have a file being played back to a Dynap-SE.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
@Luca I understand your concern. In fact I will not change anything on the caer source code but add a separate module. We'll try to stick to that generic I/O idea as much as possible.
Just to confirm: The network input is a client. Therefore it can only receive packets from the server it is connected to. Does this mean that my application should be a TCP server, streaming events to input network. I see well how to do this by processing the container returned by the TCP module.
@Federico: When streaming events to the Dynapse, which module is reponsible for timing? Ideally this woul dbe the dynapse FPGA, but I'm guessing that is not the case.
Thanks,
Emre
Last edit: Emre Neftci 2017-03-31
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ideally the FPGA would be responsible for the timing (I agree), however at the moment the logic does not support that (Luca has been adapting the logic from the DVS to the Dynap-se, and it still in progress).
At this stage, the timing of the input spikes is calculated in the gen_spike.c. In fact, we now adjust the time at which we send spikes depending on how much time it took to the previous usb transaction to complete (the precision is in the order of hundreds of microsecond, and spike trains can go up to 200 herz approximatively).
Alternatively, we were thinking of developing a logic module (in the FPGA) that serves as a generic stimulator so that we would not need to use USB bandwith to stream spikes. This module should be generic enough to allow to send poisson,gaussian, and regular spike trains. This solution would be good also because it would free lots of USB bandwith as the USB data will be high level commands, and not single spikes. However, we did not implement it yet as it is not clear how a generic stimulator would look like (we only have been thinking of it, and experimenting in sw with the gen_spike.c).
The filter should read spikes from a container and sequence them the the dynapse using gen_spike.c like code.
I found out the issue with getting TCPInput to read the events: it expects timestamps to increase. Can you confirm this? if this is the case, how can I send aerdata-ISI? Do you suggest creating a new event type?
One other question: is USB writing multi-thread enabled? i.e. can multiple threads write on the Dynapse devices without corruption?
Thanks,
Emre
Last edit: Emre Neftci 2017-04-03
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, timestamps should increase, as packets get generated by slicing time (ie. one packet contains all events within a certain time frame). You can have events with the same timestamp, but at some point it must increase. Just start at timestamp 0, and then add the ISI each time, so if Event0_TS = 0, and ISI between Spike0 and 1 is 122, Event1_TS = 122, if between Spike1 and 2 its 33, then Event2_TS = 155, and so on.
I'm not sure if a new event type is warranted, you can encode ISI by advancing the timestamp properly.
Regaring USB, yes, libusb guarantees that calls to send control transfers can happen in multiple threads without special regards having to be taken.
Hope this helps, have a nice day, Luca.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
New pyNCS version, in progress with simplified configuration
git@github.com:nmi-lab/pyNCS.git
New pyCAERdynapse, a cAER dynapse api for pyNCS:
git@github.com:nmi-lab/pyCAERdynapse.git
For the moment, the test scripts allows to monitor and configure parameters through pyNCS using cAER. Setting connectivity is in the works, as well as sending events.
Concerning sending events: can I used the control server for setting the connectivity (SRAMs and CAMs)?
Concerning sending events: I successfully used the ring buffer to get events. For the moment, I'll use gen_spikes.c to sequence them using the software workaround. A student here might help in implementing a buffer on the dynapse FPGA that can sequence events according to the time of the events. Did you make any progress on this side?
It might be useful to set up a conference call to discuss the details here.
Thanks,
Emre
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am glad to see this progress. We are now celebrating easter holidays here (we will be out of office till the 18th of April).
The control server is not used to set the connectivity as it is only used to set the parameters (biases) and to control all the module's parameters, exposed via the sshs system.
The connectivity SRAM/CAM is set using low level libcaer functions that are directly called via cAER. However, we can easily write a module that wrap these functions and expose them to the control server. In any case this will require to send lots of data via network, as the configurations are many, that is why so far we did not implement such a module (however it might work just fine).
We can set up a call Wed 19th April late afternoon (Zurich time) or any day after the 19th.
thank you for sharing your code, we will definitely test it in Capo Caccia.
all the best,
Federico
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear Developers,
I'm trying to get the TCP Input to work with Dynap-se for a multiuser setup, possibly using a new version of pyNCS.
I see that, by default, the ENABLE_NETWORK_INPUT enables a TCP client that connects to the output TCP server. Is this intentional? If so, does this mean that I can send events to that server so that they are sent to the Dynap-se device?
Thanks,
Emre
Hi Emre,
it is good to know that you are working on a new pyNCS version for the Dynap-se.
In cAER we have NETWORK_INPUT and OUTPUT modules for event streaming. In addition, you can remotely control the status of the modules and the biases when connecting to the port 4040 port using "./utils/caerctl/caer-ctl" or the "javaFX2" gui. Note that server settings are always stored in the caer-config.xml file that is created automatically (with default values) once you close cAER (control-c) after the first run.
1. Control Server:
SSH control server setting, it is always enable and does not require any flag during compilation.
/// from caer-config.html , by default the sshs server is listening only for local connections
<node name="server" path="/server/">
<attr key="backlogSize" type="short">5</attr>
<attr key="ipAddress" type="string">127.0.0.1</attr>
<attr key="portNumber" type="int">4040</attr>
</node>
// just change the ip to the local ip of the machine. This will allow any local machine to connect to the control server. Note that you can use the "TAB" to navigate the XML tree of options. Once you have modified this caer-config.xml file save it and start cAER. The options will be automatically loaded at start-up. In here we also have a python client for interacting with the caer-control server https://svn.code.sf.net/p/jaer/code/scripts/python/cAER_utils/imagers_characterization/caer_communication.py (we used it for controlling cAER during image sensors characterization, not sure weather this is updated now)
2. Network streaming of events
For streaming event data via network you can use the modules (from main_dynapse.c)
caerOutputNetTCPServer(8, 2, spike, special)
caerOutputNetUDP(9, 2, spike, special)
TCP output can be enabled by using the flag -DENABLE_NETWORK_OUTPUT=1 when running cmake. UDP output will need to be inserted manually at this stage.
These modules will stream events from one instance of cAER to another one. In other words, you can have one cAER program running on a machine that is directly connected via USB to the Dynap-se board and stream events to another instance if cAER running somewhere else (this works for all event types: spikes, polarity, frame, special events, point1d/2d etc..), and for all supported devices (Dynap-se, DVS128, DAVIS240C etc..). Network modules can use TCP or UDP protocols.
The receiver cAER program will run in the mainloop the functions container = caerInputNetTCP(moduleID) or container = caerInputNetUDP(moduleID). This will allow the container to be filled with event data as if they would come from the usb (container = caerInputDYNAPSEFX2(1)).
Again, the settings for these modules can be changed via the sshs control interface and they will be stored in the caer-config.xml file.
// change IP to a local ip if you wish to stream to another machine in the lan
// the default port for data stream is 777
<node name="5-NetTCPServerOutput" path="/1/5-NetTCPServerOutput/">
<attr key="backlogSize" type="short">5</attr>
<attr key="concurrentConnections" type="short">5</attr>
<attr key="excludeHeader" type="bool">false</attr>
<attr key="ipAddress" type="string">127.0.0.1</attr>
<attr key="maxBytesPerPacket" type="int">0</attr>
<attr key="portNumber" type="short">7777</attr>
<attr key="shutdown" type="bool">true</attr>
<attr key="validEventsOnly" type="bool">false</attr>
</node>
The binary format for the data stream is described here https://inilabs.com/support/software/fileformat/#h.kbta1pm6k3qt
At this stage it is a bit odd the way in which we have to declare modules in the main_dynapse.c program, however we are working a new release of cAER that is addressing this issue with plug-and-play and graphical interface capabilities.
Let us know if you have troubles, and keep us posted with your pyNCS integration. Definitely an interesting development!
all the best,
Federico
Thanks Federico. This is helpful information. I may have misunderstood the meaning of input server. I was hoping that I could stream events to the dynapse from the network. Is there any way I can stream events to Dynapse through the network, i.e. how we used to do it with the pyAEX server?
Hi Emre,
one way of sending stimulations via network would be to use a module that is able to inject spikes to the Dynap-se board. For example, the spikegenerator module or a modified version of that. Via network, you can change the sshs property of the spikegenerator module (it is by default enabled with the dynapse compilation). https://inilabs.com/support/hardware/user-guide-dynap-se/#h.96mbx221ja77
So that the spikegenerator will inject spikes accordingly to your settings.
However, what you would want to do here is a new cAER module that will stream to the USB what is receive via network. You can have a look at this link on how to write your own filter/application https://inilabs.com/support/software/caer/#h.lk31wtmkvdha (this docs need improvement). You could start by modifying the spike_gen.c into your pyAex Server.
sorry for the short answer, if you have more questions we could organize a quick call tomorrow or so.
best,
Federico
Thanks for the quick answer. We'll start coding the cAER module today and we can sync tomorrow morning.
all the best,
Emre
Hi Federico,
After looking into the code I find that the easiest way is to extend the output server. The server can read data sent by clients throught the libuv read callbacks and process the data to be sent to the chip in a way similar to gen_spikes.c.
I'm assuming AEDAT 3.1 data at the client. Is there a function for easily converting AEDAT 3.1 to dynapse format?
Also, is there a buffer on the dynapse board?
Would you be available for sync at 16:00 or 17:00 ?
Hi Emre,
yes, we can chat on monday 16:00 is ok for me.
In the meantime, the right way would be to use libuv to read the callbacks and process the data to be sent to the chip (you are right). Use a thread as it is done in the gen_spike.c, in that way it will be faster. Of course, you will be limited by the USB 2.0 bandwith. On the Dynap-se board we have fifos in the logic for in and out spikes.
In libcaer there are few functions that you can use to send data to the usb. I would suggest you use the function:
bool caerDynapseSendDataToUSB (caerDeviceHandle handle, const uint32_t ∗ data, size_t numConfig)
as this uses packet to send to the usb and it is faster.
The input data format is described in the manual (https://lh5.googleusercontent.com/Fw4hO1u_Il4DfoqMLQsbyEllGIgecUMbFiNUlFar41bWgs8brCrmU5pp17nuv4Jp95t0StXcTZZRV2qZOpc5DkhbYMIN7jKsnGwNCJzfVFC0-Pis4Z1Ho2j0vfrH1ZS3EA) and libcaer exposes few simple functions:
• struct caer_dynapse_info caerDynapseInfoGet (caerDeviceHandle handle)
• bool caerDynapseWriteSramWords (caerDeviceHandle handle, const uint16_t ∗ data, uint32_t baseAddr, uint32_t numWords)
• bool caerDynapseWriteSram (caerDeviceHandle handle, uint16_t coreId, uint32_t neuronId, uint16_t virtualCoreId, bool sx, uint8_t dx, bool sy, uint8_t dy, uint16_t sramId, uint16_t destinationCore)
• bool caerDynapseSendDataToUSB (caerDeviceHandle handle, const uint32_t ∗ data, size_t numConfig)
• bool caerDynapseWriteCam (caerDeviceHandle handle, uint32_t preNeuronAddr, uint32_t postNeuron Addr, uint32_t camId, int16_t synapseType)
• uint32_t caerDynapseGenerateCamBits (uint32_t preNeuronAddr, uint32_t postNeuronAddr, uint32_t camId, int16_t synapseType)
have a look in the libcaer documentation for more infos (https://github.com/inilabs/libcaer/blob/master/docs/libcaer_api_manual.pdf)
I suggest you open a github branch for this pyAEX server implementation. We will be able to review it in depth and contribute to it.
Please don't change the input/output modules. These are intended for generic event I/O, and should never know anything about devices or specific applications/formats.
The clean way to do what you want is the following:
- your application sends Spike event packes, in AEDAT 3.1 format, to any of cAER's network input modules
- the network input module parses this as any other event packet data, and makes it available to the rest of cAER (other modules)
- then you have one module that takes this Spike data and does what you want with it, which in this case would basically be wait for the ISI to pass and send Spikes, very similar to what the SpikeGenerator module does
This kind of generic implementation is nice because then you can use this with any kind of Spike events that cAER is processing, for example you could have one Dynap-SE send its spikes to another, or have a file being played back to a Dynap-SE.
@Luca I understand your concern. In fact I will not change anything on the caer source code but add a separate module. We'll try to stick to that generic I/O idea as much as possible.
Just to confirm: The network input is a client. Therefore it can only receive packets from the server it is connected to. Does this mean that my application should be a TCP server, streaming events to input network. I see well how to do this by processing the container returned by the TCP module.
@Federico: When streaming events to the Dynapse, which module is reponsible for timing? Ideally this woul dbe the dynapse FPGA, but I'm guessing that is not the case.
Thanks,
Emre
Last edit: Emre Neftci 2017-03-31
ideally the FPGA would be responsible for the timing (I agree), however at the moment the logic does not support that (Luca has been adapting the logic from the DVS to the Dynap-se, and it still in progress).
At this stage, the timing of the input spikes is calculated in the gen_spike.c. In fact, we now adjust the time at which we send spikes depending on how much time it took to the previous usb transaction to complete (the precision is in the order of hundreds of microsecond, and spike trains can go up to 200 herz approximatively).
Alternatively, we were thinking of developing a logic module (in the FPGA) that serves as a generic stimulator so that we would not need to use USB bandwith to stream spikes. This module should be generic enough to allow to send poisson,gaussian, and regular spike trains. This solution would be good also because it would free lots of USB bandwith as the USB data will be high level commands, and not single spikes. However, we did not implement it yet as it is not clear how a generic stimulator would look like (we only have been thinking of it, and experimenting in sw with the gen_spike.c).
If you have any request on the logic please write it here, I am sure Luca can help you with that.
BTW: the vhdl logic we are currently using is open source and available here: https://svn.code.sf.net/p/jaer/code/devices/logic/SystemLogic2/Spartan6_Dynapse/
Last edit: Federico Corradi 2017-04-01
Hi,
I forked the caer here with the spike2dynapse module/filter
https://github.com/nmi-lab/caer
The filter should read spikes from a container and sequence them the the dynapse using gen_spike.c like code.
I found out the issue with getting TCPInput to read the events: it expects timestamps to increase. Can you confirm this? if this is the case, how can I send aerdata-ISI? Do you suggest creating a new event type?
One other question: is USB writing multi-thread enabled? i.e. can multiple threads write on the Dynapse devices without corruption?
Thanks,
Emre
Last edit: Emre Neftci 2017-04-03
Yes, timestamps should increase, as packets get generated by slicing time (ie. one packet contains all events within a certain time frame). You can have events with the same timestamp, but at some point it must increase. Just start at timestamp 0, and then add the ISI each time, so if Event0_TS = 0, and ISI between Spike0 and 1 is 122, Event1_TS = 122, if between Spike1 and 2 its 33, then Event2_TS = 155, and so on.
I'm not sure if a new event type is warranted, you can encode ISI by advancing the timestamp properly.
Regaring USB, yes, libusb guarantees that calls to send control transfers can happen in multiple threads without special regards having to be taken.
Hope this helps, have a nice day, Luca.
All,
Here is a short update:
New pyNCS version, in progress with simplified configuration
git@github.com:nmi-lab/pyNCS.git
New pyCAERdynapse, a cAER dynapse api for pyNCS:
git@github.com:nmi-lab/pyCAERdynapse.git
For the moment, the test scripts allows to monitor and configure parameters through pyNCS using cAER. Setting connectivity is in the works, as well as sending events.
Concerning sending events: can I used the control server for setting the connectivity (SRAMs and CAMs)?
Concerning sending events: I successfully used the ring buffer to get events. For the moment, I'll use gen_spikes.c to sequence them using the software workaround. A student here might help in implementing a buffer on the dynapse FPGA that can sequence events according to the time of the events. Did you make any progress on this side?
It might be useful to set up a conference call to discuss the details here.
Thanks,
Emre
Hi Emre,
I am glad to see this progress. We are now celebrating easter holidays here (we will be out of office till the 18th of April).
The control server is not used to set the connectivity as it is only used to set the parameters (biases) and to control all the module's parameters, exposed via the sshs system.
The connectivity SRAM/CAM is set using low level libcaer functions that are directly called via cAER. However, we can easily write a module that wrap these functions and expose them to the control server. In any case this will require to send lots of data via network, as the configurations are many, that is why so far we did not implement such a module (however it might work just fine).
We can set up a call Wed 19th April late afternoon (Zurich time) or any day after the 19th.
thank you for sharing your code, we will definitely test it in Capo Caccia.
all the best,
Federico