Neuromorphic computing : SNN + Event camera
1. SNN for object detection/recognition
• Overview
• Vehicles with battery such as Electric Vehicle (EV) require applications running with low energy. However, recent applications using deep neural network (DNN) consume huge energy, eventually, they might shorten driving distance.
• In order to solve this problem, we are developing vehicle applications by using Spiking Neural Network and/or event camera due to their low energy consumption
• For the moment, we realized pedestrian detection by SNN on event camera data
• Description
• In exchange for low energy consumption, handling event data is more difficult than conventional sensor data because of its asynchronous and sparse signals
• However, our SNN can deal with such signals directory and process in real time
• As a application for object detection with SNN + event camera, we targeted on pedestrian detection surround vehicle
• In order to increase detection accuracy, we newly developed neuromorphic visual attention mechanism and inserted it into SNN, shown in the figure below
• Achievement
• Thanks to above neuromorphic visual attention mechanism, we obtained a better results than SOTAs using DNN and SNN
• Especially, we achieved remarkable detection performance in noisy situation
2. ANN2SNN
• Overview
• One of the major issues in SNN is difficulty of training
• Due to asynchronous propagation, SNN cannot be trained by back propagation
• There are many researches to overcome this issue, and one of the solution is conversion from ANN to SNN (ANN2SNN)
• It allows to train by backpropagation like ANN, then after training, it converts ANN’s synchronous network to asynchronous one, it means SNN
• Thanks to this conversion, we can compute neural network less energy than ANN
• However, in compensation for such fuss-free conversion, performance decrease after conversion
• Description
• At first, we implemented gesture recognition by using ANN2SNN, and observed that computation cost and energy consumption change dynamically
• Training set is composed of several patterns of gesture taken by event camera
• Achievement
• We achieved same level of recognition performance as conventional ANN, it means we could keep performance even after conversion
• Also we investigated the performance shift caused by the conversion.
• As we expected, the deeper network becomes, the more performance shift (decrease) exists
3. SNN on Neuromorphic hardware
• Overview
• For computing SNN, we need to use a neuromorphic hardware because regular hardware is not well suited for sparsely and asynchronous data processing
• In contrast, porting SNN to the neuromorphic hardware requires some techniques and the limitation of hardware such as maximum number of neurons and synapses should be considered
• That is why there are not so many researches of “SNN on neuromorphic hardware”
• Description
• We succeeded to port our SNN (with visual attention mechanism) on one of the representative neuromorphic hardware named “SpiNNaker (SpiNN-3)”
• As a benchmark, SNN of SOTA was also ported and we compared energy consumption between two
• Achievement
• In Runtime, our SNN achieved slightly less energy consumption than SOTA
• As described above section, our object detection performance is also better than SOTA
4. Data representation of event data
• Overview
• Event camera has several advantages such as asynchronously, sparsely, high speed and dynamic range.
• Especially, capturing signal sparsely contributes a lot on low energy consumption because if there is no event (brightness change), no signal exists, it means that energy consumption of camera itself is very low
• However, from image data processing point of view, it is difficult to handle this data because it is not frame based
• As another point, as event data is spike (binary signal), it is less rich than conventional imager whose signal has gradated intensity level. Therefore, when we develop object detection on event data, the performance decrease compared with for example on RGB data in general
• In order to make the most of event data, we are developing data representation of event data
• Description
• For feeding event data to Deep Neural Network, at first, we convert event data to frame data, then, in order to give richer information on the frame data, number of event accumulations and time stamp information were added on each pixels
• About time stamp information, we assigned pixel value that is determined by how sooner the event appeared
• Achievement
• Compared with conventional methods that make binary frame data, we improve F1 score in object detection by using above data representation
