Yes, this is key part of the project. Once we have CLA re-factored and the CLA data structure frozen it would be fairly easy to save / load the trained HTM network.
In the meantime, it would still be a good idea to add "save trained network" and "load trained network" to the UI and get it to save / load the current data structure, keeping in mind that future version of the HTM may change data structure and your trained network will not be backward compatible.
Also we need to be aware that the "settings" of the HTM or the way it was configured should be saved along with the trained network, so training can continue if desired.
We should have some metadata about the trained network:
what kind of data the network was trained on (video, logfile, http trafic, etc.)
link or URL to more info,
'owner' or 'trainer' of the trained network,
cpu cycles or some indicator of how much cpu time went into training,
info of # of processors or cores,
hash code to indicate a version of trained network,
info about the HTM engine or CLA version
Last edit: Doug King 2013-03-07
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Also we need to be aware that the "settings" of the HTM or the way it was configured should be saved along with the trained network, so training can continue if desired.
I already am providing this. I grouped the config of regions and synapses to an unique class called "NetworkConfig". In this same class, we could store parameters to access the trained data, ie by flat file or database, if the learning is ON or OFF, etc. Ie the NetworkConfig could contains all information about the HTM directly in it or by link to the (remote) data.
Up to this weekend, I think we already could have a IDE that saves and shares networks. :-)
I think I may have started the discussion about the wrong item here - I think the title "Import of Data Sources" is for the idea of connecting the HTM to any data source so we are not stuck with only the generated datasources in the UI.
I will create a couple of other TODO items, one for save/load Settings file, one for HTM Network and one for "Statistics" file. I'll leave this one here for discussing Data Source format and getting it fed into the HTM.
Last edit: Doug King 2013-03-07
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
To connect the HTM to any data-source we should standardize on an input stream. The obvious thing would be to standardize on a video stream, 1 pixel deep, so on/off grid of pixels in uncompressed video format. This can represent any stream of data, not just video, and would be easy to feed to the HTM. It would be up to the user to conform their data to this format. In many cases that would be the hard work, figuring out how to convert multidimensional data to many 1 pixel bitmap streams, but that is the only way you can feed an HTM is with streaming on/off bits in a grid to a region. I have ideas about conversion of various data streams but we can discuss that elsewhere.
There are many image processing tools and techniques all ready available and it would be the most widely accepted format out there for streaming data, something everyone is already familiar with.
For example if you have a varying temperature sensor you would convert temperature value to growing or shrinking circle of white on black.
If you have a gray-scale image, or more complex data that requires 8 bit we could extract the 8 layers out into individual 1 bit files and feed 8 regions of HTMs or 8 separate HTMs in a hierarchy with the 9th HTM receiving input from the lower 8.
So I suppose we need to be able to easily assign each 1bit video stream to a region or something like that.
The issues are:
Simple common streaming format, pixel grid with on/off pixels (1 bit video)
Assigning multiple 1bit sources to multiple regions or HTMs
Transforming 'non video' data into video stream, e.g. temp sensor to bitmap.
Managing the settings for what is connected to what, i.e. the metadata or Settings about multiple regions inputs.
Use cases:
how to transform stock market data to bitmap
how to transform audio to bitmap (i've seen this done in the music player foobar2000 with the spectral display)
how to transform text (ascii) stream or word stream (dictionary table) without converting to a bit-mapped font.
Please anyone / everyone, comment on this. I think this is open for discussion and worthy of more than a simple quick solution from me. I believe I have the right idea here, but the approach or implementation needs to be something that we will not regret in the long run.
My intuition tells me that thinking about data streams this way is the best way to understand HTMs, as they are modeled on biology and our biology deals with data that varies in size/shape over time. We are not good at processing raw data that is not correlated to size/shape changes over time.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
how to transform audio to bitmap (i've seen this done in the music player foobar2000 with the spectral display)
Do you mean applying the Fourier transform?
how to transform text (ascii) stream or word stream (dictionary table) without converting to a bit-mapped font.
Do you want it to read different fonts? You need to render that text somewhere anyway, maybe we could use standard OS text rendering to some kind of render target buffer...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We are already including Math.NET in the CLA and it should have also FFT routines in it that could be used. We don't need to be concerned with this yet, just the plugin architecture. Once we have that, I will try to implement the spectrogram plug-in some day.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As you say, one thing is right: HTM accepts only BitMap inputs (I mean multidimensional array of bits) similar to brain that accepts only a commom representation from senses like vision, hearing, etc.
What I saw from the current state of the project is that the BitMap sub-project already do this for some of these formats. It, for example, converts from Alphanumerics (discrete values which can represents a letter, number, etc) to a Bitmap accepted by HTM and then initialize the HTM in the IDE. With geometrics input, it creates a map of bits for the respective shape, ie it transforms vectorial images to a map of bits accepted by HTM.
An idea would be we use the BitmapProcessor as a indepedent application to users directly convert their data to an accepted format. Or also, we could do that most commom formats (alphanumerics, sound, image, etc) were converted directly by call from IDE (ie continue using BitMapProcessor do this task) beyound the alphanumerics and geometrics types.
Of course, we could offers templates to the user according with her/his needs. One template for alphanumerics would include few regions with a small map of bits, to images an huge map, and so on..
Last edit: David Ragazzi 2013-03-07
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
For now it would be easiest to just allow the user to load a .avi file. It would be up to them to put their images into the AVI or convert their data to 1bit AVI. We would then take that as input.
I like the idea of a BitmapProcessor module that has a plug-in API so any plugin .dll that implements the plugin interface would be loaded and could be selected by the user. The processor plug-in would know about what type of raw data it is consuming and it would transform it into the required format for our HTM.
We could take the existing BitMap sub project and turn it into one or two plug-ins to get started. A plug-in would have a UI (winform) and processor. The UI would be responsible for managing settings, interacting with the user and displaying results. The processor would do the transform from format to HTM.
One of the plug-ins that we would default to as a core feature could be one that consumes .AVI files. That would solve all of our concerns and make it easy for anyone to implement their own plugin processor.
So perhaps all we need to do at this time is come up with a IPlugInBitmapProcessor interface or something like that. Convert the existing BitMap sub-project to be a plug in and call it done. Then we could also create a .AVI video plugin later that would allow anyone with any kind of .AVI data to use it.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Plug-in framework is built in to .NET 4.0 and should be Mono compatible.
Managed Extensibility Framework (MEF) is a composition layer for .NET that improves the flexibility, maintainability and testability of large applications. MEF can be used for third-party plugin extensibility, or it can bring the benefits of a loosely-coupled plugin-like architecture to regular applications.
Status
MEF is a part of the Microsoft .NET Framework, with types primarily under the System.ComponentModel.Composition.* namespaces.
MEF has shipped with .NET 4.0
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I believe all features of .NET 4.0 that are not UI are 100% compatible with Mono, so yes. Perhaps all the UI (forms, asp.net, etc. are compatible too, but I'm not sure)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think the most important to the project in this moment is define which part do what. This is important from a documentation view, architectural, etc.
The BitMapProcessor was perfoming 2 roles in same time (create inputs and transform this inputs in map of bits), which makes confusing sometimes the understanding of how data are processed.
I've worked hard this week in order to separate the code that makes data that are generated from the code that do the reverse. This is very important, because we need wave a project where people can have a lab to create and test its inputs, where an another project (plugin) that only allows convert this raw input (image, text, sound, etc) to a acceptable format to HTM, ie a map (multidim array) of bits. These 2 projects cannot be in a same project.
I hope in the next weeks, I have good news!
Last edit: David Ragazzi 2013-03-15
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Progress report:
I've decoupled InputGenerator from IDE, so IDE will only accepts map of bits as input (ie not more integrated with sentences and animation processor). If the user want they can create manualy the file or create test data from InputGenerator accessing (future) Tools menu.
I will implemented a HelloWord.txt as default example in a Data folder together with the HelloWorld.xml HTM setting files.
Last edit: David Ragazzi 2013-03-15
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Nice job David. I saw your check-in. The IDE reads directly from a file. You create a stream processor and now InputGenerator is a studio/lab feature that creates the file for the users experimental input data.
So now all we need to do is implement a plug-in interface, and then anyone (not just core team) will be able to create new lab features to generate / transform input. Or any stand-alone application can do this too.
Should we close this ticket (make a separate ticket for the plug-in feature) or keep it open till the plug-in is done?
Last edit: Doug King 2013-03-18
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So now all we need to do is implement a plug-in interface, and then anyone (not just core team) will be able to create new lab features to generate / transform input. Or any stand-alone application can do this too.
Exactly, this is the point! Now people are free to create their own set of data, not matter its source. It's just understand the format of the file (each matrix input separate by a blank line) and either create manually or by software (anyway, we need document this format in documentation) or then use the plug-in for known formats.
It still lacks reading database, ie read from a table instead of a file, but anyway this is the same concept. So I think is close this ticket and open a new to plug-in.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi
I have a question: why do we store input bits in the text file? If the input is big enough, that will cause a significant processing slowdown and storage space overuse...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Nick, depends what you want and your needs. The advantage of the text file is that it contains pre-processed input which allow directly handle it independent of how it was generated.
For big bit maps, maybe it would be interesting use the (future) InputProcessor to transform a stream to input format directly, not using pre-processed files itself.
We aren't restrain input to text files, we're extending to them.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, this is key part of the project. Once we have CLA re-factored and the CLA data structure frozen it would be fairly easy to save / load the trained HTM network.
In the meantime, it would still be a good idea to add "save trained network" and "load trained network" to the UI and get it to save / load the current data structure, keeping in mind that future version of the HTM may change data structure and your trained network will not be backward compatible.
Also we need to be aware that the "settings" of the HTM or the way it was configured should be saved along with the trained network, so training can continue if desired.
We should have some metadata about the trained network:
Last edit: Doug King 2013-03-07
I already am providing this. I grouped the config of regions and synapses to an unique class called "NetworkConfig". In this same class, we could store parameters to access the trained data, ie by flat file or database, if the learning is ON or OFF, etc. Ie the NetworkConfig could contains all information about the HTM directly in it or by link to the (remote) data.
Up to this weekend, I think we already could have a IDE that saves and shares networks. :-)
See the prototype bellow.
Last edit: David Ragazzi 2013-03-07
I think I may have started the discussion about the wrong item here - I think the title "Import of Data Sources" is for the idea of connecting the HTM to any data source so we are not stuck with only the generated datasources in the UI.
I will create a couple of other TODO items, one for save/load Settings file, one for HTM Network and one for "Statistics" file. I'll leave this one here for discussing Data Source format and getting it fed into the HTM.
Last edit: Doug King 2013-03-07
To connect the HTM to any data-source we should standardize on an input stream. The obvious thing would be to standardize on a video stream, 1 pixel deep, so on/off grid of pixels in uncompressed video format. This can represent any stream of data, not just video, and would be easy to feed to the HTM. It would be up to the user to conform their data to this format. In many cases that would be the hard work, figuring out how to convert multidimensional data to many 1 pixel bitmap streams, but that is the only way you can feed an HTM is with streaming on/off bits in a grid to a region. I have ideas about conversion of various data streams but we can discuss that elsewhere.
There are many image processing tools and techniques all ready available and it would be the most widely accepted format out there for streaming data, something everyone is already familiar with.
For example if you have a varying temperature sensor you would convert temperature value to growing or shrinking circle of white on black.
If you have a gray-scale image, or more complex data that requires 8 bit we could extract the 8 layers out into individual 1 bit files and feed 8 regions of HTMs or 8 separate HTMs in a hierarchy with the 9th HTM receiving input from the lower 8.
So I suppose we need to be able to easily assign each 1bit video stream to a region or something like that.
The issues are:
Use cases:
how to transform stock market data to bitmap
how to transform audio to bitmap (i've seen this done in the music player foobar2000 with the spectral display)
how to transform text (ascii) stream or word stream (dictionary table) without converting to a bit-mapped font.
Please anyone / everyone, comment on this. I think this is open for discussion and worthy of more than a simple quick solution from me. I believe I have the right idea here, but the approach or implementation needs to be something that we will not regret in the long run.
My intuition tells me that thinking about data streams this way is the best way to understand HTMs, as they are modeled on biology and our biology deals with data that varies in size/shape over time. We are not good at processing raw data that is not correlated to size/shape changes over time.
Do you mean applying the Fourier transform?
Do you want it to read different fonts? You need to render that text somewhere anyway, maybe we could use standard OS text rendering to some kind of render target buffer...
Yes, exactly. FFT. I have posted links to screen shots, FFT libraries, etc. here: http://www.one-tab.com/page/kGKGRL_YRpuqTt___VWG2w
We are already including Math.NET in the CLA and it should have also FFT routines in it that could be used. We don't need to be concerned with this yet, just the plugin architecture. Once we have that, I will try to implement the spectrogram plug-in some day.
Nice thoughts..
As you say, one thing is right: HTM accepts only BitMap inputs (I mean multidimensional array of bits) similar to brain that accepts only a commom representation from senses like vision, hearing, etc.
What I saw from the current state of the project is that the BitMap sub-project already do this for some of these formats. It, for example, converts from Alphanumerics (discrete values which can represents a letter, number, etc) to a Bitmap accepted by HTM and then initialize the HTM in the IDE. With geometrics input, it creates a map of bits for the respective shape, ie it transforms vectorial images to a map of bits accepted by HTM.
An idea would be we use the BitmapProcessor as a indepedent application to users directly convert their data to an accepted format. Or also, we could do that most commom formats (alphanumerics, sound, image, etc) were converted directly by call from IDE (ie continue using BitMapProcessor do this task) beyound the alphanumerics and geometrics types.
Of course, we could offers templates to the user according with her/his needs. One template for alphanumerics would include few regions with a small map of bits, to images an huge map, and so on..
Last edit: David Ragazzi 2013-03-07
For now it would be easiest to just allow the user to load a .avi file. It would be up to them to put their images into the AVI or convert their data to 1bit AVI. We would then take that as input.
I like the idea of a BitmapProcessor module that has a plug-in API so any plugin .dll that implements the plugin interface would be loaded and could be selected by the user. The processor plug-in would know about what type of raw data it is consuming and it would transform it into the required format for our HTM.
We could take the existing BitMap sub project and turn it into one or two plug-ins to get started. A plug-in would have a UI (winform) and processor. The UI would be responsible for managing settings, interacting with the user and displaying results. The processor would do the transform from format to HTM.
One of the plug-ins that we would default to as a core feature could be one that consumes .AVI files. That would solve all of our concerns and make it easy for anyone to implement their own plugin processor.
So perhaps all we need to do at this time is come up with a IPlugInBitmapProcessor interface or something like that. Convert the existing BitMap sub-project to be a plug in and call it done. Then we could also create a .AVI video plugin later that would allow anyone with any kind of .AVI data to use it.
Plug-in framework is built in to .NET 4.0 and should be Mono compatible.
Managed Extensibility Framework (MEF) is a composition layer for .NET that improves the flexibility, maintainability and testability of large applications. MEF can be used for third-party plugin extensibility, or it can bring the benefits of a loosely-coupled plugin-like architecture to regular applications.
Status
MEF is a part of the Microsoft .NET Framework, with types primarily under the System.ComponentModel.Composition.* namespaces.
MEF has shipped with .NET 4.0
Yes, I've seen that in action. Is it compatible with Mono?
I believe all features of .NET 4.0 that are not UI are 100% compatible with Mono, so yes. Perhaps all the UI (forms, asp.net, etc. are compatible too, but I'm not sure)
I think the most important to the project in this moment is define which part do what. This is important from a documentation view, architectural, etc.
The BitMapProcessor was perfoming 2 roles in same time (create inputs and transform this inputs in map of bits), which makes confusing sometimes the understanding of how data are processed.
I've worked hard this week in order to separate the code that makes data that are generated from the code that do the reverse. This is very important, because we need wave a project where people can have a lab to create and test its inputs, where an another project (plugin) that only allows convert this raw input (image, text, sound, etc) to a acceptable format to HTM, ie a map (multidim array) of bits. These 2 projects cannot be in a same project.
I hope in the next weeks, I have good news!
Last edit: David Ragazzi 2013-03-15
Very nice work. You're right, defining architecture right now is going to pay off later on.
Progress report:
I've decoupled InputGenerator from IDE, so IDE will only accepts map of bits as input (ie not more integrated with sentences and animation processor). If the user want they can create manualy the file or create test data from InputGenerator accessing (future) Tools menu.
I will implemented a HelloWord.txt as default example in a Data folder together with the HelloWorld.xml HTM setting files.
Last edit: David Ragazzi 2013-03-15
Nice job David. I saw your check-in. The IDE reads directly from a file. You create a stream processor and now InputGenerator is a studio/lab feature that creates the file for the users experimental input data.
So now all we need to do is implement a plug-in interface, and then anyone (not just core team) will be able to create new lab features to generate / transform input. Or any stand-alone application can do this too.
Should we close this ticket (make a separate ticket for the plug-in feature) or keep it open till the plug-in is done?
Last edit: Doug King 2013-03-18
Exactly, this is the point! Now people are free to create their own set of data, not matter its source. It's just understand the format of the file (each matrix input separate by a blank line) and either create manually or by software (anyway, we need document this format in documentation) or then use the plug-in for known formats.
It still lacks reading database, ie read from a table instead of a file, but anyway this is the same concept. So I think is close this ticket and open a new to plug-in.
Hi
I have a question: why do we store input bits in the text file? If the input is big enough, that will cause a significant processing slowdown and storage space overuse...
Hi Nick, depends what you want and your needs. The advantage of the text file is that it contains pre-processed input which allow directly handle it independent of how it was generated.
For big bit maps, maybe it would be interesting use the (future) InputProcessor to transform a stream to input format directly, not using pre-processed files itself.
We aren't restrain input to text files, we're extending to them.