I've been implementing my own version of the HTM algorithm (based off of Jeff's framework outlined in On Intelligence) in MATLAB for the past month and I'm looking for applications of HTM in motor and robotic control systems for my graduate research. I was wondering if anyone knows of any previous work in using HTMs for motor control?
Thank you,
Tony
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm not sure of any existing applications. Numenta is using Grok (their commercial HTM application) to monitor wind turbine gear box temperatures. It is very good at detecting anomalies. They are also using it to predict energy consumption patterns for use in bidding on energy contracts. Once again, it's good at predicting normal and detecting anything that deviates from normal. Normal being defined as re-occurring patterns in streams of data.
I assume motor control and robotic feedback would be a good use for an HTM. Once trained on 'normal', deviations from normal could be applied as feedback to servos for applying a force to get the mechanical system back into 'normal' range. What is wonderful about HTMs is they learn to correlate many sensor streams that could be complex to understand and to write software for. The more sensor inputs the better, and the HTM will 'auto correlate' them into a pattern. The more often it sees this pattern the better it gets at predicting it.
So if you had many sensors that output 2 dimensional bit arrays, all of these can be fed into HTM regions and correlated by the HTM. It works well on streams of data that are tied by space and time. It also will do 'online learning', so it will continuously learn new patterns and re-enforce common ones to a higher degree.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you for the response. Feedback sensor auto-correlation is something of a Holy Grail in mechatronics and I see a lot of promise from the HTM theory to solve this problem. I believe that this feedforward auto-correlation mechanism for combining sensor data paired with the feedback nature of HTMs to propagate complex, learned motor commands has a lot of promise to further prove the usefulness of HTMs.
I will continue to post on this thread f I decide to implement such a robotic system.
Tony
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The way I hypothesize motor control working in an HTM (I can't find anything from numenta saying how they are handling it) would be as follows:
In addition to the sensory input data, there would also be an output array which would connect to the column(s) in exactly the same way as input data, with the difference that, when connecting columns fired, the connected output bit would be activated and the action would then be taken. Based on the desirability of the action taken, you would then either reinforce the synapses responsible for the output, or weaken them based on the same rules as govern learning for the input synapses. In this way, the output column would learn to be activated by the correct column or columns to predict the behavior that you desire.
For instance, if you had an HTM connected to stock market data, your input streams would be binary representations of the price, volume, your account balance, your current stock position, volatility, whatever other metrics you think might help, and your motor outputs would be, buy now and sell now. At first, training (hopefully on a simulated account), the HTM would take relatively random actions based on input, but by reinforcing the synapses that fire and cause an account increase, and weakening the ones that cause decrease, the output neurons should become connected to those sensory neurons that represent patterns likely to result in good stock purchases.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you for the explanation - I believe that your approach is definitely feasible for control feedback. Also, applying HTM to stock market data and buy/sell orders would be very interesting. Has anyone in the community tried to apply HTMs to Stock Market/Forex data? I'd love to see their results.
Tony
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
My worry is that determining which actions caused "good" or "bad" outcomes might be difficult because they could theoretically be separated in time. If you buy a stock and a while later it goes up, I don't know how to run back the clock and reward or inhibit the synapses that fired to make it happen without keeping a full log of connected states through time.
Ian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well an HTM would be buying in anticipation for the stock to rise at some point in the future - it should really just depend on what time scale you are looking at. I believe that training multiple low level HTMs on different time scales (i.e. months, weeks and minutes) may provide the structure to reward or inhibit the proper synapses on different time scales. I hope that makes some sense lol.
I'd be mostly interested in an HTM that can make micro transactions minute by minute instead of long term.
Tony
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ian said that good/bad outcomes might be seperated in time. That could be solved using prediction.
To determine how good or bad the outcome might be, you could use goal neurons. Each would represent a certain good/bad thing. They would be activated by ROM for simple good/bad things (such as eating and other basic stimuli. More complex basic goals would probably reequire some other mechanism.) The ROM determines what each goal neuron represents. Level of goodness=amount of positive neuron activation-amount of negative neuron activation.
To include the cortex in deciding how good something is, simply use hebbian learning to connect cortical neurons to goal neurons. That way, it will be able to recognize more good/bad stimuli.
To solve the problem Ian talked about, simply allow predictively active neurons to send data to the goal neurons. That way, if a neuron predicts eating food with a 40% chance, the neuron will add 4/10 of its normal "goodness".
Because this system cares about what will happen in the future, it can use its basic goals to create secondary goals. For example, it could add "hunt" as a secondary goal, so it can achieve its basic goal "eat".
Obviously, the goal neurons must cause the cortex to do things considered positive. I'm not sure how this would be done. Maybe dopamine causes focus/obsession with something, which could be done by keeping active neurons active and inactive neurons inactive (generally.) E.g. dopamine could cause the mental state to focus on what it is thinking. That way, if it randomly thinks of walking to where food is, it will keep thinking about that and predict doing that.
Last edit: Casey 2013-04-10
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hey everyone,
I've been implementing my own version of the HTM algorithm (based off of Jeff's framework outlined in On Intelligence) in MATLAB for the past month and I'm looking for applications of HTM in motor and robotic control systems for my graduate research. I was wondering if anyone knows of any previous work in using HTMs for motor control?
Thank you,
Tony
I'm not sure of any existing applications. Numenta is using Grok (their commercial HTM application) to monitor wind turbine gear box temperatures. It is very good at detecting anomalies. They are also using it to predict energy consumption patterns for use in bidding on energy contracts. Once again, it's good at predicting normal and detecting anything that deviates from normal. Normal being defined as re-occurring patterns in streams of data.
I assume motor control and robotic feedback would be a good use for an HTM. Once trained on 'normal', deviations from normal could be applied as feedback to servos for applying a force to get the mechanical system back into 'normal' range. What is wonderful about HTMs is they learn to correlate many sensor streams that could be complex to understand and to write software for. The more sensor inputs the better, and the HTM will 'auto correlate' them into a pattern. The more often it sees this pattern the better it gets at predicting it.
So if you had many sensors that output 2 dimensional bit arrays, all of these can be fed into HTM regions and correlated by the HTM. It works well on streams of data that are tied by space and time. It also will do 'online learning', so it will continuously learn new patterns and re-enforce common ones to a higher degree.
The way Jeff put it: Can I predict the future? Can I predict the actions that I should do to get the future that I want?
Thank you for the response. Feedback sensor auto-correlation is something of a Holy Grail in mechatronics and I see a lot of promise from the HTM theory to solve this problem. I believe that this feedforward auto-correlation mechanism for combining sensor data paired with the feedback nature of HTMs to propagate complex, learned motor commands has a lot of promise to further prove the usefulness of HTMs.
I will continue to post on this thread f I decide to implement such a robotic system.
Tony
The way I hypothesize motor control working in an HTM (I can't find anything from numenta saying how they are handling it) would be as follows:
In addition to the sensory input data, there would also be an output array which would connect to the column(s) in exactly the same way as input data, with the difference that, when connecting columns fired, the connected output bit would be activated and the action would then be taken. Based on the desirability of the action taken, you would then either reinforce the synapses responsible for the output, or weaken them based on the same rules as govern learning for the input synapses. In this way, the output column would learn to be activated by the correct column or columns to predict the behavior that you desire.
For instance, if you had an HTM connected to stock market data, your input streams would be binary representations of the price, volume, your account balance, your current stock position, volatility, whatever other metrics you think might help, and your motor outputs would be, buy now and sell now. At first, training (hopefully on a simulated account), the HTM would take relatively random actions based on input, but by reinforcing the synapses that fire and cause an account increase, and weakening the ones that cause decrease, the output neurons should become connected to those sensory neurons that represent patterns likely to result in good stock purchases.
Ian,
Thank you for the explanation - I believe that your approach is definitely feasible for control feedback. Also, applying HTM to stock market data and buy/sell orders would be very interesting. Has anyone in the community tried to apply HTMs to Stock Market/Forex data? I'd love to see their results.
Tony
There's a paper on using HTM for stock market trading, here:
http://www.ida.liu.se/divisions/aiics/publications/Exjob-2010-Evaluation-Hierarchical-Temporal.pdf
It's based on the older version of the HTM algorithm (before CLA), but the principles are the same.
Also, another one here:
http://bada.hb.se/bitstream/2320/11579/1/Gabrielsson_CIFER_2012.pdf
Tony,
My worry is that determining which actions caused "good" or "bad" outcomes might be difficult because they could theoretically be separated in time. If you buy a stock and a while later it goes up, I don't know how to run back the clock and reward or inhibit the synapses that fired to make it happen without keeping a full log of connected states through time.
Ian
Ian,
Well an HTM would be buying in anticipation for the stock to rise at some point in the future - it should really just depend on what time scale you are looking at. I believe that training multiple low level HTMs on different time scales (i.e. months, weeks and minutes) may provide the structure to reward or inhibit the proper synapses on different time scales. I hope that makes some sense lol.
I'd be mostly interested in an HTM that can make micro transactions minute by minute instead of long term.
Tony
Ian said that good/bad outcomes might be seperated in time. That could be solved using prediction.
To determine how good or bad the outcome might be, you could use goal neurons. Each would represent a certain good/bad thing. They would be activated by ROM for simple good/bad things (such as eating and other basic stimuli. More complex basic goals would probably reequire some other mechanism.) The ROM determines what each goal neuron represents. Level of goodness=amount of positive neuron activation-amount of negative neuron activation.
To include the cortex in deciding how good something is, simply use hebbian learning to connect cortical neurons to goal neurons. That way, it will be able to recognize more good/bad stimuli.
To solve the problem Ian talked about, simply allow predictively active neurons to send data to the goal neurons. That way, if a neuron predicts eating food with a 40% chance, the neuron will add 4/10 of its normal "goodness".
Because this system cares about what will happen in the future, it can use its basic goals to create secondary goals. For example, it could add "hunt" as a secondary goal, so it can achieve its basic goal "eat".
Obviously, the goal neurons must cause the cortex to do things considered positive. I'm not sure how this would be done. Maybe dopamine causes focus/obsession with something, which could be done by keeping active neurons active and inactive neurons inactive (generally.) E.g. dopamine could cause the mental state to focus on what it is thinking. That way, if it randomly thinks of walking to where food is, it will keep thinking about that and predict doing that.
Last edit: Casey 2013-04-10
very interesting thinking guys!
you could hook your htm up (it could be the new perceptron) to motor, and similar sensing makes for similar motor.
its probably alot more than that in us, but thats a good start.
Last edit: Magnus Wootton 2013-05-17