Sparse distributed input representations

[Ian]

I was reading about Grok and the input streams that it accepts and was fascinated by how the Numenta team handles things like integer and floating point inputs as well as classes and strings as described here: https://www.numenta.com/technology.html under the Sensor Region heading. Will those input encoders be something that this project will try to accomplish as well, or is that outside the current scope?

[David Ragazzi]

Hi Ian,

First of all, welcome!

Our plans include a InputProcessorPlugin that would process any character or number. My idead is use this binany representation as input to network, ie a 4x4 matrix.

For example, the inputs bellow:
1234
5000
100

would become:
0010011010010
1001110001000
0000001100100

or then:
0000
0100
1101
0010

0001
0011
1000
1000

0000
0000
0110
0100

How this would work?

Well, first of all you need ensure that region returns an output (predicted columns) that have a stable set of columns. This is obtained at first time as you can see in 3D visualizer.

For example, if the binary representation of the 1234 number is inputed to the HTM, it will return a given sparse distribution at fisrt interaction (step). However, in the next interactions, this same output (sparse distribution) will change until a time what it become fixed (this initial instability is due to competition between columns to represent something). Once a number conquest a fixed representation, this sparse representation together with its original value (number or caracter) are stored to a dictionary. As network receives new inputs, it compare the output representation with any existing representations in the dictionary. This way, the original values representating these entries in the dictionary are returned to the user.

Whe sould remember that a output can represent more of an number (ie not an exact set of columns), this way we sould base our dictionary research in the columns with more activity and looking in the dictionary representations which their set of columns overlaps/intersects the output.

This is an initial draft proposed by me, we still are discussing these isses while we refactor the project. I created a method called getPredictions() in project that try perform what I'm saying.

[Barry Matt]

Yes we plan to add some methods to at least convert simple numerical inputs into SDR (sparse distributed representation). The link to that Numenta page describes how Grok does it under the section "Sensor region":
"
For example, the [input] vector may consist of 121 bits, where 21 are randomly selected to be 1s and the rest are 0s. Scalar (numerical) values are represented by selecting 21 adjacent bits (called "W") along a variable range of total bits (called "N"). For example, the minimum value is represented by the first 21 bits of the string being 1s, and the maximum value is represented by the last 21 bits of the string being 1s. The values between min and max are represented by selecting 21 adjacent bits in the middle of the string of N bits. Think of a slider widget of width W on a track with N increments. The total number of bits is a variable, which allows Grok to divide the data in different degrees of granularity.
"

We will want to provide something similar in OpenHTM. We can then use it to provide a simple example HTM that can work with a simple sequence of numbers (and serve as nice unit tests).

David, in your example can you describe in more detail how you are going from 1234 to 0010011010010? I am not sure I follow what rule you are using to perform the conversion from value to bits.

[David Ragazzi]

David, in your example can you describe in more detail how you are going from 1234 to 0010011010010? I am not sure I follow what rule you are using to perform the conversion from value to bits.

Hi Barry,

I simply use our old known binary representation of each number for 16-bit numbers. For example:
1 => 0000.0000.0000.0001
2 => 0000.0000.0000.0010
3 => 0000.0000.0000.0011
10,000 => 0010.0111.0001.0000

For alphanumeric stream, InputProcessor would use another method to use ascii number itself:
'1' => 49 => 0000.0000.0011.0001
'2' => 50 => 0000.0000.0011.0010
'A' => 65 => 0000.0000.0100.0001

You should specify to InputProcessor which the type of your input, because it would have use a specific conversion method for each.

The code I'm using:

class Program
  {
    public static void Main(string[] args)
    {
      var integerConverter = new IntegerConverter();
      int[,] bitMap = integerConverter.ConvertToBitMap(1234);
      int value = integerConverter.ConvertFromBitMap(bitMap);

      Console.WriteLine(value);

      Console.Write("Press any key to continue . . . ");
      Console.ReadKey(true);
    }
  }

  public interface IConverter
  {
    int[,] ConvertToBitMap(int value);    
    int ConvertFromBitMap(int[,] bitMap);
  }

  public class IntegerConverter : IConverter
  {
    public int[,] ConvertToBitMap(int value)
    {
      // Convert value to binary representation
      string binary = Convert.ToString(value, 2);
      System.Console.WriteLine(binary);

      // Pass over string with binary representation for store it in bit map matrix
      int[,] bitMap = new int[4, 4];
      int c = binary.Length - 1;
      for (var i = 3; i >= 0; i--)
      {
        for (var j = 3; j >= 0; j--)
        {
          if (c >= 0)
          {
            bitMap[i, j] = (binary[c] == '1') ? 1 : 0;
            c--;
          }
          else
          {
            break;
          }
        }
      }

      return bitMap;
    }

    public int ConvertFromBitMap(int[,] bitMap)
    {      
      // Pass over bit map to store binary representation in a string
      string binary = "";
      for (var i = 0; i <= bitMap.GetUpperBound(0); i++)
      {
        for (var j = 0; j <= bitMap.GetUpperBound(1); j++)
        {
          binary += bitMap[i, j];
        }
      }

      // Convert binary representation to value
      int value = Convert.ToInt32(binary, 2);

      return value;
    }
  }

[John Blackburn]

One method to represent numbers is to present them in the form of a bar graph and then feed the bar graph into the neural network just as you would any other picture. Humans often plot graphs to visualize data better so this is a very "human" way of understanding a number. Presumably the output of the net could also be regarded as a bar graph so the prediction the net is considering could be decoded that way.

I agree with some of the comments below. There is no point feeding a net letters in a natural language. There is no way for the net to predict the next letter even if it had human like intelligence. E.g. if you give me a string of, say, Vietnamese letters, I could not predict the next one since I do not speak Vietnamese -- despite my human-like intelligence. A net which does not speak English could predict, say, that a "q" is always followed by a "u" and other trivial deductions but could not learn to understand English.

The only way for an HTM brain to learn a human language is to embody it in a robot body and for it to grow up experiencing the world and using language to encode experiences. Presenting a stream of ASCII characters and hoping the net will learn to talk is pointless.

(Just my $0.02 of course!)

[Ian]

I'll try to dig up the YouTube video where Jeff Hawkins talks about using ascii and binary codes to represent values and that the htm had a harder time recognizing them because, at least in the case of ascii codes, there is no particular meaning for each binary digit. For each field of the input data, they have created a section of the input field where classes (text) is represented just as a random representation and scalar values are represented as a sliding scale as mentioned on the page I listed above.

[David Ragazzi]

As we still don't focus is this question, these information are worth in soon future when thinking of InputProcessorPlugin. Please keep us informed about any discovery or insight.

[David Ragazzi]

I found the white paper which talk about:

"Representations used in HTM are different than say the representations used in ASCII codes. A particular bit in the eight bit ACSII code has no meaning on its
own."

http://www-edlab.cs.umass.edu/cs691jj/hawkins-and-george-2006.pdf

At least in the text, I don't see any problem in ascii representation in binary matrix, Hawkins and George are saying that a unique bit cannot represent nothing, but they don't say the set of 8-bits cannot represent nothing.

[Ian]

Here: http://www.youtube.com/watch?v=iNMbsvK8Q8Y
Is the video where he talks about each bit needing to have its own semantic meaning and ascii in general is the example he uses as a code that doesn't carry bit-specific meaning. That said, I can't really foresee needing to represent individual characters as anything but classes, which he goes on to describe in some detail in the video.

[David Ragazzi]

I saw some parts of the video. What he speaks already is in the white paper, ie only 20% of input is enough. Certainly, this is not useful for a 16-bit ascii representation, as a single bit missing can lead to great errors. However this is due to inibition factor, ie a column in a radius that inhibit neighboors. My propose (I still not tested it) is reduce this inhibition to zero. I can be WRONG, but according to I read in the ultimate numenta HTM paper, this is done for otimization issues. Of course, each region should have its own inhibition rules. For ascii/numbers values, only the first region would have zero inhibition.

EDITED: This post is very old. I posted before hardcoded option be implemented by Barry. With Hardcoded option, similar results can be obtained. But still the overlapping columns "ghost" walks around.

[Ian]

But what is the purpose of representing individual ascii characters to the HTM in the first place? Other than as classes, what could it be learning from acsii input? If you were doing natural language processing, I would think phonemes or maybe even concepts would be a more natural atomic unit than letters. I'm just trying to imagine the input set that would have single letters as part of the data where those letters didn't represent classes.