#71 Reading Byte-Data in Big-Endian-Format for Machine Learning

[Manfred Zindel]

Hello. First thank you, David and Daniel, for your answers to my ticket #70.

Now there is another problem, that has something to do with introductory machine learning, and I'm wondering whether I will be able to perform it in FMSLogo.

I downloaded two identically structured groups of 4 files each, MNIST and FashionMNIST, which are meant for experiments or Benchmarks when training and testing a neural network with one hidden layer (the simplest type of deep learning).

All of these files have a header (some with a length of 16 bytes, some with a length of 8 Bytes, all containing a "magic number" to mark the file-type and the others with al length of 16 byte including additionally width and height (both 28) of grayscale images to follow. There are 60.000 training-images and 10.000 test-images of 28 x 28 pixels as well as sorter labels. All are binary files with bytes, each of them containing mostly images or labels. 28x 28 greayscale levels 0 to 255.

What I have to do is reading all the images and normalize the grayscale levels from 0 .. 255 to 0 .. 1, as the input-layer of a neural network uses to unterstand, process and recognize the images like that. If I can do this, I could program a neural network with one hidden layer (simplest form of deep learning) just in FMSLogo without any library like Torch, PyTorch or TensorFlow. This could prove that FMSLogo can go to the borderline of introductory programming for young people.

MNINST is an often used (some say overused) application for introducing beginners into machine learning, and as a Benchmark it is said to be too optimistic (97% success). FashionMNIST is a newer alternative using 10 fashion objects instead of 10 decimal digits. Both examples are structured the same, so that you need only one program for both.

Here's what my AI-bot (Opera AI) and Github say about the file data:

Details of the MNIST IDX-file format
• First 4 Bytes (32 Bit): Magic Number
◦ For Image Data the value is 2051 (in Hex: 0x00000803).
◦ For Label-Data the value is 2049 (in Hex: 0x00000801).
• They are followed by 4-Byte-Integers describing the Dimensions with and height (landscape):
◦ For Images: Number of Images (60.000), number of rows (28), number of columns (28).
◦ For Labels: number of labels.
• All Integer Values are in Big Endian.

Now here are my questions:

1. Can I read the files in FMSLogo as binary files using READCHARS, byte after byte, and how?

2. The FMSLogo-Documentation says: "If the read stream is a file that was opened in binary mode, then READCHARS reads number bytes from the file and outputs a word where each character uses the code point of the corresponding byte (0 - 255)." Now what is a code point?

3. What I need is a number from range 0 .. 255 as a grayscale level, taken from Big-Endian.
   How can I extract the grayscale level as an integer from a byte coded in Big-Endian and transfer them into an input layer (a list of two-dimensional arrays of single precision numbers 28 x 28) by normalizing them to 0 .. 1?

[David Costanzo]

1) Can I read the files in FMSLogo as binary files using READCHARS, byte after byte, and how?

As of FMSLogo 8.0.0, you can process binary files. Since you ask for the data "byte after byte" you might find READCHAR easier than READCHARS. With READCHARS, you can read a four-byte big-endian integer value with

LOCALMAKE "word READCHARS 4
SHOW (BITOR
  LSHIFT ASCII ITEM 1 :word 24
  LSHIFT ASCII ITEM 2 :word 16
  LSHIFT ASCII ITEM 3 :word 8
         ASCII ITEM 4 :word)

Of course, the file must be opened in binary mode.

2) ...Now what is a code point?

Wikipedia has a good description of code point.

READCHARS returns a word (characters). You want the code points (the byte values). For example, suppose READCHARS 1 outputs "A, the Unicode LATIN CAPITAL LETTER A. The code point for this character is 65. You can use ASCII to get the code point of a character.

3) What I need is a number from range 0 .. 255 as a grayscale level, taken from Big-Endian.
How can I extract the grayscale level as an integer from a byte coded in Big-Endian and transfer them into an input layer (a list of two-dimensional arrays of single precision numbers 28 x 28) by normalizing them to 0 .. 1?

Endianness doesn't apply to one-byte values (0..255). You can just use ASCII READCHAR to get the next byte in the image an place it to the correct location in the array. Or you can read an entire row into an array with:

LISTTOARRAY MAP [ASCII?] REDUCE "SENTENCE READCHARS 28

REPEAT that 28 times and you've got a 2D array.

I don't know what you mean "by normalizing them to 0..1". Are you normalizing single byte values or the entire 2D array? Where does the list of 2D arrays come in?

[Manfred Zindel]

The data are meant for the input layer of a neural network. Usually intensities in the input layer are "normalized" to 1. Than means 0 remains 0 and 255 becomes 1, generally by dividing the intensity (the grayscale level) by 255. The input layer remains unchanged, because the weights of the other layers reflect the learning process. If there's more than one layer, deep learning (abstraction) is possible.'

Meanwhile I experimented wioth RAWASCII instead of ASCII and got all the results I needed.

What I thought is that we should not use libraries like PyTorch or TensorFlow, even if we could, but write the program explicitly, to enable clear understanding what a neural network really does, and this kind of understanding the "blackbox" is valuable for stuents.

[Manfred Zindel]

What I should add is, that I need no two-dimensional array for the input. A one-dimensional list or array of"neurons (single bytes for grayscale levels) is sufficient, as the neural network "knows" that it is subdivided into 28-byte-rows, but the learning process changing the "weights" (synapses) in the hidden layer doesn't take care of it and can be done with a one-dimensional input - only the visualization of the process and of the results refer to it. The hidden layer (only one in this case) and the output-layer are still one-dimensional. I have no Idea, how long the learning process lasts in FMSLogo (in Lua it takes several minutes), but if it is reasonable, I could try a better neural network with two hidden layers, still without PyTorch or Tensorflow. Every additional hidden layer creates mor nonlinear abstraction (the big models wordwide have dozens or even hundreds of layers with those formidable Nvidia processors.

Do you think, FMSLogo (and KISS.go, which still goes through the last tests before being published) will be able to participate in teaching AI?

Lua (instead of Python or C#) could be the next step to higher levels starting with 17 years of age in school. Nevertheless, there are not so many efforts to achieve the power of Lua even for FMSLogo. What do you think about itas a Logo-enthusiast?

[David Costanzo]

I have no Idea, how long the learning process lasts in FMSLogo (in Lua it takes several minutes), but if it is reasonable, I could try a better neural network with two hidden layers, still without PyTorch or Tensorflow

If you get something that works but is too slow, you could post it and ask for help to make it faster. I expect that whatever deep learning code you write would also work in UCBLogo, so you could also try tapping into their expertise, as well.

Do you think, FMSLogo ... will be able to participate in teaching AI?

I know almost nothing about AI, but I've heard that deep learning is computationally expensive, especially during training. So you might have selected the single aspect of AI that's hardest for Logo. There are other areas of AI that don't require as much computational power that would be more within FMSLogo's reach. For those areas, I expect that FMSLogo could be used to help demystify AI for non-professional programmers by giving hands-on exercises for some types of machine learning. For example, I'm hopeful that you could write a phishing classifier in FMSLogo. And if the students don't learn AI, maybe they'll learn to be better at discerning phishing emails as humans, which is also is a valuable life skill.

Lua (instead of Python or C#) could be the next step to higher levels starting with 17 years of age in school. Nevertheless, there are not so many efforts to achieve the power of Lua even for FMSLogo. What do you think about itas a Logo-enthusiast?

I don't know anything about Lua and very little about Python. When compared to Python, Logo has several disadvantages with respect to performance:

1. Python is compiled to byte code before execution. FMSLogo caches some aspects of evaluation, but I doubt it's nearly as fast as executing byte code.
2. Python has a large development community, some of which is dedicated to making things run faster. Logo has no such community.
3. Python has rudimentary support for multi-threading, enabling you to use as many cores as your computer has. FMSLogo can only use one core.
4. Python libraries are written in native code when performance matters. FMSLogo can do this with DLLCALL, but then it wouldn't be a pure Logo implementation. Python programmers don't care about this purity. I expect that PyTorch can use you GPU if you have one, but (pure) FMSLogo can't.

If Lua is compiled to byte code, supports multi-threading, or has a large community that has optimized it, then FMSLogo would be at a similar disadvantage.