File structure?

[Mato]

Hi there,

I just installed waffless and I want to test its neuratnet learning on simple
example of teaching it or. I created mydata.csv file like this:

0;0;0;

0;1;1;

1;0;1;

1;1;1;

I created arff file from it:

@RELATION mydata.csv

@ATTRIBUTE attr1 {0;0;0;,0;1;1;,1;0;1;,1;1;1;}

@DATA

0;0;0;

0;1;1;

1;0;1;

1;1;1;

and when i run the command "waffles_learn train mydata.arff orthogonalize
neuralnet -addlayer 4 -addlayer 4 > model.twt"

I got answer: "The network has not been trained"

It is understandable because i have no idea how to tell waffles which data are
inputs(first two columns) and which are outputs(last column). I look for
manual but i cant find anything. Can you please help me? Or tell me where i
can find the manual.

Thanks,

Mato

[Nobody/Anonymous]

The problem appears to be that you have used semicolons to separate the values
instead of commas. Your ARFF file should look like this:

@RELATION mydata.csv

@ATTRIBUTE attr1 continuous

@ATTRIBUTE attr2 continuous

@ATTRIBUTE attr3 continuous

@DATA

0,0,0

0,1,1

1,0,1

1,1,1

Waffles always assumes that the last attribute is the class label. If you want
to predict values for a different attribute, you must first swap columns to
move that attribute to the end. Also, you can specify the number of label
dimensions using the "-labeldims " flag. For example, if you use "-labeldims
2", then the last 2 attributes are the label vector that it will learn to
predict.

[Mato]

Thanks for tip, it was helpful. I assume that now my network is trained, i
have file model.twt but when i run command:

waffles_learn evaluate model.twt test.arff

the output will be:

Changing the network structure after some training has begun is not yet
supported.

But I dont want to change the network. I want to test trained network in file
model.twt with data in file test.arff. What am I doing wrong?

[Mike Gashler]

Hmm, I'd need to debug this to know what's going on. Could you post or send me
model.twt and test.arff? At the very least, I'd like to make the messages more
helpful.

Here are some other things you can try if you don't want to wait for me to
debug it:

1- Do other models besides neuralnet have the same problem? Maybe try
"waffles_learn train mydata.arff decisiontree > model.twt" or "waffles_learn
train mydata.arff knn 3 > model.twt"

2- I've made several changes to the neural net class lately. You could try
getting the latest code from the Subversion repository and see if it works
better.

[Mato]

I want my network to learn sum.

From file mydata.csv:

0,0,0
0,1,1
1,0,1
1,1,2
2,1,3
1,3,4
2,2,4
2,0,2
3,1,4
4,0,4
3,0,3
0,2,2

...I generated mydata.arff (waffles_transform import mydata.csv >
mydata.arff):

@RELATION mydata.csv

@ATTRIBUTE attr1  real
@ATTRIBUTE attr2  real
@ATTRIBUTE attr3  real

@DATA
0,0,0
0,1,1
1,0,1
1,1,2
2,1,3
1,3,4
2,2,4
2,0,2
3,1,4
4,0,4
3,0,3
0,2,2

... then I make neural network model.twt (waffles_learn train mydata.arff
orthogonalize neuralnet -addlayer 4 -addlayer 4 > model.twt):

0 o
1 afterRelation:o
2 valueCounts:l 3
3 i 0
3 i 0
3 i 0
1 beforeRelation:o
2 valueCounts:l 3
3 i 0
3 i 0
3 i 0
1 afterLabels:i 1
1 beforeLabels:i 1
1 modeler:o
2 squashFinalOutputs:b t
2 incIters:i 20
2 incMem:i 400
2 momentum:d 0
2 learningDecay:d 1
2 learningRate:d 0.1
2 weights:l 37
3 d 2.9543845745375
3 d 1.3331898670041
3 d 2.0456842046331
3 d 1.5853003315985
3 d 1.681697825459
3 d 0.31139267605695
3 d 0.10179331714022
3 d 0.21638499708133
3 d 0.18546727289795
3 d 0.2663529954966
3 d 0.84355648883779
3 d 0.38186401513661
3 d 0.26861726972072
3 d 0.33173248686992
3 d 0.10280057893877
3 d 0.42392357758311
3 d 0.31923343547518
3 d 0.21192850861526
3 d 0.096172840486453
3 d 0.11373002723472
3 d 0.41042977856231
3 d 0.31738023701244
3 d 0.37433625369286
3 d 0.30049730257104
3 d 0.20207659223504
3 d 0.20127742286974
3 d 0.14266461368197
3 d 0.16415294286764
3 d -0.10729899034519
3 d 0.19003371026227
3 d 0.15095800785934
3 d -0.032055730671777
3 d 0.084018301310757
3 d 0.018620609658671
3 d 0.10998365557925
3 d 0.021017744147733
3 d -0.0088956091713833
2 hiddenLayers:l 2
3 i 4
3 i 4
2 featureDims:i 2
2 labelDims:i 1
2 class:s GNeuralNet
1 label:o
2 before:o
3 valueCounts:l 1
4 i 0
2 valueCap:i 12
2 class:s GOrthogonalizeNominalValues
1 feature:o
2 before:o
3 valueCounts:l 2
4 i 0
4 i 0
2 valueCap:i 12
2 class:s GOrthogonalizeNominalValues
1 class:s GTransformedLearner

... and when I generate test data from test.csv:

1,3,0
2,0,0
3,1,0
0,4,0

... with command (waffles_transform import test.csv > test.arff):

@RELATION test.csv

@ATTRIBUTE attr1  real
@ATTRIBUTE attr2  real
@ATTRIBUTE attr3  real

@DATA
1,3,0
2,0,0
3,1,0
0,4,0

and then finnaly i want to evaluate network with command

waffles_learn evaluate model.twt test.arff

I have this result:

Changing the network structure after some training has begun is not yet
supported.

(

this reminds me an mistake in manual:

http://waffles.sourceforge.net/command/learn.html -> train -> evaluate -> example:

Example: evaluate testdata.arff model.twt

when i try this, error is obvious:

waffles_learn evaluate test.arff model.twt

Expected a level number at the start of line 1

)

the same result with evaluateonepattern:

waffles_learn evaluateonepattern model.twt "1,2"

Changing the network structure after some training has begun is not yet
supported.

BTW: Is there a way to add files to post as attachment? This should make my
post more readable.

[Mike Gashler]

It turns out there was a bug in the NeuralNet class which occurs when you load
a network with hidden layers (sorry). The bug was discovered a few weeks ago,
and has already been fixed. A unit test was also added to ensure that it
doesn't happen again. So, the best solution now is to get the latest code.

If you use Linux, here's the command to get the latest code:

sudo aptitude install subversion

svn co svn://gashler.com/waffles

If you use Windows, you'll need to download a Subversion client (like Tortoise
SVN) and check out svn://gashler.com/waffles.

With the current code, your commands work just fine, although you can get
better results if you modify them a little bit. Here are my suggestions:

1- In the new code, the "orthogonalize" filter has been renamed to
"categorize". Anyway, you don't need to use this filter because your
attributes are already real. This filter is for converting nominal values to a
categorical distribution of real values.

2- By default, the neural net uses logistic sigmoid units, which squash
predictions such that it can only predict values between 0 and 1. You can use
the "-dontsquashoutputs" flag to specify that the output layer should not be
squashed, which is more suitable for regression problems like the one you're
doing.

3- Since you're trying to solve a linear problem, adding hidden layers only
gives the neural net more flexibility to overfit the training set. You'll get
better results with this problem just using a single layer of units.

Here's the results I get now:

$ waffles_learn train mydata.arff neuralnet -dontsquashoutputs > model.twt

$ waffles_learn evaluate model.twt test.arff

@RELATION test.csv

@ATTRIBUTE attr1 real

@ATTRIBUTE attr2 real

@ATTRIBUTE attr3 real

@DATA

1,3,4

2,0,2

3,1,4

0,4,4

(Incidentally, this command should work with the Waffles code you already
have, since that bug only occurs when hidden layers are used.)

(And I don't know how to do attachments in this forum.)

Good luck!

-Mike

[Mato]

Thanks very much for helpful tips Mike. Now I have network technically
functional.

For example for summing:

1,3,3.8673894659273

2,0,1.8837559415707

3,1,3.8607651397085

0,4,3.8608422047539

But I want to go forward. I want to teach it a multiplication. I make learning
data:

@RELATION multip.csv

@ATTRIBUTE attr0 real

@ATTRIBUTE attr1 real

@ATTRIBUTE attr2 real

@DATA

1,1,1

1,3,3

1,5,5

2,5,10

2,4,8

2,0,0

3,2,6

3,0,0

3,1,3

4,4,16

4,3,12

4,2,8

5,2,10

5,3,15

5,4,20

I try to teach it without hidden layers, with hidden layers, with up to 10
nodes in each layer, I also try "categorize" before "neuralnet", but results
are allways like this:

(for exapmle: waffles_learn train multip.arff neuralnet -addlayer 4 -addlayer
10 -dontsquashoutputs > multip.twt)

then prediction is not good:

waffles_learn predict multip.twt test.arff

@RELATION test.csv

@ATTRIBUTE attr0 real

@ATTRIBUTE attr1 real

@ATTRIBUTE attr2 real

@DATA

1,3,7.9328524464962

2,0,7.9328281756444

3,1,7.9327739261765

0,4,7.9629649888597

What am I doing wrong? I am only beginner with neural networks. I cant tell if
my train data is too short, or network for this purpose require much more
layers....

But I dont want to take much of your time, I appreciate all your advices you
already gave me.

[Mike Gashler]

Here's the things I'd try:

1- Use more training data. (It should be easy to generate a lot of it with
these problems.)

2- Train longer. Example: waffles_learn train multip.arff neuralnet -addlayer
4 -addlayer 10 -dontsquashoutputs -windowepochs 1000 > multip.twt

Neural nets aren't really ideal for doing arithmetic, although it should work
if you have enough training data, at least in the region of values that are
well-sampled by the training data. If you were to replace the sigmoid units in
the neural net with lots of random math operators, it might be able to find
the exact one that corresponds with the function you are trying to learn. I
think suspect such a network would be very difficult to train, though, due to
problems with local optima. I think this is still an open problem in research.