autotune NN

[Todd Morrison]

The first cross validation in the NN Autotune impl needs some default layers:

void GNeuralNet::autoTune(GMatrix& features, GMatrix& labels)
{
// Try a plain-old single-layer network
size_t hidden = std::max((size_t)4, (features.cols() + 3) / 4);
Holder<GNeuralNet> hCand0(new GNeuralNet());
Holder<GNeuralNet> hCand1;
double scores[2];
scores[0] = hCand0.get()->crossValidate(features, labels, 2);
scores[1] = 1e308;

should be:

void GNeuralNet::autoTune(GMatrix& features, GMatrix& labels)
{
// Try a plain-old single-layer network
size_t hidden = std::max((size_t)4, (features.cols() + 3) / 4);
Holder<GNeuralNet> hCand0(new GNeuralNet());
Holder<GNeuralNet> hCand1;
double scores[2];
hCand0->addLayer(new GLayerClassic(FLEXIBLE_SIZE, hidden));
scores[0] = hCand0.get()->crossValidate(features, labels, 2);
scores[1] = 1e308;

Else the cross validate will error out with layer enforcement.

Have faith in chaos,
Todd

[Mike Gashler]

Thank you! I have pushed this fix into the repository.

[Todd Morrison]

You might want to consider drifting momentum and learning rate for auto-tune.
This adds compute time but ends up being needed if MSE minimization is required in automated workflows.

[Mike Gashler]

Agreed. However, I have somewhat lost my motivation to work on auto-tune. I thought it was a good idea several years ago, but it ended up being so slow with neural networks that I never really used it. Maybe it might be more tolerable if it gave better intermediate feedback and allowed the user to provide optional hints about which parameters to sweep. I think to do it right, it should detect the presence of a GPGPU and utilize it, and employ various heuristics based on the size of the training data to pick suitable layer types and sizes.