AnvaMiba

Thanks for doing this AMA!

In the paper Critical Factors in the Performance of HyperNEAT van den Berg and Whiteson provide some negative results obtained on their implementation of HyperNEAT which call into question the ability of HyperNEAT to learn in complex ("fractured") tasks.

In your response CPPNs Effectively Encode Fracture: A Response to Critical Factors in the Performance of HyperNEAT you refute van den Berg and Whiteson negative results using your own implementation and PicBreader, attributing the negative results to improper hyperparameter selection and other implementation details.

Does this imply that HyperNEAT is fragile and requires lots of fiddling and tweaking of hyperparameters and other implementation details to make it work?

I think this is an important point, because the main attractive point of structural neural evolution like HyperNEAT compared to gradient-based deep learning is that structural neural evolution directly optimizes the network topology and architecture while in conventional deep learning the network topology and architecture are high-dimensional discrete hyperparameters and getting them right is kind of an art. But if HyperNEAT is no less sensitive to high-dimensional hyperparameters and architectural details than conventionan deep learning, then what is its advantage?

Neural networks often achieve state of the art performance on supervised learning tasks where lots of training data is available. However current architectures can't be effectively used in data-limited tasks such one-shot learning for visual classification or most system biology machine learning applications.

These tasks are typically tackled with Bayesian graphical models engineered by humans with considerable domain expertise, but they allow for strong priors that enable learning even with limited amounts of training data.

There is considerable research in combining Bayesian inference with neural networks (e.g. with variational autoencoders and derived models), but, to my knowledge, these neural Bayesian models are still far away in terms of predictive power and reliance on large training sets from traditional domain-engineered Bayesian models.

Do you think that neural networks could be eventually applied in the small-data regime? What are the main obstacles to that?