/sci/ - Science & Math

Okay /sci/, I'm giving each of you the opportunity to make a big advancement in the field of machine learning. I started this project off as a short curiosity, and three days later it is taking over my life. I feel like I'm extremely close to solving this, and the possibilities are tantalizing, but I've felt like success was right around the corner for a couple days now. The only way I see out is to dump this project on someone else, in hopes that they can either finish it themselves or give some helpful feedback.

There's this thing in machine learning called "The Vanishing Gradient Problem." I would explain what it is, but if you don't already know then you probably won't be able to help here. I'm trying to get around vanishing gradients by using logical error signals instead of gradient based ones. In other words, I start out at the bottom of the network, compare the output to the ideal output, and then determine whether each of the output neurons should have put out a "bigger" or "smaller" value. Instead of propagating the gradient, I propagate this "bigger" or "smaller" signal up the network.

Not only can I quickly train arbitrarily deep networks this way, but the computation is made orders of magnitude more efficient. Learning AND and OR gates works great. The trouble is that I'm having a hard time achieving nonlinear separability. I'm using a modified step function (f(x) = -1 for x <= 0, and f(x) = 1 for x > 0) so I *ought* to be able to learn XOR since I have a nonlinear activation function, but for some reason it still doesn't work.

Could someone please try to figure this out? You can take all the credit.

Here's the code:
https://gist.github.com/anonymous/e02d15e82975f9aa5b18831a7dff5a56