[D] Neural Network Implementation
Hello reddit,
I’m following this lecture series: https://www.youtube.com/watch?v=SGZ6BttHMPw&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH and am trying to implement a neural network from scratch.
This is my forward pass:
res_1 =
np.dot
(W_1.T, x) + b1
res_1_activation = sigmoid(res_1)
output =
np.dot
(W_2.T, res_1_activation) + b2
output_activation = sigmoid(output)
This is my backward pass:
grad_pre_output = -(y[i] - output_activation)
grad_W_2 =
np.dot
(np.expand_dims(res_1_activation, axis=1),np.expand_dims(grad_pre_output, 1).T)
grad_b2 = np.expand_dims(grad_pre_output, 1)
grad_post_1 =
np.dot
(W_2, np.expand_dims(grad_pre_output, axis=1))
grad_pre_1 =np.multiply(grad_post_1.T, d_sigmoid(res_1))
grad_W_1 =
np.dot
(np.expand_dims(x, axis=1), grad_pre_1)
grad_b1 = grad_pre_1.T
W_1 = np.add(W_1, lr * ((grad_W_1) - lamda * 2 * W_1))
W_2 = np.add(W_2, lr * ((grad_W_2) - lamda * 2 * W_2))
b1 = np.add(b1, np.reshape(grad_b1, (grad_b1.shape[0],)))
b2 = np.add(b2, np.reshape(grad_b2, (grad_b2.shape[0],)))
I think this implementation should work, but when I run it for >2 epochs all the output neurons get saturated and always return a value of 1. I tried using L2 regularization but it still saturates. Can anyone please tell me what I am doing wrong?
Any help would be appreciated.
Thank you!
submitted by /u/cronoz30
[link] [comments]