Hello reddit,

I’m following this lecture series: https://www.youtube.com/watch?v=SGZ6BttHMPw&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH and am trying to implement a neural network from scratch.

This is my forward pass:

res_1 = np.dot(W_1.T, x) + b1

res_1_activation = sigmoid(res_1)

output =
np.dot(W_2.T, res_1_activation) + b2

output_activation = sigmoid(output)

This is my backward pass:

grad_pre_output = -(y[i] - output_activation)

grad_W_2 = np.dot(np.expand_dims(res_1_activation, axis=1),np.expand_dims(grad_pre_output, 1).T)

grad_b2 = np.expand_dims(grad_pre_output, 1)

grad_post_1 = np.dot(W_2, np.expand_dims(grad_pre_output, axis=1))

grad_pre_1 =np.multiply(grad_post_1.T, d_sigmoid(res_1))

grad_W_1 = np.dot(np.expand_dims(x, axis=1), grad_pre_1)

grad_b1 = grad_pre_1.T

W_1 = np.add(W_1, lr * ((grad_W_1) - lamda * 2 * W_1))

W_2 = np.add(W_2, lr * ((grad_W_2) - lamda * 2 * W_2))

b1 = np.add(b1, np.reshape(grad_b1, (grad_b1.shape[0],)))

b2 = np.add(b2, np.reshape(grad_b2, (grad_b2.shape[0],)))

I think this implementation should work, but when I run it for >2 epochs all the output neurons get saturated and always return a value of 1. I tried using L2 regularization but it still saturates. Can anyone please tell me what I am doing wrong?

Any help would be appreciated.

Thank you!