在过去的几天里，我一直在调试我的NN，但我找不到问题。

我已经创建了用于识别MNIST数据集图像的多层感知器的完整原始实现。

网络似乎学会了，因为火车循环测试后数据精度高于94%精度。我对损失函数有问题——一段时间后，当测试/val精度达到~76%时，它开始增加。

有人能检查我的前进/后退数学，告诉我我的损失函数是否正确实现，或者建议可能出了什么问题吗？

NN结构：

• 输入层：758个节点，（每像素1个节点）
• 隐藏层1:300个节点
• 隐藏层2:75个节点
• 输出层：10个节点

NN激活功能：

• 输入层-

NN损失函数：

• 类别交叉熵

完整的CLEAN代码可在此处作为Jupyter Notebook获得。

神经网络向前/向后传递：

def train(self, features, targets):
        n_records = features.shape[0]

        # placeholders for weights and biases change values
        delta_weights_i_h1 = np.zeros(self.weights_i_to_h1.shape)
        delta_weights_h1_h2 = np.zeros(self.weights_h1_to_h2.shape)
        delta_weights_h2_o = np.zeros(self.weights_h2_to_o.shape)
        delta_bias_i_h1 = np.zeros(self.bias_i_to_h1.shape)
        delta_bias_h1_h2 = np.zeros(self.bias_h1_to_h2.shape)
        delta_bias_h2_o = np.zeros(self.bias_h2_to_o.shape)

        for X, y in zip(features, targets):
            ### forward pass
            # input to hidden 1
            inputs_to_h1_layer = np.dot(X, self.weights_i_to_h1) + self.bias_i_to_h1
            inputs_to_h1_layer_activated = self.activation_ReLU(inputs_to_h1_layer)

            # hidden 1 to hidden 2
            h1_to_h2_layer = np.dot(inputs_to_h1_layer_activated, self.weights_h1_to_h2) + self.bias_h1_to_h2
            h1_to_h2_layer_activated = self.activation_ReLU(h1_to_h2_layer)

            # hidden 2 to output
            h2_to_output_layer = np.dot(h1_to_h2_layer_activated, self.weights_h2_to_o) + self.bias_h2_to_o
            h2_to_output_layer_activated = self.softmax(h2_to_output_layer)

            # output
            final_outputs = h2_to_output_layer_activated 

            ### backpropagation
            # output to hidden2
            error = y - final_outputs
            output_error_term = error.dot(self.dsoftmax(h2_to_output_layer_activated))

            h2_error = np.dot(output_error_term, self.weights_h2_to_o.T)
            h2_error_term = h2_error * self.activation_dReLU(h1_to_h2_layer_activated)

            # hidden2 to hidden1
            h1_error = np.dot(h2_error_term, self.weights_h1_to_h2.T) 
            h1_error_term = h1_error * self.activation_dReLU(inputs_to_h1_layer_activated)

            # weight & bias step (input to hidden)
            delta_weights_i_h1 += h1_error_term * X[:, None]
            delta_bias_i_h1 = np.sum(h1_error_term, axis=0)

            # weight & bias step (hidden1 to hidden2)
            delta_weights_h1_h2 += h2_error_term * inputs_to_h1_layer_activated[:, None]
            delta_bias_h1_h2 = np.sum(h2_error_term, axis=0)

            # weight & bias step (hidden2 to output)
            delta_weights_h2_o += output_error_term * h1_to_h2_layer_activated[:, None]
            delta_bias_h2_o = np.sum(output_error_term, axis=0)

        # update the weights and biases     
        self.weights_i_to_h1 += self.lr * delta_weights_i_h1 / n_records
        self.weights_h1_to_h2 += self.lr * delta_weights_h1_h2 / n_records
        self.weights_h2_to_o += self.lr * delta_weights_h2_o / n_records
        self.bias_i_to_h1 += self.lr * delta_bias_i_h1 / n_records
        self.bias_h1_to_h2 += self.lr * delta_bias_h1_h2 / n_records
        self.bias_h2_to_o += self.lr * delta_bias_h2_o / n_records

激活功能实现：

def activation_ReLU(self, x):
    return x * (x > 0)

def activation_dReLU(self, x):
    return 1. * (x > 0)

def softmax(self, x):
    z = x - np.max(x)
    return np.exp(z) / np.sum(np.exp(z))

def dsoftmax(self, x):
    # TODO: vectorise math
    vec_len = len(x)
    J = np.zeros((vec_len, vec_len))
    for i in range(vec_len):
        for j in range(vec_len):
            if i == j:
                J[i][j] = x[i] * (1 - x[j])
            else:
                J[i][j] = -x[i] * x[j]
    return J

损失函数实现：

def categorical_cross_entropy(pred, target): 
    return (1/len(pred)) * -np.sum(target * np.log(pred))