今天这篇博文针对Assignment3的全连接网络作业，对前面学习的内容进行一些总结
在前面的作业中我们建立神经网络的操作比较简单，也不具有模块化的特征，在A3作业中，引导我们对前面的比如linear layer,Relu layer,Loss layer以及dropout layer(这个前面课程内容未涉及 但是在cs231n中有出现),以及梯度下降不同方法(SGD,SGD+Momentum,RMSprop,Adam)等等进行模块化的实现
Linear与Relu单层实现
class Linear(object): @staticmethod def forward(x, w, b): """ Computes the forward pass for an linear (fully-connected) layer. The input x has shape (N, d_1, ..., d_k) and contains a minibatch of N examples, where each example x[i] has shape (d_1, ..., d_k). We will reshape each input into a vector of dimension D = d_1 * ... * d_k, and then transform it to an output vector of dimension M. Inputs: - x: A tensor containing input data, of shape (N, d_1, ..., d_k) - w: A tensor of weights, of shape (D, M) - b: A tensor of biases, of shape (M,) Returns a tuple of: - out: output, of shape (N, M) - cache: (x, w, b) """ out = None out = x.view(x.shape[0],-1).mm(w)+b cache = (x, w, b) return out, cache @staticmethod def backward(dout, cache): """ Computes the backward pass for an linear layer. Inputs: - dout: Upstream derivative, of shape (N, M) - cache: Tuple of: - x: Input data, of shape (N, d_1, ... d_k) - w: Weights, of shape (D, M) - b: Biases, of shape (M,) Returns a tuple of: - dx: Gradient with respect to x, of shape (N, d1, ..., d_k) - dw: Gradient with respect to w, of shape (D, M) - db: Gradient with respect to b, of shape (M,) """ x, w, b = cache dx, dw, db = None, None, None db = dout.sum(dim = 0) dx = dout.mm(w.t()).view(x.shape) dw = x.view(x.shape[0],-1).t().mm(dout) return dx, dw, db class ReLU(object): @staticmethod def forward(x): """ Computes the forward pass for a layer of rectified linear units (ReLUs). Input: - x: Input; a tensor of any shape Returns a tuple of: - out: Output, a tensor of the same shape as x - cache: x """ out = None out = x.clone() out[out<0] = 0 cache = x return out, cache @staticmethod def backward(dout, cache): """ Computes the backward pass for a layer of rectified linear units (ReLUs). Input: - dout: Upstream derivatives, of any shape - cache: Input x, of same shape as dout Returns: - dx: Gradient with respect to x """ dx, x = None, cache dx = dout.clone() dx[x<0] = 0 return dx class Linear_ReLU(object): @staticmethod def forward(x, w, b): """ Convenience layer that performs an linear transform followed by a ReLU. Inputs: - x: Input to the linear layer - w, b: Weights for the linear layer Returns a tuple of: - out: Output from the ReLU - cache: Object to give to the backward pass """ a, fc_cache = Linear.forward(x, w, b) out, relu_cache = ReLU.forward(a) cache = (fc_cache, relu_cache) return out, cache @staticmethod def backward(dout, cache): """ Backward pass for the linear-relu convenience layer """ fc_cache, relu_cache = cache da = ReLU.backward(dout, relu_cache) dx, dw, db = Linear.backward(da, fc_cache) return dx, dw, db 从上面的代码我们可以看到，针对linear与relu层，我们可以将前向传播与反向传播分开实现，具体过程在上一篇我的博文中有讨论：https://www.cnblogs.com/dyccyber/p/17764347.html
不同的是我们要对x进行一个reshape，将其转换为N*D的矩阵，才能与矩阵进行点积
在分别实现了linear与relu之后，因为神经网络的架构往往是在linear之后立马加入一个relu层，所以我们可以再建立一个linear-relu class，将这两个层的前向与反向传播合并
LossLayer实现
def svm_loss(x, y): """ Computes the loss and gradient using for multiclass SVM classification. Inputs: - x: Input data, of shape (N, C) where x[i, j] is the score for the jth class for the ith input. - y: Vector of labels, of shape (N,) where y[i] is the label for x[i] and 0 <= y[i] < C Returns a tuple of: - loss: Scalar giving the loss - dx: Gradient of the loss with respect to x """ N = x.shape[0] correct_class_scores = x[torch.arange(N), y] margins = (x - correct_class_scores[:, None] + 1.0).clamp(min=0.) margins[torch.arange(N), y] = 0. loss = margins.sum() / N num_pos = (margins > 0).sum(dim=1) dx = torch.zeros_like(x) dx[margins > 0] = 1. dx[torch.arange(N), y] -= num_pos.to(dx.dtype) dx /= N return loss, dx def softmax_loss(x, y): """ Computes the loss and gradient for softmax classification. Inputs: - x: Input data, of shape (N, C) where x[i, j] is the score for the jth class for the ith input. - y: Vector of labels, of shape (N,) where y[i] is the label for x[i] and 0 <= y[i] < C Returns a tuple of: - loss: Scalar giving the loss - dx: Gradient of the loss with respect to x """ shifted_logits = x - x.max(dim=1, keepdim=True).values Z = shifted_logits.exp().sum(dim=1, keepdim=True) log_probs = shifted_logits - Z.log() probs = log_probs.exp() N = x.shape[0] loss = (-1.0/ N) * log_probs[torch.arange(N), y].sum() dx = probs.clone() dx[torch.arange(N), y] -= 1 dx /= N return loss, dx 上面损失函数层我们在之前已经实现过，具体实现需要用到一些矩阵微分的知识，具体可以参考这两篇博文：
http://giantpandacv.com/academic/算法科普/深度学习基础/SVM Loss以及梯度推导/
https://blog.csdn.net/qq_27261889/article/details/82915598
多层神经网络
关于多层神经网络，首先是类的初始化定义，我们可以看神经网络的结构{linear - relu - [dropout]} x (L - 1) - linear - softmax，有L-1个linear层与relu层与dropout层的组合，最后再以linear-softmax的结构结束输出结果，初始化我们要遍历每个隐藏层，初始化权重矩阵与偏置项，最后再去初始化最后一个linear层，要注意矩阵的维度
class FullyConnectedNet(object): """ A fully-connected neural network with an arbitrary number of hidden layers, ReLU nonlinearities, and a softmax loss function. For a network with L layers, the architecture will be: {linear - relu - [dropout]} x (L - 1) - linear - softmax where dropout is optional, and the {...} block is repeated L - 1 times. Similar to the TwoLayerNet above, learnable parameters are stored in the self.params dictionary and will be learned using the Solver class. """ def __init__(self, hidden_dims, input_dim=3*32*32, num_classes=10, dropout=0.0, reg=0.0, weight_scale=1e-2, seed=None, dtype=torch.float, device='cpu'): """ Initialize a new FullyConnectedNet. Inputs: - hidden_dims: A list of integers giving the size of each hidden layer. - input_dim: An integer giving the size of the input. - num_classes: An integer giving the number of classes to classify. - dropout: Scalar between 0 and 1 giving the drop probability for networks with dropout. If dropout=0 then the network should not use dropout. - reg: Scalar giving L2 regularization strength. - weight_scale: Scalar giving the standard deviation for random initialization of the weights. - seed: If not None, then pass this random seed to the dropout layers. This will make the dropout layers deteriminstic so we can gradient check the model. - dtype: A torch data type object; all computations will be performed using this datatype. float is faster but less accurate, so you should use double for numeric gradient checking. - device: device to use for computation. 'cpu' or 'cuda' """ self.use_dropout = dropout != 0 self.reg = reg self.num_layers = 1 + len(hidden_dims) self.dtype = dtype self.params = {} ############################################################################ # TODO: Initialize the parameters of the network, storing all values in # # the self.params dictionary. Store weights and biases for the first layer # # in W1 and b1; for the second layer use W2 and b2, etc. Weights should be # # initialized from a normal distribution centered at 0 with standard # # deviation equal to weight_scale. Biases should be initialized to zero. # ############################################################################ # Replace "pass" statement with your code last_dim = input_dim for n ,hidden_dim in enumerate(hidden_dims): i = n+1 self.params['W{}'.format(i)] = torch.zeros(last_dim, hidden_dim, dtype=dtype,device = device) self.params['W{}'.format(i)] += weight_scale*torch.randn(last_dim, hidden_dim, dtype=dtype,device= device) self.params['b{}'.format(i)] = torch.zeros(hidden_dim, dtype=dtype,device= device) last_dim = hidden_dim i+=1 self.params['W{}'.format(i)] = torch.zeros(last_dim, num_classes, dtype=dtype,device = device) self.params['W{}'.format(i)] += weight_scale*torch.randn(last_dim, num_classes, dtype=dtype,device= device) self.params['b{}'.format(i)] = torch.zeros(num_classes, dtype=dtype,device= device) # When using dropout we need to pass a dropout_param dictionary to each # dropout layer so that the layer knows the dropout probability and the mode # (train / test). You can pass the same dropout_param to each dropout layer. self.dropout_param = {} if self.use_dropout: self.dropout_param = {'mode': 'train', 'p': dropout} if seed is not None: self.dropout_param['seed'] = seed 其次，我们可以定义save与load函数，对模型参数等等进行存储与加载：
def save(self, path): checkpoint = { 'reg': self.reg, 'dtype': self.dtype, 'params': self.params, 'num_layers': self.num_layers, 'use_dropout': self.use_dropout, 'dropout_param': self.dropout_param, } torch.save(checkpoint, path) print("Saved in {}".format(path)) def load(self, path, dtype, device): checkpoint = torch.load(path, map_location='cpu') self.params = checkpoint['params'] self.dtype = dtype self.reg = checkpoint['reg'] self.num_layers = checkpoint['num_layers'] self.use_dropout = checkpoint['use_dropout'] self.dropout_param = checkpoint['dropout_param'] for p in self.params: self.params[p] = self.params[p].type(dtype).to(device) print("load checkpoint file: {}".format(path)) 最后是前向传播与反向传播的实现，这里直接使用前面基础的linear与relu的前向与反向传播即可，注意一下神经网络的结构，不要把顺序搞错即可
def loss(self, X, y=None): """ Compute loss and gradient for the fully-connected net. Input / output: Same as TwoLayerNet above. """ X = X.to(self.dtype) mode = 'test' if y is None else 'train' # Set train/test mode for batchnorm params and dropout param since they # behave differently during training and testing. if self.use_dropout: self.dropout_param['mode'] = mode scores = None ############################################################################ # TODO: Implement the forward pass for the fully-connected net, computing # # the class scores for X and storing them in the scores variable. # # # # When using dropout, you'll need to pass self.dropout_param to each # # dropout forward pass. # ############################################################################ # Replace "pass" statement with your code cache_dict = {} last_out = X for n in range(self.num_layers-1): i=n+1 last_out, cache_dict['cache_LR{}'.format(i)] = Linear_ReLU.forward(last_out,self.params['W{}'.format(i)],self.params['b{}'.format(i)]) if self.use_dropout: last_out, cache_dict['cache_Dropout{}'.format(i)] = Dropout.forward(last_out,self.dropout_param) i+=1 last_out, cache_dict['cache_L{}'.format(i)] = Linear.forward(last_out,self.params['W{}'.format(i)],self.params['b{}'.format(i)]) scores = last_out # If test mode return early if mode == 'test': return scores loss, grads = 0.0, {} ############################################################################ # TODO: Implement the backward pass for the fully-connected net. Store the # # loss in the loss variable and gradients in the grads dictionary. Compute # # data loss using softmax, and make sure that grads[k] holds the gradients # # for self.params[k]. Don't forget to add L2 regularization! # # NOTE: To ensure that your implementation matches ours and you pass the # # automated tests, make sure that your L2 regularization includes a factor # # of 0.5 to simplify the expression for the gradient. # ############################################################################ # Replace "pass" statement with your code loss, dout = softmax_loss(scores, y) loss += (self.params['W{}'.format(i)]*self.params['W{}'.format(i)]).sum()*self.reg last_dout, dw, db = Linear.backward(dout, cache_dict['cache_L{}'.format(i)]) grads['W{}'.format(i)] = dw + 2*self.params['W{}'.format(i)]*self.reg grads['b{}'.format(i)] = db for n in range(self.num_layers-1)[::-1]: i = n +1 if self.use_dropout: last_dout = Dropout.backward(last_dout, cache_dict['cache_Dropout{}'.format(i)]) last_dout, dw, db = Linear_ReLU.backward(last_dout, cache_dict['cache_LR{}'.format(i)]) grads['W{}'.format(i)] = dw + 2*self.params['W{}'.format(i)]*self.reg grads['b{}'.format(i)] = db loss += (self.params['W{}'.format(i)]*self.params['W{}'.format(i)]).sum()*self.reg return loss, grads 不同梯度下降方法
SGD,SGD+Momentum,RMSprop,Adam(Momentum+RMSprop+bias)的实现
具体原理介绍可参考之前的一篇博文：https://www.cnblogs.com/dyccyber/p/17759697.html
这里特别提及一下在Adam中我们加入了偏置项，是为了防止在初期进行梯度下降的过程中，下降的过快
def sgd(w, dw, config=None): """ Performs vanilla stochastic gradient descent. config format: - learning_rate: Scalar learning rate. """ if config is None: config = {} config.setdefault('learning_rate', 1e-2) w -= config['learning_rate'] * dw return w, config def sgd_momentum(w, dw, config=None): """ Performs stochastic gradient descent with momentum. config format: - learning_rate: Scalar learning rate. - momentum: Scalar between 0 and 1 giving the momentum value. Setting momentum = 0 reduces to sgd. - velocity: A numpy array of the same shape as w and dw used to store a moving average of the gradients. """ if config is None: config = {} config.setdefault('learning_rate', 1e-2) config.setdefault('momentum', 0.9) v = config.get('velocity', torch.zeros_like(w)) next_w = None ############################################################################# # TODO: Implement the momentum update formula. Store the updated value in # # the next_w variable. You should also use and update the velocity v. # ############################################################################# # Replace "pass" statement with your code v = config['momentum']*v - config['learning_rate'] * dw next_w = w + v ############################################################################# # END OF YOUR CODE # ############################################################################# config['velocity'] = v return next_w, config def rmsprop(w, dw, config=None): """ Uses the RMSProp update rule, which uses a moving average of squared gradient values to set adaptive per-parameter learning rates. config format: - learning_rate: Scalar learning rate. - decay_rate: Scalar between 0 and 1 giving the decay rate for the squared gradient cache. - epsilon: Small scalar used for smoothing to avoid dividing by zero. - cache: Moving average of second moments of gradients. """ if config is None: config = {} config.setdefault('learning_rate', 1e-2) config.setdefault('decay_rate', 0.99) config.setdefault('epsilon', 1e-8) config.setdefault('cache', torch.zeros_like(w)) next_w = None ########################################################################### # TODO: Implement the RMSprop update formula, storing the next value of w # # in the next_w variable. Don't forget to update cache value stored in # # config['cache']. # ########################################################################### # Replace "pass" statement with your code config['cache'] = config['decay_rate'] * config['cache'] + (1 - config['decay_rate']) * dw**2 w += -config['learning_rate'] * dw / (torch.sqrt(config['cache']) + config['epsilon']) next_w = w ########################################################################### # END OF YOUR CODE # ########################################################################### return next_w, config def adam(w, dw, config=None): """ Uses the Adam update rule, which incorporates moving averages of both the gradient and its square and a bias correction term. config format: - learning_rate: Scalar learning rate. - beta1: Decay rate for moving average of first moment of gradient. - beta2: Decay rate for moving average of second moment of gradient. - epsilon: Small scalar used for smoothing to avoid dividing by zero. - m: Moving average of gradient. - v: Moving average of squared gradient. - t: Iteration number. """ if config is None: config = {} config.setdefault('learning_rate', 1e-3) config.setdefault('beta1', 0.9) config.setdefault('beta2', 0.999) config.setdefault('epsilon', 1e-8) config.setdefault('m', torch.zeros_like(w)) config.setdefault('v', torch.zeros_like(w)) config.setdefault('t', 0) next_w = None ############################################################################# # TODO: Implement the Adam update formula, storing the next value of w in # # the next_w variable. Don't forget to update the m, v, and t variables # # stored in config. # # # # NOTE: In order to match the reference output, please modify t _before_ # # using it in any calculations. # ############################################################################# # Replace "pass" statement with your code config['t'] += 1 config['m'] = config['beta1']*config['m'] + (1-config['beta1'])*dw mt = config['m'] / (1-config['beta1']**config['t']) config['v'] = config['beta2']*config['v'] + (1-config['beta2'])*(dw*dw) vc = config['v'] / (1-(config['beta2']**config['t'])) w = w - (config['learning_rate'] * mt)/ (torch.sqrt(vc) + config['epsilon']) next_w = w ############################################################################# # END OF YOUR CODE # ############################################################################# return next_w, config Dropout层
注意在前面多层全连接网络的实现中，dropout只有在我们进行train的时候才使用，在test的时候是不使用的
dropout层是一个非常高效与简单的正则化方法，具体来说，在训练时，dropout 是通过仅以一定概率 p 保持神经元活跃来实现的，如果我们设置的随机数小于p就将其设置为零，如下图所示：
用另一种视角去看，dropout实际上是一种对全神经网络进行抽样的方法，可以减少不同神经元之间复杂的关系
具体论文原文见：https://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf
代码实现：
class Dropout(object): @staticmethod def forward(x, dropout_param): """ Performs the forward pass for (inverted) dropout. Inputs: - x: Input data: tensor of any shape - dropout_param: A dictionary with the following keys: - p: Dropout parameter. We *drop* each neuron output with probability p. - mode: 'test' or 'train'. If the mode is train, then perform dropout; if the mode is test, then just return the input. - seed: Seed for the random number generator. Passing seed makes this function deterministic, which is needed for gradient checking but not in real networks. Outputs: - out: Tensor of the same shape as x. - cache: tuple (dropout_param, mask). In training mode, mask is the dropout mask that was used to multiply the input; in test mode, mask is None. NOTE: Please implement **inverted** dropout, not the vanilla version of dropout. See http://cs231n.github.io/neural-networks-2/#reg for more details. NOTE 2: Keep in mind that p is the probability of **dropping** a neuron output; this might be contrary to some sources, where it is referred to as the probability of keeping a neuron output. """ p, mode = dropout_param['p'], dropout_param['mode'] if 'seed' in dropout_param: torch.manual_seed(dropout_param['seed']) mask = None out = None if mode == 'train': ########################################################################### # TODO: Implement training phase forward pass for inverted dropout. # # Store the dropout mask in the mask variable. # ########################################################################### # Replace "pass" statement with your code mask = torch.rand(x.shape) > p out = x.clone() out[mask] = 0 ########################################################################### # END OF YOUR CODE # ########################################################################### elif mode == 'test': ########################################################################### # TODO: Implement the test phase forward pass for inverted dropout. # ########################################################################### # Replace "pass" statement with your code out = x cache = (dropout_param, mask) return out, cache @staticmethod def backward(dout, cache): """ Perform the backward pass for (inverted) dropout. Inputs: - dout: Upstream derivatives, of any shape - cache: (dropout_param, mask) from Dropout.forward. """ dropout_param, mask = cache mode = dropout_param['mode'] dx = None if mode == 'train': ########################################################################### # TODO: Implement training phase backward pass for inverted dropout # ########################################################################### # Replace "pass" statement with your code dx = dout dx[mask] = 0 elif mode == 'test': dx = dout return dx 
![洛谷 P11345 [KTSC 2023 R2] 基地简化 题解](http://www.itfaba.com/wp-content/themes/kemi/timthumb.php?src=http://www.itfaba.com/wp-content/themes/kemi/img/random/3.jpg&w=218&h=124&zc=1)
