From 0d8f17cae0adbeccdc32f9dfba995719e767a39c Mon Sep 17 00:00:00 2001 From: iridiumR Date: Mon, 26 Dec 2022 17:09:57 +0800 Subject: [PATCH] feat(AIandML): e3.2 --- AIandML/e3_deep_learning/e3.2_lenet.ipynb | 392 ++++++++++++++++++++++ 1 file changed, 392 insertions(+) create mode 100644 AIandML/e3_deep_learning/e3.2_lenet.ipynb diff --git a/AIandML/e3_deep_learning/e3.2_lenet.ipynb b/AIandML/e3_deep_learning/e3.2_lenet.ipynb new file mode 100644 index 0000000..1c0be78 --- /dev/null +++ b/AIandML/e3_deep_learning/e3.2_lenet.ipynb @@ -0,0 +1,392 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 实验3-3 LeNet模型定义和训练\n", + "\n", + "实验目标：\n", + "\n", + "* 初步掌握模型构建和训练\n", + "\n", + "\n", + "### 1. 定义网络\n", + "\n", + "首先定义一个LeNet网络:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "LeNet(\n", + " (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n", + " (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n", + " (fc1): Linear(in_features=400, out_features=120, bias=True)\n", + " (fc2): Linear(in_features=120, out_features=84, bias=True)\n", + " (fc3): Linear(in_features=84, out_features=10, bias=True)\n", + ")\n" + ] + } + ], + "source": [ + "import torch\n", + "import torch.nn as nn # 类\n", + "import torch.nn.functional as F # 函数\n", + "\n", + "class LeNet(nn.Module):\n", + " def __init__(self):\n", + " super().__init__()\n", + " # 1 input image channel, 6 output channels, 5x5 square convolution\n", + " # kernel\n", + " self.conv1 = nn.Conv2d(1, 6, 5) # 通道: 1 => 6, kernel size: 5\n", + " self.conv2 = nn.Conv2d(6, 16, 5)\n", + " # an affine operation: y = Wx + b\n", + " self.fc1 = nn.Linear(16 * 5 * 5, 120) # 5*5 from image dimension \n", + " self.fc2 = nn.Linear(120, 84)\n", + " self.fc3 = nn.Linear(84, 10)\n", + "\n", + " def forward(self, x):\n", + " ''' 定义网络前馈的过程，而其反向推导则基于此自动进行\n", + " '''\n", + " # Max pooling over a (2, 2) window\n", + " x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n", + " # If the size is a square, you can specify with a single number\n", + " x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n", + " x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension\n", + " x = F.relu(self.fc1(x))\n", + " x = F.relu(self.fc2(x))\n", + " x = self.fc3(x)\n", + " return x\n", + "\n", + "\n", + "net = LeNet()\n", + "print(net)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> 请回答：\n", + "> 1. 前文代码块中`nn.Conv2d`和`nn.Linear`分别是什么模块？\n", + "> 2. `nn.Conv2d`和`nn.Linear`在初始化时，其构造函数的参数是怎样的？" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "ename": "SyntaxError", + "evalue": "invalid syntax (1425913863.py, line 1)", + "output_type": "error", + "traceback": [ + "\u001b[0;36m Cell \u001b[0;32mIn[7], line 1\u001b[0;36m\u001b[0m\n\u001b[0;31m 1. 卷积层和线性层\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n" + ] + } + ], + "source": [ + "1. 卷积层和线性层\n", + "2.分别为:输入通道数量 输出通道数量 卷积核大小\n", + " 输入维度 输出维度" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "您只需要重新定义`nn.Module`类的`forward`成员函数（即前向传播），而无需定义`backward`成员函数（即反向传播）。在`forward`函数中，可以自由地使用任何张量运算。\n", + "\n", + "这是因为，`autograd`计算图机制将自动化的定义`backward`成员函数。\n", + "\n", + "\n", + "### 2. 获取权重\n", + "\n", + "在定义好神经网络（本例中即为`LeNet`）后，通过 `net.parameters()`成员函数可获取该网络中可学习的参数（又称权重）。\n", + "\n", + "所获取的权重可以交给PyTorch利用梯度计算机制统一更新。" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "10\n", + "torch.Size([6, 1, 5, 5])\n" + ] + } + ], + "source": [ + "params = list(net.parameters())\n", + "print(len(params))\n", + "print(params[0].size()) # conv1's .weight\n", + "#print(params)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3. 测试输入输出\n", + "\n", + "让我们尝试一个随机的 32x32 输入。\n", + "\n", + "注意：此网络`LeNet`的预期输入大小为 32x32。要将此网络用于MNIST 数据集，请将数据集中的图像大小调整为 32x32。\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "tensor([[ 0.0719, 0.0145, -0.1102, -0.0903, -0.0850, -0.0460, 0.0280, -0.0440,\n", + " -0.0256, 0.0834]], grad_fn=)\n" + ] + } + ], + "source": [ + "input = torch.randn(1, 1, 32, 32)\n", + "out = net(input)\n", + "print(out)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> 请回答：\n", + "> 1. 请在代码中测试，若输入不是(1, 1, 32, 32)尺寸的张量，会导致什么效果？" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "将所有参数的梯度缓存清零，并用伪真值（随机值）计算损失并反向传播（BP）。\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "tensor(0.8519, grad_fn=)\n" + ] + } + ], + "source": [ + "output = net(input)\n", + "target = torch.randn(10) # a dummy target, for example\n", + "target = target.view(1, -1) # make it the same shape as output\n", + "criterion = nn.MSELoss()\n", + "\n", + "loss = criterion(output, target)\n", + "print(loss)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "至此，从`input`到`loss`计算过程，可以用如下计算图（computational graph）来表示：\n", + "\n", + " input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d\n", + " -> flatten -> linear -> relu -> linear -> relu -> linear\n", + " -> MSELoss\n", + " -> loss\n", + "\n", + "所以，当调用`loss.backward()`时, 整个计算图是可以求解导数的，即该网络中所有具有`requires_grad=True`属性的张量皆会计算其梯度，并保存于`.grad`属性之中。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4. 反向推导Backprop\n", + "\n", + "\n", + "要计算反向传播的梯度，我们所要做的就是调用`loss.backward()`成员函数。\n", + "\n", + "但请注意，需要预先清除已有的权重梯度值，否则权重的梯度值将是若干次梯度反向传播的累积值。\n", + "\n", + "现在我们将调用`loss.backward()`函数，并观察conv1的梯度的前后变化\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "conv1.bias.grad before backward\n", + "None\n", + "conv1.bias.grad after backward\n", + "tensor([ 0.0115, 0.0092, 0.0135, 0.0001, -0.0111, 0.0149])\n" + ] + } + ], + "source": [ + "net.zero_grad() # zeroes the gradient buffers of all parameters\n", + "\n", + "print('conv1.bias.grad before backward')\n", + "print(net.conv1.bias.grad)\n", + "\n", + "loss.backward()\n", + "\n", + "print('conv1.bias.grad after backward')\n", + "print(net.conv1.bias.grad)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> 请回答：\n", + "> 1. 说明前文代码块中，conv1的梯度在`loss.backward()`执行前后变化。以及为什么会这样？" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "执行之后产生梯度值. 执行损失计算后梯度得到了更新" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5. 更新权重\n", + "\n", + "实际中最简单的权重优化原则，即为随机梯度下降Stochastic Gradient Descent (SGD)\n", + "\n", + " ``weight = weight - learning_rate * gradient``\n", + "\n", + "以下Python代码来实现SGD功能：\n", + "\n", + "```python\n", + " learning_rate = 0.01\n", + " for f in net.parameters():\n", + " f.data.sub_(f.grad.data * learning_rate)\n", + "```\n", + "\n", + "但是，当使用神经网络时，通常希望使用各种不同的权重优化规则，如SGD，Nesterov-SGD，Adam，RMSProp等。因此，前文所述代码在实际中并不常用。\n", + "\n", + "为了实现这一点，需要用到PyTorch的包：`torch.optim`，其中实现所有这些方法。使用非常简单。\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import torch.optim as optim\n", + "\n", + "# create your optimizer\n", + "optimizer = optim.SGD(net.parameters(), lr=0.01)\n", + "\n", + "for i in range(10):\n", + " # input = ...\n", + " # in your training loop:\n", + " optimizer.zero_grad() # zero the gradient buffers\n", + " output = net(input)\n", + " loss = criterion(output, target)\n", + " loss.backward()\n", + " optimizer.step() # Does the update" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> 请回答：\n", + "> 1. 请结合实验结果，解释前文代码块中`for`循环内每行代码的功能。" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "分别为:\n", + "清空梯度数据\n", + "数据输入网络得到输出\n", + "计算损失值\n", + "反向传播\n", + "更新参数" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6. 完整训练LeNet网络（可选）\n", + "\n", + "参考 https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html\n", + " \n", + "> 在此执行一次完整的LeNet训练。给出实验代码和效果。" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.16" + }, + "vscode": { + "interpreter": { + "hash": "0733c54d9044ea299f7b7f48049f3576c8ad4e6ff5a97e2c60d8a9e3bff0bc54" + } + } + }, + "nbformat": 4, + "nbformat_minor": 1 +}