This repository has been archived on 2024-01-06. You can view files and clone it, but cannot push or open issues or pull requests.
justhomework/AIandML/e3_deep_learning/e3.2_lenet.ipynb

393 lines
11 KiB
Text
Raw Normal View History

2022-12-26 09:09:57 +00:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 实验3-3 LeNet模型定义和训练\n",
"\n",
"实验目标:\n",
"\n",
"* 初步掌握模型构建和训练\n",
"\n",
"\n",
"### 1. 定义网络\n",
"\n",
"首先定义一个LeNet网络:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"LeNet(\n",
" (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
" (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
" (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
" (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
" (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
")\n"
]
}
],
"source": [
"import torch\n",
"import torch.nn as nn # 类\n",
"import torch.nn.functional as F # 函数\n",
"\n",
"class LeNet(nn.Module):\n",
" def __init__(self):\n",
" super().__init__()\n",
" # 1 input image channel, 6 output channels, 5x5 square convolution\n",
" # kernel\n",
" self.conv1 = nn.Conv2d(1, 6, 5) # 通道: 1 => 6, kernel size: 5\n",
" self.conv2 = nn.Conv2d(6, 16, 5)\n",
" # an affine operation: y = Wx + b\n",
" self.fc1 = nn.Linear(16 * 5 * 5, 120) # 5*5 from image dimension \n",
" self.fc2 = nn.Linear(120, 84)\n",
" self.fc3 = nn.Linear(84, 10)\n",
"\n",
" def forward(self, x):\n",
" ''' 定义网络前馈的过程,而其反向推导则基于此自动进行\n",
" '''\n",
" # Max pooling over a (2, 2) window\n",
" x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
" # If the size is a square, you can specify with a single number\n",
" x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n",
" x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension\n",
" x = F.relu(self.fc1(x))\n",
" x = F.relu(self.fc2(x))\n",
" x = self.fc3(x)\n",
" return x\n",
"\n",
"\n",
"net = LeNet()\n",
"print(net)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> 请回答:\n",
"> 1. 前文代码块中`nn.Conv2d`和`nn.Linear`分别是什么模块?\n",
"> 2. `nn.Conv2d`和`nn.Linear`在初始化时,其构造函数的参数是怎样的?"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"ename": "SyntaxError",
"evalue": "invalid syntax (1425913863.py, line 1)",
"output_type": "error",
"traceback": [
"\u001b[0;36m Cell \u001b[0;32mIn[7], line 1\u001b[0;36m\u001b[0m\n\u001b[0;31m 1. 卷积层和线性层\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
]
}
],
"source": [
"1. 卷积层和线性层\n",
"2.分别为:输入通道数量 输出通道数量 卷积核大小\n",
" 输入维度 输出维度"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"您只需要重新定义`nn.Module`类的`forward`成员函数(即前向传播),而无需定义`backward`成员函数(即反向传播)。在`forward`函数中,可以自由地使用任何张量运算。\n",
"\n",
"这是因为,`autograd`计算图机制将自动化的定义`backward`成员函数。\n",
"\n",
"\n",
"### 2. 获取权重\n",
"\n",
"在定义好神经网络(本例中即为`LeNet`)后,通过 `net.parameters()`成员函数可获取该网络中可学习的参数(又称权重)。\n",
"\n",
"所获取的权重可以交给PyTorch利用梯度计算机制统一更新。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"10\n",
"torch.Size([6, 1, 5, 5])\n"
]
}
],
"source": [
"params = list(net.parameters())\n",
"print(len(params))\n",
"print(params[0].size()) # conv1's .weight\n",
"#print(params)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. 测试输入输出\n",
"\n",
"让我们尝试一个随机的 32x32 输入。\n",
"\n",
"注意:此网络`LeNet`的预期输入大小为 32x32。要将此网络用于MNIST 数据集,请将数据集中的图像大小调整为 32x32。\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[ 0.0719, 0.0145, -0.1102, -0.0903, -0.0850, -0.0460, 0.0280, -0.0440,\n",
" -0.0256, 0.0834]], grad_fn=<AddmmBackward0>)\n"
]
}
],
"source": [
"input = torch.randn(1, 1, 32, 32)\n",
"out = net(input)\n",
"print(out)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> 请回答:\n",
"> 1. 请在代码中测试,若输入不是(1, 1, 32, 32)尺寸的张量,会导致什么效果?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"将所有参数的梯度缓存清零并用伪真值随机值计算损失并反向传播BP。\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor(0.8519, grad_fn=<MseLossBackward0>)\n"
]
}
],
"source": [
"output = net(input)\n",
"target = torch.randn(10) # a dummy target, for example\n",
"target = target.view(1, -1) # make it the same shape as output\n",
"criterion = nn.MSELoss()\n",
"\n",
"loss = criterion(output, target)\n",
"print(loss)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"至此,从`input`到`loss`计算过程可以用如下计算图computational graph来表示\n",
"\n",
" input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d\n",
" -> flatten -> linear -> relu -> linear -> relu -> linear\n",
" -> MSELoss\n",
" -> loss\n",
"\n",
"所以,当调用`loss.backward()`时, 整个计算图是可以求解导数的,即该网络中所有具有`requires_grad=True`属性的张量皆会计算其梯度,并保存于`.grad`属性之中。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4. 反向推导Backprop\n",
"\n",
"\n",
"要计算反向传播的梯度,我们所要做的就是调用`loss.backward()`成员函数。\n",
"\n",
"但请注意,需要预先清除已有的权重梯度值,否则权重的梯度值将是若干次梯度反向传播的累积值。\n",
"\n",
"现在我们将调用`loss.backward()`函数并观察conv1的梯度的前后变化\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"conv1.bias.grad before backward\n",
"None\n",
"conv1.bias.grad after backward\n",
"tensor([ 0.0115, 0.0092, 0.0135, 0.0001, -0.0111, 0.0149])\n"
]
}
],
"source": [
"net.zero_grad() # zeroes the gradient buffers of all parameters\n",
"\n",
"print('conv1.bias.grad before backward')\n",
"print(net.conv1.bias.grad)\n",
"\n",
"loss.backward()\n",
"\n",
"print('conv1.bias.grad after backward')\n",
"print(net.conv1.bias.grad)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> 请回答:\n",
"> 1. 说明前文代码块中conv1的梯度在`loss.backward()`执行前后变化。以及为什么会这样?"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"执行之后产生梯度值. 执行损失计算后梯度得到了更新"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5. 更新权重\n",
"\n",
"实际中最简单的权重优化原则即为随机梯度下降Stochastic Gradient Descent (SGD)\n",
"\n",
" ``weight = weight - learning_rate * gradient``\n",
"\n",
"以下Python代码来实现SGD功能\n",
"\n",
"```python\n",
" learning_rate = 0.01\n",
" for f in net.parameters():\n",
" f.data.sub_(f.grad.data * learning_rate)\n",
"```\n",
"\n",
"但是当使用神经网络时通常希望使用各种不同的权重优化规则如SGDNesterov-SGDAdamRMSProp等。因此前文所述代码在实际中并不常用。\n",
"\n",
"为了实现这一点需要用到PyTorch的包`torch.optim`,其中实现所有这些方法。使用非常简单。\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch.optim as optim\n",
"\n",
"# create your optimizer\n",
"optimizer = optim.SGD(net.parameters(), lr=0.01)\n",
"\n",
"for i in range(10):\n",
" # input = ...\n",
" # in your training loop:\n",
" optimizer.zero_grad() # zero the gradient buffers\n",
" output = net(input)\n",
" loss = criterion(output, target)\n",
" loss.backward()\n",
" optimizer.step() # Does the update"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> 请回答:\n",
"> 1. 请结合实验结果,解释前文代码块中`for`循环内每行代码的功能。"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"分别为:\n",
"清空梯度数据\n",
"数据输入网络得到输出\n",
"计算损失值\n",
"反向传播\n",
"更新参数"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 6. 完整训练LeNet网络可选\n",
"\n",
"参考 https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html\n",
" \n",
"> 在此执行一次完整的LeNet训练。给出实验代码和效果。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.16"
},
"vscode": {
"interpreter": {
"hash": "0733c54d9044ea299f7b7f48049f3576c8ad4e6ff5a97e2c60d8a9e3bff0bc54"
}
}
},
"nbformat": 4,
"nbformat_minor": 1
}