N1H111SM's Miniverse

Pytorch - nn.ModuleList vs nn.Sequential

字数统计: 678阅读时长: 3 min
2020/05/17 Share

Materials

Problem Description

在实现GCN的时候,为了动态地定义图网络中间graph convolution layer的层数,我写了这样一段代码(GraphConvolution这个类已经定义完毕)。基本动机是为了创建一个彼此连接的GC层,同时还需要兼顾multi-scale features能够在未来被灵活地采用(例如可以concatenate, average, sum等),因此不能够将相互连接的网络层写在一个nn.Sequential()中。所以我采取了将他们全部放在一个list中的策略。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import torch.nn as nn
import torch.nn.functional as F

class GCN(nn.Module):
def __init__(self, d_input, d_hids, d_output, dropout):
super(GCN, self).__init__()

# connect the graph convolution layer
# 错误的重点在这一段
gcs = []
d_hids = [d_input, *d_hids, d_output]
for i in range(len(d_hids)-1):
gcs.append(GraphConvolution(d_hids[i], d_hids[i+1]))

self.gcs = gcs
self.dropout = dropout

def forward(self, x, adj):
multi_scale_output = []
for gc in self.gcs:
x = F.relu(gc(x, adj))
# something not sure here, whether
# the dropout will affect performance
x = F.dropout(x, self.dropout, training=self.training)
multi_scale_output.append(x)

return multi_scale_output

结果在以下测试代码运行的时候报了错误 “ValueError: optimizer got an empty parameter list”

1
2
model = GCN(d_input=x.shape[1], d_hids=[3, 4], d_output=5, dropout=0.9)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

幸好我是将所有的参数都放进了list中,如果一半一半的话这个错误就会被漏过去了。这就来到了一个基本的问题:为什么pytorch找不到list中的layer呢?或者说pytorch中的.parameters()是采用什么机制进行参数列表的注册的?

Mechanism

.parameters() works like this: it walks all members of the class (anything added to self) and does one of three things with each member:

  • If the member is a parameter (something registered with register_parameter(...) or of type nn.Parameter), it adds it to the parameters list.
  • If the member is of type (or is a subclass of) nn.Module, .parameters() is called recursively.
  • Otherwise, it is ignored

In theory, you could add a 4th option to handle lists, but nn.ModuleList was chosen instead.

根据以上分析,我们得到了报错原因:因为我们add to self的object是一个list,自然不是nn.Module的子类,所以pytorch忽略了它。同时给出了解决方法nn.ModuleList,这个解决方法在我们的问题中完美契合。从而我们只需要将

1
self.gcs = gcs

改写成

1
self.gcs = nn.ModuleList(gcs)

即可。

Difference

nn.Sequentialnn.ModuleList 有以下两点不同:

  • Sequential实现定义了forward函数,可以直接拿来作为Module接受输入输出;而ModuleList只是将一个列表的带参tensor集合在一起,forward函数需要自己再进行手动定义。
  • Sequential定义好了不同layer之间的连接顺序;而ModuleList可以在forward函数中更加自由地定义层与层之间的连接关系,甚至可以重复使用实现参数共享。
CATALOG
  1. 1. Problem Description
  2. 2. Mechanism
  3. 3. Difference