[Mongodb] mongodb sharding集群配置


本文总阅读量次

0、前提需求

为了满足mongodb数据库高可用环境，可以使用配置副本集CSRS（config server replica set)，搭建方法点击此处,但是当业务吞吐量特别高的时刻，单台节点已经不能满足访问需求，因此需要考虑sharding cluster 模式，提高访问性能。

1、将集群架构从csrs调整为sharding cluster,下文按照上面搭建的csrs架构继续调整：

1.1、节点信息如下：
主：10.0.7.53
从：10.0.7.51
仲裁：10.0.7.50
依次关闭仲裁、从、主节点，修改配置文件mongodb.conf，添加如下信息：

1
2
sharding:
   clusterRole: shardsvr

1.2、修改完三个节点之后，依次启动主，从，仲裁三个节点，进入控制台使用rs.status()，查看集群状态是否正常。为了保证停机时间较短，可使用rs.stepDown()分别进行主从切换，对从库进行配置并重启。

2、新建一个shard1集群节点，操作如下(三个节点都要执行）：

2.1、节点信息如下：
从：10.0.7.53
主：10.0.7.51
仲裁：10.0.7.50
2.2、拷贝一份mongodb.conf文件，并重新命名
cp mongodb.conf shard1mongodb.conf
2.3、主节点需要在无需验证的模式下，先配置用户名密码。之后修改（三个节点）shard1mongodb.conf配置文件内容如下：
cat shard1mongodb.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# mongod.conf
# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/
# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /data/mongodb/shard1/log/mongod.log

# Where and how to store data.
storage:
  dbPath: /data/mongodb/shard1/data/
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /data/mongodb/shard1/log/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 30001
  bindIp: 127.0.0.1,10.0.7.53  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.
#security:
security:
   authorization: enabled
   keyFile: /data/mongodb/shard1/mongodb.keyfile
#operationProfiling:
#replication:
replication:
   replSetName: "shard1"
#sharding:
sharding:
   clusterRole: shardsvr
## Enterprise-Only Options
#auditLog:
#snmp:

注意修改sharding、port、path、replname等参数

2.4、创建相应的数据目录和日志目录：

1
mkdir -p /data/mongodb/shard1/{log,data}

2.5、生成keyfile文件并拷贝keyfile文件到指定目录：

1
cp mongodb.keyfile /data/mongodb/shard1/

2.6、三个节点启动mongodb-shard1:

1
/data/mongodb/bin/mongod -f /data/mongodb/shard1mongodb.conf

2.7、初始化集群配置：
/data/mongodb/bin/mongo –port 30001
–
rs.initiate(
{
_id : “shard”,
members: [
{ _id : 0, host : “10.0.7.53:30001” },
{ _id : 1, host : “10.0.7.50:30001” },
{ _id : 2, host : “10.0.7.51:30001”,”arbiterOnly” : true }
]
}
)
2.8、查看集群节点信息：

1
rs.status();
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
{
	"set" : "shard",
	"date" : ISODate("2018-08-09T03:13:34.697Z"),
	"myState" : 2,
	"term" : NumberLong(1),
	"syncingTo" : "10.0.7.50:30001",
	"syncSourceHost" : "10.0.7.50:30001",
	"syncSourceId" : 1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1533784405, 1),
			"t" : NumberLong(1)
		},
		"readConcernMajorityOpTime" : {
			"ts" : Timestamp(1533784405, 1),
			"t" : NumberLong(1)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1533784405, 1),
			"t" : NumberLong(1)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1533784405, 1),
			"t" : NumberLong(1)
		}
	},
	"lastStableCheckpointTimestamp" : Timestamp(1533784355, 1),
	"members" : [
		{
			"_id" : 0,
			"name" : "10.0.7.53:30001",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 47039,
			"optime" : {
				"ts" : Timestamp(1533784405, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2018-08-09T03:13:25Z"),
			"syncingTo" : "10.0.7.50:30001",
			"syncSourceHost" : "10.0.7.50:30001",
			"syncSourceId" : 1,
			"infoMessage" : "",
			"configVersion" : 1,
			"self" : true,
			"lastHeartbeatMessage" : ""
		},
		{
			"_id" : 1,
			"name" : "10.0.7.50:30001",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 46862,
			"optime" : {
				"ts" : Timestamp(1533784405, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1533784405, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2018-08-09T03:13:25Z"),
			"optimeDurableDate" : ISODate("2018-08-09T03:13:25Z"),
			"lastHeartbeat" : ISODate("2018-08-09T03:13:33.395Z"),
			"lastHeartbeatRecv" : ISODate("2018-08-09T03:13:32.735Z"),
			"pingMs" : NumberLong(1),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1533737562, 1),
			"electionDate" : ISODate("2018-08-08T14:12:42Z"),
			"configVersion" : 1
		},
		{
			"_id" : 2,
			"name" : "10.0.7.51:30001",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 46862,
			"lastHeartbeat" : ISODate("2018-08-09T03:13:34.660Z"),
			"lastHeartbeatRecv" : ISODate("2018-08-09T03:13:34.658Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"configVersion" : 1
		}
	],
	"ok" : 1,
	"operationTime" : Timestamp(1533784405, 1),
	"$gleStats" : {
		"lastOpTime" : Timestamp(0, 0),
		"electionId" : ObjectId("000000000000000000000000")
	},
	"lastCommittedOpTime" : Timestamp(1533784405, 1),
	"$configServerState" : {
		"opTime" : {
			"ts" : Timestamp(1533784394, 1),
			"t" : NumberLong(1)
		}
	},
	"$clusterTime" : {
		"clusterTime" : Timestamp(1533784408, 1),
		"signature" : {
			"hash" : BinData(0,"dToZVSHbihPkQILJVl0qcSY+8ys="),
			"keyId" : NumberLong("6587344633552961565")
		}
	}
}

3、新建一个config-server集群节点，操作如下(三个节点都要执行）：

3.1、节点信息如下：

1
2
3
从：10.0.7.53
从：10.0.7.51
主：10.0.7.50

3.2、拷贝一份mongodb.conf文件，并重新命名

1
cp shard1mongodb.conf configmongodb.conf

3.3、主节点需要在无需验证的模式下，先配置用户名密码。之后修改（三个节点）configmongodb.conf配置文件内容如下：
cat configmongodb.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# mongod.conf
# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/
# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /data/mongodb/config/log/mongod.log

# Where and how to store data.
storage:
  dbPath: /data/mongodb/config/data/
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /data/mongodb/config/log/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 30002
  bindIp: 127.0.0.1,10.0.7.53  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.
#security:
#security:
#   authorization: enabled
#   keyFile: /data/mongodb/config/mongodb.keyfile
#operationProfiling:
#replication:
replication:
   replSetName: "config"
#sharding:
sharding:
   clusterRole: configsvr
## Enterprise-Only Options
#auditLog:
#snmp:

注意修改sharding、repl、port、path
3.4、创建相应的数据目录和日志目录：

1
mkdir -p /data/mongodb/config/{log,data}

3.5、三个节点启动mongodb-shard1:

1
/data/mongodb/bin/mongod --configsvr -f /data/mongodb/configmongodb.conf

启动config-server服务需要指定参数，否则初始化集群的时候会报错：

1
"errmsg" : "Nodes being used for config servers must be started with the --configsvr flag"

3.6、初始化集群配置：
/data/mongodb/bin/mongo –port 30002
–
rs.initiate(
{
_id : “config”,
configsvr: true,
members: [
{ _id : 0, host : “10.0.7.53:30002” },
{ _id : 1, host : “10.0.7.50:30002” },
{ _id : 2, host : “10.0.7.51:30002” }
]
}
)
3.7、查看集群信息：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
rs.status();
{
	"set" : "config",
	"date" : ISODate("2018-08-06T08:29:14.472Z"),
	"myState" : 1,
	"term" : NumberLong(1),
	"syncingTo" : "",
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1533544151, 2),
			"t" : NumberLong(1)
		},
		"readConcernMajorityOpTime" : {
			"ts" : Timestamp(1533544151, 2),
			"t" : NumberLong(1)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1533544151, 2),
			"t" : NumberLong(1)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1533544151, 2),
			"t" : NumberLong(1)
		}
	},
	"lastStableCheckpointTimestamp" : Timestamp(1533544151, 1),
	"members" : [
		{
			"_id" : 0,
			"name" : "10.0.7.53:30002",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 15,
			"optime" : {
				"ts" : Timestamp(1533544151, 2),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1533544151, 2),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2018-08-06T08:29:11Z"),
			"optimeDurableDate" : ISODate("2018-08-06T08:29:11Z"),
			"lastHeartbeat" : ISODate("2018-08-06T08:29:14.161Z"),
			"lastHeartbeatRecv" : ISODate("2018-08-06T08:29:12.674Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "10.0.7.51:30002",
			"syncSourceHost" : "10.0.7.51:30002",
			"syncSourceId" : 2,
			"infoMessage" : "",
			"configVersion" : 1
		},
		{
			"_id" : 1,
			"name" : "10.0.7.50:30002",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 15,
			"optime" : {
				"ts" : Timestamp(1533544151, 2),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1533544151, 2),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2018-08-06T08:29:11Z"),
			"optimeDurableDate" : ISODate("2018-08-06T08:29:11Z"),
			"lastHeartbeat" : ISODate("2018-08-06T08:29:14.164Z"),
			"lastHeartbeatRecv" : ISODate("2018-08-06T08:29:12.680Z"),
			"pingMs" : NumberLong(1),
			"lastHeartbeatMessage" : "",
			"syncingTo" : "10.0.7.51:30002",
			"syncSourceHost" : "10.0.7.51:30002",
			"syncSourceId" : 2,
			"infoMessage" : "",
			"configVersion" : 1
		},
		{
			"_id" : 2,
			"name" : "10.0.7.51:30002",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 59,
			"optime" : {
				"ts" : Timestamp(1533544151, 2),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2018-08-06T08:29:11Z"),
			"syncingTo" : "",
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "could not find member to sync from",
			"electionTime" : Timestamp(1533544150, 1),
			"electionDate" : ISODate("2018-08-06T08:29:10Z"),
			"configVersion" : 1,
			"self" : true,
			"lastHeartbeatMessage" : ""
		}
	],
	"ok" : 1,
	"operationTime" : Timestamp(1533544151, 2),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1533544151, 2),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}

4、配置mongs(mongodb-router),用于客户端连接到mongodb集群，为防止单点故障，可配置多个mongos节点

4.1、拷贝一份配置文件，并重命名

1
cp configmongodb.conf routermongodb.conf

4.2、修改配置文件参数，内容如下：
cat routermongodb.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/routeruration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /data/mongodb/router/log/mongod.log

# Where and how to store data.
#storage:
#  dbPath: /data/mongodb/router/data/
#  journal:
#    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /data/mongodb/router/log/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 30004
  bindIp: 127.0.0.1,10.0.7.53  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.

#security:
#security:
#   authorization: enabled
#operationProfiling:

#replication:
#sharding:
sharding:
   configDB: config/10.0.7.53:30002,10.0.7.51:30002,10.0.7.50:30002

## Enterprise-Only Options

#auditLog:

#snmp:

注意修改sharding,port参数，注释掉storage参数
4.3、创建router日志目录

1
mkdir -p /data/mongodb/router/log

4.4、启动mongos

1
/data/mongodb/bin/mongos -f /data/mongodb/routermongodb.conf

4.5、为模拟生产环境数据，对testrepl集群节点，插入部分数据,批量插入脚本如下：

1
2
3
4
5
6
7
8
9
10
use test
var bulk = db.test_collection.initializeUnorderedBulkOp();
people = ["Marc", "Bill", "George", "Eliot", "Matt", "Trey", "Tracy", "Greg", "Steve", "Kristina", "Katie", "Jeff"];
for(var i=0; i<1000000; i++){
   user_id = i;
   name = people[Math.floor(Math.random()*people.length)];
   number = Math.floor(Math.random()*10001);
   bulk.insert( { "user_id":user_id, "name":name, "number":number });
}
bulk.execute();

4.6、连接到mongos

1
2
3
shell > mongo --host 10.0.7.50 --port 30004
mongos> use admin
mongos> db.auth('admin1','admin123');

4.7、将节点添加到集群内：

1
2
mongos> sh.addShard("testrepl/10.0.7.53:30000")
mongos> sh.addShard("shard1/10.0.7.53:30001")

4.8、使数据库test支持sharding

1
2
3
4
5
6
mongos> sh.enableSharding("test")
mongos> use test
#在拆分键上创建索引
mongos> db.test_collection.createIndex( { number : 1 } )
#拆分集合
mongos> sh.shardCollection('test.mycol2', {'_id': 1})

4.9、查看拆分结果,(可多次重复4.5步骤进行测试）
sh.status()或者db.printShardingStatus() 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
--- Sharding Status --- 
  sharding version: {
  	"_id" : 1,
  	"minCompatibleVersion" : 5,
  	"currentVersion" : 6,
  	"clusterId" : ObjectId("5b6af30b5025146bfd45717b")
  }
  shards:
        {  "_id" : "shard",  "host" : "shard/10.0.7.50:30001,10.0.7.53:30001",  "state" : 1 }
        {  "_id" : "testrepl",  "host" : "testrepl/10.0.7.50:30000,10.0.7.53:30000",  "state" : 1 }
  active mongoses:
        "4.0.0" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                4 : Success
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard	1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard Timestamp(1, 0) 
        {  "_id" : "test",  "primary" : "testrepl",  "partitioned" : true,  "version" : {  "uuid" : UUID("185a3842-639f-42b0-89a1-afdc8dcdfbe7"),  "lastMod" : 1 } }
                test.test_collection
                        shard key: { "number" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard	4
                                testrepl	4
                        { "number" : { "$minKey" : 1 } } -->> { "number" : 2394 } on : shard Timestamp(2, 0) 
                        { "number" : 2394 } -->> { "number" : 4791 } on : shard Timestamp(3, 0) 
                        { "number" : 4791 } -->> { "number" : 7194 } on : shard Timestamp(4, 0) 
                        { "number" : 7194 } -->> { "number" : 7892 } on : shard Timestamp(5, 0) 
                        { "number" : 7892 } -->> { "number" : 8591 } on : testrepl Timestamp(5, 1) 
                        { "number" : 8591 } -->> { "number" : 9287 } on : testrepl Timestamp(3, 4) 
                        { "number" : 9287 } -->> { "number" : 9589 } on : testrepl Timestamp(3, 5) 
                        { "number" : 9589 } -->> { "number" : { "$maxKey" : 1 } } on : testrepl Timestamp(4, 1)

5、大功告成，客户端访问mongodb集群，只需连接mongos对应的ip、port即可

mongodb
目录
  1. 1. 0、前提需求
  2. 2. 1、将集群架构从csrs调整为sharding cluster,下文按照上面搭建的csrs架构继续调整：
  3. 3. 2、新建一个shard1集群节点，操作如下(三个节点都要执行）：
  4. 4. 3、新建一个config-server集群节点，操作如下(三个节点都要执行）：
  5. 5. 4、配置mongs(mongodb-router),用于客户端连接到mongodb集群，为防止单点故障，可配置多个mongos节点
  6. 6. 5、大功告成，客户端访问mongodb集群，只需连接mongos对应的ip、port即可

Proudly powered by Hexo and Theme by Lap
本站访客数人次
© 2020 zeven0707's blog