金瓶梅足本插图 epub:【auto shard】MongoDB auto shard配置说明

mongodb支持自动分片功能，这无疑给DBA和开发人员带来很大便利，我们无需要手动按一定规则进行拆分DB，当数据量增大后，我们只需要扩充设备即可。
mongodb shard配置文档如下：
auo shard 试验设备列表主机列表主机名 ip 用途 监听端口 web访问urllocalhost 192.168.100.245 config server 27019 http://192.168.100.245:28019/TEST40 192.168.100.212 shard 1 27018 http://192.168.100.212:28018/chn-xg-5-5c1 192.168.100.210 shard 2 27018 http://192.168.100.210:28018/band-all 192.168.100.229 mongos process/mongos Router 27017 http://192.168.100.229:27017/分析发现config server、shard、mongos的监听端口和web端口均是第二位的差异。监听端口第二位是7，web端口是8，其它全部相同。
mongod --fork --logpath /var/log/mongodb.log --logappendmogod默认启动为前台进程，当连接session断开，mongod程序自动关闭
一、配置shard 各个模块1、启动shard1
[root@TEST40 ~]# mongod --shardsvr --fork --logpath /var/log/mongodb.log --logappend[1] 28521[root@TEST40 ~]# Tue Aug 17 03:01:57 MongoDB starting : pid=28521 port=27018 dbpath=/data/db/ 32-bit 
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data**       see http://blog.mongodb.org/post/137788967/32-bit-limitations
Tue Aug 17 03:01:57 db version v1.6.0, pdfile version 4.5Tue Aug 17 03:01:57 git version: 2c7f164b653f0d703947572ede064aed41cc2185Tue Aug 17 03:01:57 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37Tue Aug 17 03:01:57 [initandlisten] waiting for connections on port 27018Tue Aug 17 03:01:57 [websvr] web admin interface listening on port 28018
2、启动shard2，shard2的dbpath与默认路径不同，因此需要制定dbpath[root@chn-xg-5-5c1 cache2]# mongod --shardsvr --dbpath /data/cache2/db --fork --logpath /var/log/mongodb.log --logappend[1] 31264[root@chn-xg-5-5c1 cache2]# Tue Aug 17 03:11:48 MongoDB starting : pid=31264 port=27018 dbpath=/data/cache2/db 32-bit 
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data**       see http://blog.mongodb.org/post/137788967/32-bit-limitations
Tue Aug 17 03:11:48 db version v1.6.0, pdfile version 4.5Tue Aug 17 03:11:48 git version: 2c7f164b653f0d703947572ede064aed41cc2185Tue Aug 17 03:11:48 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37Tue Aug 17 03:11:48 [initandlisten] waiting for connections on port 27018Tue Aug 17 03:11:48 [websvr] web admin interface listening on port 28018
[root@chn-xg-5-5c1 cache2]# 

3、配置config server，config server需要dbpath和repairpathdbpath默认为/data/configdb，repairpath默认为/data/db/，这与shard是不同的[root@localhost data]# mongod  --configsvr --fork --logpath /var/log/mongodb.log --logappend[1] 2144[root@localhost data]# Tue Aug 17 03:13:54 MongoDB starting : pid=2144 port=27019 dbpath=/data/configdb 32-bit 
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data**       see http://blog.mongodb.org/post/137788967/32-bit-limitations
Tue Aug 17 03:13:54 db version v1.6.0, pdfile version 4.5Tue Aug 17 03:13:54 git version: 2c7f164b653f0d703947572ede064aed41cc2185Tue Aug 17 03:13:54 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37Tue Aug 17 03:13:54 [initandlisten] diagLogging = 1Tue Aug 17 03:13:54 [initandlisten] waiting for connections on port 27019Tue Aug 17 03:13:54 [websvr] web admin interface listening on port 28019
[root@localhost data]# 
4、配置mongos router[root@band-all local]# mongos --configdb 192.168.100.245:27019 --fork --logpath /var/log/mongodb.log --logappend[1] 6384[root@band-all local]# Tue Aug 17 03:46:12 mongos db version v1.6.0, pdfile version 4.5 starting (--help for usage)Tue Aug 17 03:46:12 git version: 2c7f164b653f0d703947572ede064aed41cc2185Tue Aug 17 03:46:12 sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37Tue Aug 17 03:46:12 [websvr] web admin interface listening on port 28017Tue Aug 17 03:46:12 waiting for connections on port 27017
[root@band-all local]# 二、配置shard 集群
1、登陆mongos server，添加shard[root@localhost data]# mongo 192.168.100.229:27017/adminMongoDB shell version: 1.6.0connecting to: 192.168.100.229:27017/adminTue Aug 17 05:59:38 [conn4] building new index on { _id: 1 } for config.databasesTue Aug 17 05:59:38 [conn4] Buildindex config.databases idxNo:0 { name: "_id_", ns: "config.databases", key: { _id: 1 } }Tue Aug 17 05:59:38 [conn4] done for 0 records 0.025secs添加shard 1>  db.runCommand( { addshard : "192.168.100.210:27018" } );{ "shardAdded" : "shard0000", "ok" : 1 }添加shard 2创建shard时，两个shard不能有相同的数据库>  db.runCommand( { addshard : "192.168.100.212:27018" } );{        "ok" : 0,        "errmsg" : "trying to add shard 192.168.100.212:27018 because local database recommender exists in another shard0000:192.168.100.210:27018"}可以在192.168.100.212:27018通过 db.dropDatabase() 删除recommenderdb.runCommand( { addshard : "192.168.100.212:27018" } );查看shard节点列表> db.runCommand( { listshards : 1 } );{        "shards" : [                {                        "_id" : "shard0000",                        "host" : "192.168.100.210:27018"                },                {                        "_id" : "shard0001",                        "host" : "192.168.100.212:27018"                }        ],        "ok" : 1}> 将数据库设置为shard模式> db.runCommand( { enablesharding : "test" } );{ "ok" : 1 }
> db.runCommand( { enablesharding : "recommender" } );{ "ok" : 1 }> 
2、登陆mongos查看对集合进行分片
[root@band-all data]# mongo 192.168.100.229:27017/adminMongoDB shell version: 1.6.0connecting to: 192.168.100.229:27017/admin> show dbsadminlocal> use adminswitched to db admin> db.runCommand({listshards:1}){        "shards" : [                {                        "_id" : "shard0000",                        "host" : "192.168.100.210:27018"                },                {                        "_id" : "shard0001",                        "host" : "192.168.100.212:27018"                }        ],        "ok" : 1}> db.printShardingStatus(); --- Sharding Status ---   sharding version: { "_id" : 1, "version" : 3 }  shards:      { "_id" : "shard0000", "host" : "192.168.100.210:27018" }      { "_id" : "shard0001", "host" : "192.168.100.212:27018" }  databases:        { "_id" : "admin", "partitioned" : false, "primary" : "config" }
> config = connect("192.168.100.245:27019") //登陆config serverconnecting to: 192.168.100.245:27019192.168.100.245:27019> test = db.getSisterDB("test")       //新增数据库test
> db.runCommand( { enablesharding : "test" } ) //将test 数据库进行分片> db.runCommand( { shardcollection : "test.people", key : {name : 1} } )     //将数据集people进行分片>  db.printShardingStatus(); --- Sharding Status ---   sharding version: { "_id" : 1, "version" : 3 }  shards:      { "_id" : "shard0000", "host" : "192.168.100.210:27018" }      { "_id" : "shard0001", "host" : "192.168.100.212:27018" }  databases:        { "_id" : "admin", "partitioned" : false, "primary" : "config" }        { "_id" : "test", "partitioned" : true, "primary" : "shard0000" }                test.people chunks:                        { "name" : { $minKey : 1 } } -->> { "name" : { $maxKey : 1 } } on : shard0000 { "t" : 1000, "i" : 0 }

> use testswitched to db test> show collectionspeoplesystem.indexes批量插入数据for (var i = 1; i <= 200000; i++) db.people.save({name:"fly_people",sex:"female",age:25});
for (var i = 1; i <= 20000000; i++) db.people.save({name:"houxuefeng",sex:"male",age:30});
> db.people.stats(){        "sharded" : true,        "ns" : "test.people",        "count" : 462008,        "size" : 33264600,        "avgObjSize" : 72.00005194715243,        "storageSize" : 902522624,        "nindexes" : 2,        "nchunks" : 4,        "shards" : {                "shard0000" : {                        "ns" : "test.people",                        "count" : 402006,                        "size" : 28944448,                        "avgObjSize" : 72.00003980040098,                        "storageSize" : 891340544,                        "numExtents" : 22,                        "nindexes" : 2,                        "lastExtentSize" : 155079936,                        "paddingFactor" : 1,                        "flags" : 1,                        "totalIndexSize" : 58966016,                        "indexSizes" : {                                "_id_" : 18694144,                                "name_1" : 40271872                        },                        "ok" : 1                },                "shard0001" : {                        "ns" : "test.people",                        "count" : 60002,                        "size" : 4320152,                        "avgObjSize" : 72.00013332888904,                        "storageSize" : 11182080,                        "numExtents" : 6,                        "nindexes" : 2,                        "lastExtentSize" : 8388608,                        "paddingFactor" : 1,                        "flags" : 1,                        "totalIndexSize" : 5275648,                        "indexSizes" : {                                "_id_" : 2498560,                                "name_1" : 2777088                        },                        "ok" : 1                }        },        "ok" : 1}> 
至此MongoDB auto shard配置完毕
备注：问题1、当test.people数据量达到一定量时，这也就是32位限制，新数据无法入库，查看数据库文件也不在增加
>   for (var i = 1; i <= 20000; i++) db.people.save({name:"houxuefeng",sex:"male",age:30});can't map file memory - mongo requires 64 bit build for larger datasets> 
> db.people.stats();{        "ns" : "test.people",        "count" : 10128817,        "size" : 729274896,        "avgObjSize" : 72.00000710843132,        "storageSize" : 891340544,        "numExtents" : 22,        "nindexes" : 2,        "lastExtentSize" : 155079936,        "paddingFactor" : 1,        "flags" : 1,        "totalIndexSize" : 1241788192,        "indexSizes" : {                "_id_" : 420325264,                "name_1" : 821462928        },        "ok" : 1}> [root@chn-xg-5-5c1 db]# ll -lh总用量 3.0G-rwxr-xr-x  1 root root    6  8月 18 17:42 mongod.lockdrwxr-xr-x  3 root root 4.0K  8月 19 10:48 moveChunk-rw-------  1 root root  64M  8月 18 17:56 test.0-rw-------  1 root root 128M  8月 18 17:56 test.1-rw-------  1 root root 256M  8月 19 10:46 test.2-rw-------  1 root root 512M  8月 19 10:48 test.3-rw-------  1 root root 512M  8月 19 10:52 test.4-rw-------  1 root root 512M  8月 19 10:58 test.5-rw-------  1 root root 512M  8月 19 11:04 test.6-rw-------  1 root root 512M  8月 19 11:12 test.7-rw-------  1 root root  16M  8月 18 17:56 test.nsdrwxr-xr-x  2 root root 4.0K  8月 18 17:58 _tmp[root@chn-xg-5-5c1 db]# 问题2、创建shard时，两个shard不能有相同的数据库>  db.runCommand( { addshard : "192.168.100.212:27018" } );{        "ok" : 0,        "errmsg" : "trying to add shard 192.168.100.212:27018 because local database recommender exists in another shard0000:192.168.100.210:27018"}可以在192.168.100.212:27018通过 db.dropDatabase() 删除recommender问题3、replica set 配置完毕后才可以添加为shard，否则会出错> db.runCommand( { addshard : "foo/192.168.100.212:27018,192.168.100.212:27019,192.168.100.212:27020" } );{ "shardAdded" : "shard0001", "ok" : 1 }>