bigchaindb集群部署（5节点）

在安装bigchainDB集群之前，我们需要了解bigchainDB的一些相关术语：

• bigchainDB节点：是一台或一群相近的节点的集合。bigchainDB节点上运行了mongoDB server、bigchainDB server及一些相关的软件
• bigchainDB集群：一群互联的bigchainDB节点组成了bigchainDB集群。集群中的节点运行相同的软件。一个集群可能需要有其他的额外的节点来完成集群监控
• bigchainDB联盟（Consortium）：一个集群中管理节点的组织或个人属于bigchainDB联盟。当集群由一个公司管理运行，则联盟为该公司

bigchainDB的节点按照使用目的可以分为dev/test节点（开发与测试）、bare-bones节点（部署在云上）或者production节点（拥有更多的组件）。一个production节点必须包括BigchainDB Server、MongoDB Server 3.4+ (mongod)以及Scalable storage for MongoDB。这些关键组件的关系如下图所示。其中BigChainDB server必须能够与所有mongod实例相连，所有的mongod也相互连接。

安装环境

hostname	ip	os
node-admin	10.0.0.70	ubuntu 14.04 desktop
node1	10.0.0.71	ubuntu 14.04 server
node2	10.0.0.72	ubuntu 14.04 server
node3	10.0.0.73	ubuntu 14.04 server
node4	10.0.0.74	ubuntu 14.04 server
node5	10.0.0.75	ubuntu 14.04 server

其中node-admin用来使用ansible来执行远程命令（为了部署便利），node1-node5为bigchain DB节点。所有机器用户均为root

为了使用ansible，我们首先在所有节点上安装openssh-server，并允许root用户ssh。具体方式为修改/etc/ssh/sshd_config，修改为PermitRootLogin yes，然后重启ssh服务。

安装与配置ansible

接下来在node-admin上安装与配置ansible。

apt-get update && apt-get upgrade
apt-get install ansible

修改/etc/ansible/hosts:

root@clean:~# grep -Ev  "^$|#" /etc/ansible/hosts
[bigchain]
10.0.0.71 ansible_ssh_pass=123
10.0.0.72 ansible_ssh_pass=123
10.0.0.73 ansible_ssh_pass=123
10.0.0.74 ansible_ssh_pass=123
10.0.0.75 ansible_ssh_pass=123
root@clean:~#

设置为首次链接不需要key认证

sed -i "s/^#\(host_key_checking\).*/\1 = False/g" /etc/ansible/ansible.cfg

测试：ansible bigchain -m ping

安装bigchainDB

以下所有命令默认在node-admin上执行

• 同步时钟

# 测试是否能ping通cn.pool.ntp.org
ansible bigchain -m command -a "ping cn.pool.ntp.org -c 4"
# 同步
ansible bigchain -m command -a "ntpdate cn.pool.ntp.org"

• 安装mongoDB

编辑sources.list

echo "deb [ arch=amd64 ] http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.4 multiverse" > mongodb-org-3.4.list

备用 清华的源

echo "deb [ arch=amd64 ] https://mirrors.tuna.tsinghua.edu.cn/mongodb/apt/ubuntu trusty/mongodb-org/3.4 multiverse" > mongodb-org-3.4.list

编辑mongod.yml，内容如下：

---

- hosts: bigchain
  remote_user: root
  # invoke setup module to gather facts before executing tasks
  gather_facts: true

  tasks:
  - name: debug
    debug: msg="myhostname={{ansible_hostname}}"

  - name: apt-key
    command: sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6

  - name: sources.list
    template: 
      src: mongodb-org-3.4.list
      dest: /etc/apt/sources.list.d/mongodb-org-3.4.list
      owner: root
      group: root
      mode: 0644

  - name: update
    command: apt-get update -y

  - name: install packages
    apt:
      name: "{{item}}"
      force: yes
    with_items:
      - mongodb-org
      - g++ 
      - python3-dev
      - libffi-dev
      - python3-pip

  - name: setuptools
    command: pip3 install --upgrade pip setuptools

  - name: db
    command: mkdir -p /data/db

运行yml(耗时很长，因为需要完成apt-get update与安装许多包)

ansible-playbook mongod.yml

• 启动mongod

# 启动mongo
ansible bigchain -m command -a "mongod --replSet bigchain --fork --logpath /var/log/mongodb/mongod.log"

若出现address已经被绑定这种错误的话，可以在错误节点上执行

kill -9 $(ps -ef | grep mongod | awk 'NR>1{print p}{p=$2}')
# kill全部
ansible bigchain -m raw -a "kill -9 \$(ps -ef | grep mongod | awk 'NR>1{print p}{p=\$2}')"

若mongod出现错误：

2017-06-21T09:56:34.552+0800 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2017-06-21T09:56:34.568+0800 E NETWORK  [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock Operation not permitted
2017-06-21T09:56:34.568+0800 I -        [initandlisten] Fatal Assertion 28578 at src/mongo/util/net/listen.cpp 195

可以在错误节点上执行：

rm /data/db/mongod.lock  
mongod --repair

ansible bigchain -m raw -a "rm /data/db/mongd.lock"
ansible bigchain -m raw -a "mongod --repair"

• 创建副本集

在node1上执行

mongo

    config = {_id: 'bigchain', members: [{
    "_id": 0,
    "host":"10.0.0.71:27017"
    }]
    }

    rs.initiate(config);
    rs.add("10.0.0.72:27017")
    rs.add("10.0.0.73:27017")
    rs.add("10.0.0.74:27017")
    rs.add("10.0.0.75:27017")

• 安装bigchainDB

ansible bigchain -m command -a "pip3 install --upgrade bigchaindb"
ansible bigchain -m command -a "bigchaindb -y configure mongodb"

• 修改bigchainDB配置

允许接收所有地址信息

ansible bigchain -m raw -a "sed -i 's/\(\"bind\": \"\)localhost:9984\"/\10.0.0.0:9984\"/g' /root/.bigchaindb"

修改replSet名称为bigchain（之前mongod的replset名称）：

ansible bigchain -m raw -a "sed -i 's/\(\"replicaset\": \"\).*/\1bigchain\",/g' /root/.bigchaindb"

还需要修改keyring选项，使之存储除本节点之外的所有其他的节点的keyring的公钥。先来获取所有节点的keyring的公钥

ansible bigchain -m raw -a "cat .bigchaindb | grep public | awk -F\\\" '{print \$4}'"

输出如下：

10.0.0.71 | success | rc=0 >>
7772APkwHENC8j3tDaUK2WJYPF3AMrTkVgR7sW1y3bkZ


10.0.0.72 | success | rc=0 >>
GRTkTmFuYETDaXAftSZW1SdCMMwaYs6p6yhAn5C4QBZv


10.0.0.74 | success | rc=0 >>
Eok1FnDbKpak9t6SpJVpFsMqkvNiVGsys6BP8UbSiCTv


10.0.0.73 | success | rc=0 >>
8bXEbEJVCDNhptYyAJ5WWHCngiie6VuwTKF5NmZ4Fazv


10.0.0.75 | success | rc=0 >>
GH3uAPwi1MzXsxy4PJdj4p5m55nXuLAakNtpFNJw7cqH

我们将这些信息写入一个脚本(conf.py)里

import sys
import json

keyring = {
    "10.0.0.71": "7772APkwHENC8j3tDaUK2WJYPF3AMrTkVgR7sW1y3bkZ",
    "10.0.0.72": "GRTkTmFuYETDaXAftSZW1SdCMMwaYs6p6yhAn5C4QBZv",
    "10.0.0.74": "Eok1FnDbKpak9t6SpJVpFsMqkvNiVGsys6BP8UbSiCTv",
    "10.0.0.73": "8bXEbEJVCDNhptYyAJ5WWHCngiie6VuwTKF5NmZ4Fazv",
    "10.0.0.75": "GH3uAPwi1MzXsxy4PJdj4p5m55nXuLAakNtpFNJw7cqH"
}

rets = []
for key, value in keyring.items():
    if key != sys.argv[1]:
        rets.append(value)

conf = json.load(open("/root/.bigchaindb"))
conf['keyring'] = rets

json.dump(conf, open("/root/.bigchaindb", "w"), indent=2)

编辑bigchain.yml，用来分发该脚本，并执行该脚本

---

- hosts: bigchain
  remote_user: root
  # invoke setup module to gather facts before executing tasks
  gather_facts: true

  tasks:
  - name: debug
    debug: msg="my ip of eth0 is {{ansible_eth0.ipv4.address}}"

  - name: copy file
    template:
      src: conf.py
      dest: /root/conf.py
      owner: root
      group: root
      mode: 0644

  - name: modify configuration
    command: python conf.py {{ansible_eth0.ipv4.address}}

执行：

ansible-playbook bigchain.yml

执行成功后，可以看到node1-node5的.bigchaindb里keyring均成功写入。

• 启动bigchainDB

任选一节点（如node1上）执行

bigchaindb init

然后启动所有节点的bigchaindb

bigchaindb start

# 或者后台启动
nohup bigchaindb start > /dev/null 2>&1 &

# 启动全部
ansible bigchain -m shell -a "nohup bigchaindb start > /dev/null 2>&1 &"

# kill全部
ansible bigchain -m raw -a "kill -9 \$(ps -ef | grep bigchaindb | awk 'NR>1{print p}{p=\$2}')"

使用bigchaindb

首先得安装bigchaindb的python driver

ansible bigchain -m command -a "apt-get install git -y"

ansible bigchain -m command -a "apt-get install libssl-dev -y"

ansible bigchain -m command -a "pip3 install --process-dependency-links git+https://github.com/bigchaindb/bigchaindb-driver.git"

官网上的测试用例：https://docs.bigchaindb.com/projects/py-driver/en/latest/usage.html

# coding=utf-8
# author: Wu Luo

import bigchaindb_driver
from bigchaindb_driver import BigchainDB
from bigchaindb_driver.crypto import generate_keypair
from time import sleep
import json
from sys import exit
import sys

class BigChainAPI(object):

    def __init__(self, host, port=9984, conf='/root/.bigchaindb', timeout=60):
        '''
        :param host: cluster host of bigchaindb
        :param port: port of bigchaindb
        :param conf: the configuration file of bigchaindb
        :param timeout: waiting vote for `timeout` seconds
        :return:
        '''
        self.host = host
        self.port = int(port)
        self.bdb = BigchainDB("http://%s:%d" % (self.host, self.port))

        self.conf = conf
        self.timeout = int(timeout)

        self.loadUserKey()

    def loadUserKey(self):
        '''
        load user key from configuration files, as the valid keypairs are
         designated in our deployment
        :return:
        '''
        configuration = json.load(open(self.conf, "r"))
        self.user = configuration['keypair']

    def waitVote(self, txid):
        '''
        wait the transaction to be voted as valid
        :param txid: the transaction id
        :return: True/False
        '''
        print("txid is ", txid)
        trials = 0
        while trials < self.timeout:
            try:
                if self.bdb.transactions.status(txid).get('status') == 'valid':
                    print('Tx valid in:', trials, 'secs')
                    break
                elif self.bdb.transactions.status(txid).get('status') == 'invalid':
                    print('Tx invalid in:', trials, 'secs')
                    print(self.bdb.transactions.status(txid))
                    break
                else:
                    trials += 1
                    print("trials " + str(trials))
                    sleep(1)
            except bigchaindb_driver.exceptions.NotFoundError:
                trials += 1
                sleep(1)

        if trials == self.timeout:
            print('Tx is still being processed... Bye!')
            return False

        return True

    def writeTransaction(self, data, metadata=None):
        '''
        write data into bigchaindb
        :param data: the data to write. it should be a dict
        :param metadata: the metadata to write. it should be a dict
        :return: the transaction id
        '''
        _asset = {
            'data': data
        }

        prepared_creation_tx = self.bdb.transactions.prepare(
            operation='CREATE',
            signers=self.user['public'],
            asset=_asset,
            metadata=metadata
        )

        fulfilled_creation_tx = self.bdb.transactions.fulfill(
            prepared_creation_tx,
            private_keys=self.user['private']
        )

        sent_creation_tx = self.bdb.transactions.send(fulfilled_creation_tx)

        txid = fulfilled_creation_tx['id']

        if not self.waitVote(txid):
            return False

        return txid

    def transferAsset(self, asset_id, publicKeyOfTargetUser):
        '''
        transfer an asset to another user
        :param asset_id: the asset to transfer
        :param publicKeyOfTargetUser: the public key of the target user
         It should be generated by generate_keypair()
        :return: the transaction id
        '''

        transfer_asset = {
            'id': asset_id
        }

        # query the transaction which contains the asset
        data = self.bdb.transactions.retrieve(asset_id)

        output_index = 0
        output = data['outputs'][output_index]

        transfer_input = {
            'fulfillment': output['condition']['details'],
            'fulfills': {
                'output': output_index,
                # in other version of BigchainDB, the key should be 'txid'
                'transaction_id': data['id']
            },
            'owners_before': output['public_keys']
        }

        prepared_transfer_tx = self.bdb.transactions.prepare(
            operation='TRANSFER',
            asset=transfer_asset,
            inputs=transfer_input,
            recipients=publicKeyOfTargetUser,
        )

        fulfilled_transfer_tx = self.bdb.transactions.fulfill(
            prepared_transfer_tx,
            private_keys=self.user['private'],
        )

        sent_transfer_tx = self.bdb.transactions.send(fulfilled_transfer_tx)

        txid = fulfilled_transfer_tx['id']

        if not self.waitVote(txid):
            return False

        return txid

    def queryTransaction(self, txid):
        '''
        retrieve a transaction
        :param txid: the txid of the transaction to query
        :return: the data of the retrieved transaction
        '''
        data = self.bdb.transactions.retrieve(txid)

        return data['asset']

def test():
    bigchainAPI = BigChainAPI(host='10.0.0.71', port=9984, timeout=300)

    # write an transaction
    data = {
        'author': 'Wu Luo'
    }
    txid = bigchainAPI.writeTransaction(data)
    if txid == False:
        print(">>> waiting for vote")
        return False

    # query this transaction
    query_data = bigchainAPI.queryTransaction(txid)
    print(">>> txid is " + txid)
    print(">>> data retrieved from bigchainDB")
    print(query_data)

    # transfer
    user = generate_keypair()
    newid = bigchainAPI.transferAsset(txid, user.public_key)

    print(txid, newid)
    transfer_data = bigchainAPI.queryTransaction(newid)
    print(transfer_data)

test()

运行后

root@node1:~# python3 bigchainAPI.py
txid is  6a82600a7cd8683f8e83cc6cfb765b219a8543404da2a9ad5b56114b9f1a8dc5
trials 1
trials 2
Tx valid in: 2 secs
>>> txid is 6a82600a7cd8683f8e83cc6cfb765b219a8543404da2a9ad5b56114b9f1a8dc5
>>> data retrieved from bigchainDB
{'data': {'author': 'Wu Luo'}}
txid is  c6a7b301d485243a7404fd048d0cec3912410f58ccd20701bc8937d7587516a8
trials 1
trials 2
Tx valid in: 2 secs
6a82600a7cd8683f8e83cc6cfb765b219a8543404da2a9ad5b56114b9f1a8dc5 c6a7b301d485243a7404fd048d0cec3912410f58ccd20701bc8937d7587516a8
{'id': '6a82600a7cd8683f8e83cc6cfb765b219a8543404da2a9ad5b56114b9f1a8dc5'}

部署成功