【编者的话】本次分享给大家介绍使用Packer一键解决方案安装Jenkins，以及使用一些gcloud API将我们build出来的Jenkins image部署到GCP等， 平台搭建好之后就开心的开发你的产品吧。
【深圳站｜3天烧脑式Kubernetes训练营】培训内容包括：Kubernetes概述和架构、部署和核心机制分析、进阶篇——Kubernetes调工作原理及源码分析等。
我们一直在为服务提供商研究新的创新的解决方案，使得各个领域都可以使用我们的服务，因此我们需要一个普遍性的Google Cloud云平台解决方案。这个平台可以解决移动应用和桌面应用的部署和build工作，同时简化使用Google Cloud平台的工具。
除了应用程序平台本身，还需要精简和重复的自动化流程和技术部署，以支持每个平台环境的配置，并支持所有标准的SDLC开发任务，以满足行业或组织要求。
面对选择IaaS / PaaS提供商和开发技术的任务时，谷歌云平台和Firebase是明显的选择。
为什么选择Firebase？
- Firebase为现代应用程序开发提供了最佳选择，支持Android，iOS和Web
- Firebase与Google OAuth验证集成out of the box（OOTB）
- Firebase支持Google，Email / Password，GitHub，Facebook，Twitter的联合身份验证提供商，甚至支持自定义实现
- Firebase为您的应用程序数据和使用情况提供Analytics
- Firebase提供应用程序测试自动化和视频播放的执行与Test Lab的Android和iOS应用程序
- Firebase由Google Cloud Platform基础架构资源和服务提供100％的支持
- 阅读更多关于Firebase features
为什么谷歌云平台?
- GCP provides……
- GCP支持集成……
- GCP在Kubernetes的集装箱计算领域处于领先地位，并提供最实用和最易于管理的解决方案
- GCP是一个真正的No-Ops平台，具有最方便的工具，如他们的Cloud Shell和Stackdriver监控解决方案
试点平台配置
GCP Cloud Shell说明
需要在GCP控制台中从Google Cloud Shell运行以下说明。 这些脚本及其中使用的工具将自动配置为支持FSO平台的CI / CD进程的Jenkins，配置Compute Engine VM映像。需要的工具：
1.在您的本地开发环境中安装firebase-tools包：
npm install -g firebase-tools
2.确保您在GCP项目的帐户中拥有管理员/所有者权限。
步骤：
在us-east1创建新的GCP项目
通过Firebase Web控制台将GCP项目导入到Firebase
为我们的环境配置获取一个Firebase CI token
a. 在jenkins VM中，运行以下命令获取CI token
i. firebase login:ci
ii. 在管理Jenkins->配置系统 - >更新FIREBASE_TOKEN界面，将token放入您的jenkins-url中
b. 或者，如果您无权访问本地主机，则可以运行以下操作：
i. firebase login:ci --no-localhost
ii. 从弹出窗口复制文本并粘贴回您的命令提示符
ii. 复制显示的token并将其添加到配置中
启用新GCP项目的计费
在新的Firebase项目上启用“Blaze”计费包
启用/配置以下API
a. Cloud Resource Manager
b. Compute Engine APIs
c. Firebase Rules APIs (automatically enabled if GCP project is imported into Firebase)
d. Google Cloud Messaging API (for the firebase functions feature)
e. Google Identity and Access Management API (for creating svc-acct keys/modifying IAM policies etc)
f. Google Cloud Testing API (for integration with FB Test Lab for android)
g. Google Cloud Tool Results API (for integration with FB Test Lab for android)
h. Google Cloud DNS (for creating DNS entries for the new jenkins vm)
i. Google Cloud Container Builder API (for app engine deployments)
j. Google Cloud Functions API (Private API - may require admin to enable)
k. Identity Toolkit API (For Google OAuth)
l. Google Maps API For Javascript
m. Google Maps Android API
n. Google Maps Geocoding API
i. 注意：启用并检查提供程序列表中的“Google”框后，您必须导航到“设置”选项卡
在此沙箱项目的Firebase项目中启用密码验证提供程序（以支持Android测试）
a. 导航到与此GCP项目关联的Firebase项目，然后从左侧面板菜单中单击Auth
b. 在“验证”面板中，单击“登录方法”
c. 点击电子邮件/密码，然后选择“启用”
d. 点击Google，然后选择“启用”
在GCP项目中创建OAuth ID
a. Go to API Manager > Credentials
b. Create new OAuth ID
c. Select Web Application as the type and using 'Jenkins CI' as the name
d.对于授权重定向字段，请根据以下模式指定此新Jenkins环境的最终URL
i.http://$JENKINS_URL-ci.$GCDNS_ZONE.$OAUTH_DOMAIN/securityRealm/finishLogin
e.保存我们的配置文件的OAuth Client ID和Client Secret
获取我们的环境配置文件的Firebase数据库密码
a. Navigate to Firebase Console > Project Settings > Database > Show Secret copy the value into our config file.
Clone your devops project
通过替换您的环境值创建您的GCP环境配置
export GCP_PROJECT_ID="cicd-test"
export AUTOMATION_BUCKET="cicd-test-automation"
export OAUTH_CLIENT_ID="oauth-client-id-here"
export OAUTH_CLIENT_SECRET="somerandomstringLfQOG0kpFuaveQOjiIAS"
export OAUTH_DOMAIN="example.net"
export SVC_ACCT_NAME="cicd-test"
export GCE_AZ="us-east1-b"
export GCDNZ_ZONE="project-name"
export JENKINS_URL="cicd-test"
export GIT_REPO="my-repo"
export GIT_ORG="my-github-org"
export GIT_USER="jenkins-ci-user:my-secret"
export FIREBASE_TOKEN="firebase-token"
export FIREBASE_PROJECT="cicd-test"
export FIREBASE_DB_SECRET="firebase-db-secret"
export SLACK_CHANNEL="project-slack-channel"
export SLACK_TOKEN="slack-token"
export SLACK_DOMAIN="project-slack-domain"
export SLACK_CI_SERVER="http://$JENKINS_URL.$GCDNZ_ZONE.$OAUTH_DOMAIN"
export CREATE_DNS_ENTRIES=false
export CREATE_FIREWALL_RULES="true"
注意：不要修改以下变量：JENKINS_URL，SLACK_CI_SERVER
./create-jenkins-instance.sh -f ./config.env
运行后脚本手动任务：
- 复制新推出的Jenkins Compute Engine VM的外部IP地址
- In the google project, navigate to Networking > Cloud DNS:
>使用以下模式创建2个新的DNS记录：
脚本是如何工作的
读取config# set explicitly through the packerInstall function
PACKER_HOME=""
PACKER_URL="https://releases.hashicorp.com/packer/0.12.3/packer_0.12.3_linux_amd64.zip"
GCP_PROJECT_ID=""
SVC_ACCT_NAME=""
AUTOMATION_BUCKET=""
OAUTH_CLIENT_ID=""
OAUTH_CLIENT_SECRET=""
OAUTH_DOMAIN=""
GCE_AZ=""
GCDNS_ZONE=""
JENKINS_URL=""
GIT_REPO=""
GIT_ORG=""
GIT_USER=""
FIREBASE_TOKEN=""
FIREBASE_PROJECT=""
FIREBASE_DB_SECRET=""
SLACK_CHANNEL=""
SLACK_TOKEN=""
SLACK_DOMAIN=""
SLACK_CI_SERVER=""
CREATE_FIREWALL_RULES=""
CREATE_DNS_ENTRIES=""
# change this value if jenkins is installed elsewhere on the target machine
JENKINS_HOME="/var/lib/jenkins"
while getopts ":f:p:n:b:i:s:d:a:z:j:o:r:u:t:P:" opt; do
case $opt in
f)
processPropertyFile $OPTARG
break
;;
p)
GCP_PROJECT_ID=$OPTARG
;;
n)
SVC_ACCT_NAME=$OPTARG
;;
b)
AUTOMATION_BUCKET=$OPTARG
;;
i)
OAUTH_CLIENT_ID=$OPTARG
;;
s)
OAUTH_CLIENT_SECRET=$OPTARG
;;
d)
OAUTH_DOMAIN=$OPTARG
;;
a)
GCE_AZ=$OPTARG
;;
z)
GCDNS_ZONE=$OPTARG
;;
j)
JENKINS_URL=$OPTARG
;;
o)
GIT_ORG=$OPTARG
;;
u)
GIT_USER=$OPTARG
;;
r)
GIT_REPO=$OPTARG
;;
t)
FIREBASE_TOKEN=$OPTARG
;;
P)
FIREBASE_PROJECT=$OPTARG
;;
\?)
echo -e "\nUnknown option: $OPTARG"
printUsage
exit 1
esac
done
1.安装packer
verifyToolExists()
{
if [ $# -eq 1 ]; then
command -v $1 >/dev/null 2>&1 || { echo "Required tool '$1' is not installed on PATH. Aborting." >&2; exit 1; }
else
echo -e "Incorrect number of arguments: $#\nUsage: verifyToolExists <tool-name>\n"
fi
}
packerInstall()
{
local TIMESTAMP=`date +%s`
local PACKER_DIR="/tmp/packer-$TIMESTAMP"
if [ -d $PACKER_DIR ]; then
echo "Director already exists."
else
echo "Creating directory $PACKER_DIR"
mkdir -p $PACKER_DIR
echo "Performing pre-req install tasks..."
verifyToolExists unzip
verifyToolExists curl
curl -o $PACKER_DIR/packer.zip $PACKER_URL
unzip $PACKER_DIR/packer.zip -d $PACKER_DIR # && unzip PACKER_DIR/packer.zip
#change the name of the binary so it doesn't collide with native packer binary
export PATH=$PATH:$PACKER_DIR
echo "Verifying successful installation..."
verifyToolExists packer
echo "Successfully installed packer locally."
# set global variable for other shell processes to detect
PACKER_HOME=$PACKER_DIR
fi
}
2.创建firewall规则
gcloud compute firewall-rules create $rule_name --allow $rule_port --target-tags $rule_tags
3.创建存储配置文件的bucket
4.创建ssh key
TIMESTAMP=`date +%s`
SSH_PUBKEY=""
# create a github deployment ssh-key and save to GCS bucket.
# User will need to manually add this key to repo settings later.
ssh-keygen -f /tmp/$SVC_ACCT_NAME -t rsa -C "$SVC_ACCT_NAME" -N ''
SSH_PUBKEY=`cat /tmp/$SVC_ACCT_NAME.pub`
5.创建 deploy key and webhook
6.使用 gsutil 复制 config文件到bucket
7.创建 service account gcloud iam service-accounts create
8.更新 service account role
# create a service account for performing deployments etc
gcloud iam service-accounts create $SVC_ACCT_NAME --display-name "$SVC_ACCT_NAME-jenkins" > /tmp/service-account.info
# store the name of the service-account so that we can perform actions on it
#SVC_ACCT_EMAIL=`cat /tmp/service-account.info | grep 'email' | awk '{print $2}'`
# the output from the create command for service accounts no longer provides the email, so we must build it ourselves for now...
SVC_ACCT_EMAIL="$SVC_ACCT_NAME@$GCP_PROJECT_ID.iam.gserviceaccount.com"
# grant the required roles for the jenkins service-account to interact with the required services
updateProjectIamRole $GCP_PROJECT_ID $SVC_ACCT_EMAIL "roles/appengine.deployer"
updateProjectIamRole $GCP_PROJECT_ID $SVC_ACCT_EMAIL "roles/logging.logWriter"
updateProjectIamRole $GCP_PROJECT_ID $SVC_ACCT_EMAIL "roles/storage.admin"
updateProjectIamRole $GCP_PROJECT_ID $SVC_ACCT_EMAIL "roles/editor"
9.Packer 去build image文件
Jenkins-master.vars
{
"gcp_project_id": "VAR_GCP_PROJECT_ID",
"gce_zone_id" : "VAR_GCE_ZONE_ID",
"gce_image_id" : "centos-7-v20160803",
"gce_svc_acct_name" : "VAR_GCE_SVC_ACCT_NAME",
"gce_svc_acct_id" : "VAR_GCE_SVC_ACCT_ID",
"gcs_bucket_name" : "VAR_GCS_BUCKET_NAME",
"oauth_client_id" : "VAR_GCP_OAUTH_CLIENT_ID",
"oauth_client_secret" : "VAR_GCP_OAUTH_CLIENT_SECRET",
"oauth_domain" : "VAR_GCP_OAUTH_DOMAIN",
"git_repo" : "VAR_GIT_REPO",
"firebase_token" : "VAR_FIREBASE_TOKEN",
"firebase_project" : "VAR_FIREBASE_PROJECT",
"firebase_db_secret" : "VAR_FIREBASE_DB_SECRET",
"slack_domain" : "VAR_SLACK_DOMAIN",
"slack_token" : "VAR_SLACK_TOKEN",
"slack_channel" : "VAR_SLACK_CHANNEL",
"slack_ci_server" : "VAR_SLACK_CI_SERVER"
}
Jenkins-master.json
{
"variables" : {
"gce_image_id": "",
"gcp_project_id" : "",
"gce_zone_id" : "",
"gce_svc_acct_name" : "",
"gce_svc_acct_id" : "",
"gcs_bucket_name" : "",
"oauth_client_id" : "",
"oauth_client_secret" : "",
"oauth_domain" : "",
"image_name" : "",
"slack_token" : "",
"firebase_token" : "",
"firebase_project" : "",
"firebase_db_secret" : "",
"slack_domain" : "",
"slack_channel" : "",
"slack_ci_server" : ""
},
"builders" : [{
"type": "googlecompute",
"communicator": "ssh",
"ssh_pty" : "true",
"project_id": "{{user `gcp_project_id`}}",
"source_image": "{{user `gce_image_id`}}",
"zone": "{{user `gce_zone_id`}}",
"ssh_username": "packer",
"image_name": "{{user `image_name`}}",
"metadata" : {
"service-account-name" : "{{user `gce_svc_acct_name`}}",
"service-account-id" : "{{user `gce_svc_acct_id`}}",
"automation-bucket" : "{{user `gcs_bucket_name`}}",
"oauth-client-id" : "{{user `oauth_client_id`}}",
"oauth-client-secret" : "{{user `oauth_client_secret`}}",
"oauth-domain" : "{{user `oauth_domain`}}",
"git-repo" : "{{user `git_repo`}}",
"firebase-token" : "{{user `firebase_token`}}",
"firebase-project" : "{{user `firebase_project`}}",
"firebase-db-secret" : "{{user `firebase_db_secret`}}",
"slack-channel" : "{{user `slack_channel`}}",
"slack-domain" : "{{user `slack_domain`}}",
"slack-token" : "{{user `slack_token`}}",
"slack-ci-server" : "{{user `slack_ci_server`}}"
}
}],
"provisioners" : [{
"type" : "shell",
"script" : "packer/jenkins/scripts/jenkins-install-script.sh",
"execute_command": "echo 'packer' | sudo -S sh '{{.Path}}'"
}]
}
packer build -var-file=/tmp/packer.vars -var image_name="$PACKER_IMAGE_NAME" packer/jenkins/templates/jenkins-master.json
Jenkins-install-script.sh
yum install -y epel-release
yum install -y bzip2
# install jq for processing json files
yum install -y jq.x86_64
# install nodejs 6.x and npm 3.x for RHEL for compatability with newer node libraries
curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -
yum install -y nodejs
# configure this jenkins instance for being able to publish libraries to npmjs.org (TODO: Needs password automated to work)
#echo "pwc-business-os-jenkins" | npm add-user --scope=@pwc-business-os
#npm config set scope @pwc-business-os
# install firebase sdk for build/test process and enable functions deployments
npm install -g firebase-tools
npm install -g gulp
npm install -g firebase-import
firebase --open-sesame functions
# install firebase-bolt for testing
npm install --global firebase-bolt
# install google-oauth-login
java -jar jenkins-cli.jar -s http://localhost:8080 -i /var/lib/jenkins/.ssh/id_rsa install-plugin google-oauth-plugin
# install google-login
java -jar jenkins-cli.jar -s http://localhost:8080 -i /var/lib/jenkins/.ssh/id_rsa install-plugin google-login
# install job-dsl
java -jar jenkins-cli.jar -s http://localhost:8080 -i /var/lib/jenkins/.ssh/id_rsa install-plugin job-dsl
# install workflow-aggregator (jenkins pipeline plugin)
java -jar jenkins-cli.jar -s http://localhost:8080 -i /var/lib/jenkins/.ssh/id_rsa install-plugin workflow-aggregator
# install build-timeout
java -jar jenkins-cli.jar -s http://localhost:8080 -i /var/lib/jenkins/.ssh/id_rsa install-plugin build-timeout
# install email-ext
java -jar jenkins-cli.jar -s http://localhost:8080 -i /var/lib/jenkins/.ssh/id_rsa install-plugin email-ext
# restart jenkins to apply the updated security login credentials
service jenkins stop && sleep 10
service jenkins start
echo "y" | gcloud components install beta
10.从packer build出来的image， 用gcloud 创建 image
gcloud compute instances create $SVC_ACCT_NAME-jenkins --image $PACKER_IMAGE_NAME --custom-cpu 4 --custom-memory 8GiB --zone $GCE_AZ --boot-disk-type pd-standard --boot-disk-size 250GB --tags http-server,https-server,http-appserver --maintenance-policy MIGRATE
11.部署一个sample to unlock default app engine
# clone the google repo and deploy a dummy app-engine app to unlock the 'default' module so that we can begin doing real deployments
APP_ENGINE_DIR="/tmp/app-engine-setup-$TIMESTAMP"
mkdir -p $APP_ENGINE_DIR && cd $APP_ENGINE_DIR
git clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git
cd nodejs-docs-samples/appengine/hello-world
npm set progress=false && npm install
gcloud app deploy app.yaml
# clean-up the app-engine work dir etc
cd && rm -rf $APP_ENGINE_DIR
12.创建default container engine cluster(kubernets cluster)
gcloud container clusters create default --num-nodes 5
13.安装kubectl in jenkins vm
gcloud components install kubectl
gcloud auth application-default login
gcloud container clusters get-credentials default
gcloud config set compute/zone us-central1-b
kubectl cluster-info
sudo ln -s /usr/local/share/google/google-cloud-sdk/bin/kubectl /bin/kubectl
我们如何部署产品
每个产品的repo Jenkinsfile- 部署之前，到Jenkins VM里面创建ssh key给你想要部署的程序使用
- 创建Webhook
- A sample Jenkinsfile is like，假定你已经有configMap.yaml，deployment.yaml 和service.yaml
configMap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: your-product-config
data:
SERVER_API_KEY: your-api-key
PROJECT_ID: your-google-project-id
TEMPLATE_ID_WO_COD: some-config-string-here
TEMPLATE_ID_WO_NONCOD: some-config-string-here
TEMPLATE_ID_TS: some-config-string-here
REST_API_ENDPOINTS: some-config-string-here
Deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: your-product
spec:
replicas: 2
revisionHistoryLimit: 25
template:
metadata:
labels:
app: your-product
spec:
containers:
- image: gcr.io/PROJECT_ID/your-product
name: your-product
volumeMounts:
- name: config-volume
mountPath: /etc/config
env:
- name: SERVER_API_KEY
valueFrom:
configMapKeyRef:
name: your-product-config
key: SERVER_API_KEY
- name: PROJECT_ID
valueFrom:
configMapKeyRef:
name: your-product-config
key: PROJECT_ID
- name: TEMPLATE_ID_WO_COD
valueFrom:
configMapKeyRef:
name: your-product-config
key: TEMPLATE_ID_WO_COD
- name: TEMPLATE_ID_WO_NONCOD
valueFrom:
configMapKeyRef:
name: your-product-config
key: TEMPLATE_ID_WO_NONCOD
- name: TEMPLATE_ID_TS
valueFrom:
configMapKeyRef:
name: your-product-config
key: TEMPLATE_ID_TS
- name: REST_API_ENDPOINTS
valueFrom:
configMapKeyRef:
name: your-product-config
key: REST_API_ENDPOINTS
ports:
- containerPort: 8080
volumes:
- name: config-volume
configMap:
name: your-product-service-acct
items:
- key: service-acct
path: svc-acct.json
- key: pubsub
path: pubsub.json
service.yaml
apiVersion: v1
kind: Service
metadata:
name: your-product-service
labels:
app: your-product
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
name: http
selector:
app: your-product
type: LoadBalancer
A good dockerfile for node.js
Dockerfile
FROM mhart/alpine-node:6
# we need grpc in node, thats why we add libc6-compat
RUN \
apk upgrade && \
apk add --update libc6-compat
# Provides cached layer for node_modules
ADD package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir -p /src && cp -a /tmp/node_modules /src/
# From here we load our application's code in, therefore the previous docker
# "layer" thats been cached will be used if possible
WORKDIR /src
ADD . /src
ENV NODE_ENV=production
EXPOSE 8080
CMD npm start
Jenkinsfile
def config
fileLoader.withGit('git@github.com:my-ord/my-jenkinsci-lib.git', 'master', null, '') {
config = fileLoader.load('config/cicd-config');
}
// Parse out the scm details and store in config object so we can reference later
config.parseGitConfig(env.JOB_NAME)
stage 'Pull CICD Config'
node {
deleteDir()
config.fetchEnvConfig(env.CONFIG_BUCKET)
sh "cat config.env"
sh "cat svc-acct.json"
stash includes: '**', name: 'build-config'
}
stage 'Install Nodejs sources'
node {
try {
if(config.isPR())
checkout scm
else
git url: "git@${repo}:${org}/${repo}", branch: "${branch}"
} catch (e) {
// If there was an exception thrown, the build failed
currentBuild.result = "FAILED"
throw e
}
}
stage 'Build image, push to GCR'
node {
project = "${env.FIREBASE_PROJECT}"
appName = 'my-app-name'
imageTag = "gcr.io/${project}/${appName}:${env.BRANCH_NAME}.${env.BUILD_NUMBER}"
sh("docker build -t ${imageTag} .")
sh("gcloud docker -- push ${imageTag}")
}
if(config.isPR() == false) {
stage 'Deploy Application'
node {
project = "${env.FIREBASE_PROJECT}"
appName = 'my-app-name'
svcName = "${appName}-service"
imageTag = "gcr.io/${project}/${appName}:${env.BRANCH_NAME}.${env.BUILD_NUMBER}"
try {
unstash 'build-config'
// create configMap
sh("kubectl create configmap my-app-name-service-acct --from-file=service-acct=./svc-acct.json --dry-run -o yaml | kubectl replace configmap my-app-name-service-acct -f -")
sh("kubectl apply -f k8s/configMap.yaml")
// Change image to the one we just built
sh("sed -i.bak 's#gcr.io/PROJECT_ID/my-app-name#${imageTag}#' ./k8s/deployment.yaml")
sh("kubectl apply -f k8s/deployment.yaml")
sh("kubectl apply -f k8s/service.yaml")
} catch (e) {
// If there was an exception thrown, the build failed
currentBuild.result = "FAILED"
throw e
}
}
}
echo "Successfully completed Pipeline."
现在你可以到kubernetes cluster里面查看你的应用程序
Note, we can also integrate with container build trigger to build images for us.
https://cloud.google.com/conta ... ggers
以上内容根据2017年04月11日晚微信群分享内容整理。分享人李修玉，普华用到技术专家，负责FSO的架构和DevOps，对Node.js、Golang、GCP和kubernetes感兴趣。 DockOne每周都会组织定向的技术分享，欢迎感兴趣的同学加微信：liyingjiesz，进群参与，您有想听的话题或者想分享的话题都可以给我们留言。
有疑问加站长微信联系（非本文作者）
