conveyor

Transport log-entity via conveyor. Inspired by AliyunContainerService/log-pilot. But more flexible.

项目地址：github.com/chenjiandongx/conveyor

conveyor 是采一个负责采集 docker 容器日志的组件，使用 Porter 将特定容器产生 的日志输出到指定后端，如 kafka/elasticsearch/redis/... 目前 porter 具体实现有 filebeat-porter.

使用

本地开发构建

安装

GOPATH mode

$ go get -u github.com/chenjiandongx/conveyor/...

GOMODULE mode

require (
  github.com/chenjiandongx/conveyor
)

示例

package main

import (
    conveyor "github.com/chenjiandongx/conveyor/pkg"
)

func main() {
    // 实例化 porter
    porter := conveyor.NewFileBeatPorter(nil)
    // 实例化 conveyor
    cy := conveyor.NewConveyor("")
    // 将 porter 注册到 conveyor 中
    cy.RegisterPorter(porter)
    // 运行 conveyor
    cy.Run()
}

Porter Interface/ ContainerInfo Struct

type Porter interface {
    List(containers []*ContainerInfo) error
    Create(container *ContainerInfo) error
    Delete(container *ContainerInfo) error
    Run()
}

type ContainerInfo struct {
    ID      string
    Name    string
    Env     map[string]string
    Labels  map[string]string
    LogPath []string
}

使用 Docker 运行

运行 conveyor

容器启动的时候会优先读取 /etc/filebeat/filebeat.yaml 和 /etc/filebeat/configs/config.tmpl 两个配置文件，不存在则使用默认 配置。${your_var} 均为可选参数，非必须。

$ docker run -d --restart=always --name conveyor \
   -v /var/run/docker.sock:/var/run/docker.sock
   -v /:/host:ro
   -v ${your_filebeat_data_dir}:/etc/filebeat/data
   -v ${your_filebeat_base_confile_file}:/etc/filebeat/filebeat.yaml
   -v ${your_filebeat_custom_confile_tmpl}:/etc/filebeat/configs/config.tmpl
   -e CONVEYOR_NAME=${your_conveyor_name}
   -e CONVEYOR_PATH=${your_custom_path}
   chenjiandongx/conveyor:latest

默认 /etc/filebeat/filebeat.yaml

# 标准 filebeat 配置文件
filebeat.config.inputs:
  enabled: true
  path: /etc/filebeat/configs/*.yaml
  reload.enabled: true
  reload.period: 10s
output.console:
  pretty: true

默认 /etc/filebeat/configs/config.tmpl

# 标准 golang 模板语言
- type: log
  paths:
  - "/tmp/tmp.log"
{{- range . }}
- type: log
  paths:
  {{- range .LogPath }}
  - "{{ . }}"
  {{- end }}
  fields:
  {{- range $k, $v := .Labels }}
    {{ $k }}: {{ $v }}
  {{- end }}
{{- end }}

运行示例容器

启动容器后向 nginx 发送请求再查看 conveyor 的日志，可以看到日志被输出到标准输出。

$ docker run -d -e CONVEYOR_ENABLED=true --name ngx nginx

容器环境变量

使用 Kubernetes 运行

使用 DaemonSet 形式部署 conveyor

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: conveyor-es
spec:
  selector:
    matchLabels:
      name: conveyor-es
  template:
    metadata:
      labels:
        name: conveyor-es
    spec:
      containers:
      - name: conveyor
        image: chenjiandongx/conveyor:latest
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        # 挂载 docker.sock 文件，监听 dockerd 事件
        - name: docker-sock
          mountPath: /var/run/docker.sock
        # 挂载 / 路径，只读权限，用于日志收集
        - name: docker-log
          mountPath: /host
          readOnly: true
        - name: filebeat-config
          mountPath: /etc/filebeat/filebeat.yaml
          subPath: filebeat.yaml
      volumes:
      - name: docker-sock
        hostPath:
          path: /var/run/docker.sock
      - name: docker-log
        hostPath:
          path: /
      - name: filebeat-config
        configMap:
          name: filebeat-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
data:
  filebeat.yaml: |
    filebeat.config.inputs:
      enabled: true
      path: /etc/filebeat/configs/*.yaml
      reload.enabled: true
      reload.period: 10s
    output.elasticsearch:
      hosts: ["http://elasticsearch-svc:9200"]

部署 nginx depolyments 实例

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ngx
spec:
  replicas: 2
  selector:
    matchLabels:
      run: ngx
  template:
    metadata:
      labels:
        run: ngx
    spec:
      containers:
      - image: nginx
        # 可另外挂载空白卷追踪自定义日志文件
        volumeMounts:
         - mountPath: /tmp/logs
           name: tmp-log
        name: ngx
        env:
        - name: CONVEYOR_ENABLED
          value: "true"
        # 定义自定义日志路径
        - name: CONVEYOR_PATH
          value: "/tmp/logs/*.log"
      # 声明空白卷
      volumes:
      - emptyDir: {}
        name: tmp-log