golang kafka client

lanyangsh · · 10428 次点击 · · 开始浏览    
这是一个创建于 的文章,其中的信息可能已经有所发展或是发生改变。

针对golang的 kafka client 有很多开源package,例如sarama, confluent等等。在使用sarama 包时,高并发中偶尔遇到crash。于是改用confluent-kafka-go,其简单易用,并且表现稳定。

本文主要介绍confluent-kafka-go的使用方法。
confluent-kafka-go,是kafka官网推荐的golang package。

confluent-kafka-go is Confluent's Golang client for Apache Kafka and the Confluent Platform.

编译环境搭建

安装librdkafka

下载

$ git clone https://github.com/edenhill/librdkafka.git
$ cd librdkafka

配置、编译、安装

$ ./configure --prefix /usr
$ make
$ sudo make install

配置PKG_CONFIG_PATH

在文件~/.bashrc 末尾添加

export PKG_CONFIG_PATH=/usr/lib/pkgconfig

下载go client

$ go get -u github.com/confluentinc/confluent-kafka-go/kafka

自动下载到GOPATH目录下,也可到github上自行下载,然后放到GOPATH中。

Example

// Example function-based Apache Kafka producer
package main

/**
 * Copyright 2016 Confluent Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

import (
    "fmt"
    "github.com/confluentinc/confluent-kafka-go/kafka"
    "os"
)

func main() {

    if len(os.Args) != 3 {
        fmt.Fprintf(os.Stderr, "Usage: %s <broker> <topic>\n",
            os.Args[0])
        os.Exit(1)
    }

    broker := os.Args[1]
    topic := os.Args[2]

    p, err := kafka.NewProducer(&kafka.ConfigMap{"bootstrap.servers": broker})

    if err != nil {
        fmt.Printf("Failed to create producer: %s\n", err)
        os.Exit(1)
    }

    fmt.Printf("Created Producer %v\n", p)

    // Optional delivery channel, if not specified the Producer object's
    // .Events channel is used.
    deliveryChan := make(chan kafka.Event)

    value := "Hello Go!"
    err = p.Produce(&kafka.Message{TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny}, Value: []byte(value)}, deliveryChan)

    e := <-deliveryChan
    m := e.(*kafka.Message)

    if m.TopicPartition.Error != nil {
        fmt.Printf("Delivery failed: %v\n", m.TopicPartition.Error)
    } else {
        fmt.Printf("Delivered message to topic %s [%d] at offset %v\n",
            *m.TopicPartition.Topic, m.TopicPartition.Partition, m.TopicPartition.Offset)
    }

    close(deliveryChan)
}

注意:
如果需要链接静态库,可删除/usr/lib/下面关于rdkafka的动态库文件(.so文件)。然后,go build编译时加上选项 –tags static
例如:

go build -tags static produer.go

更多example,可参考
https://github.com/confluentinc/confluent-kafka-go/tree/master/examples

参考

https://github.com/confluentinc/confluent-kafka-go


有疑问加站长微信联系(非本文作者)

本文来自:博客园

感谢作者:lanyangsh

查看原文:golang kafka client

入群交流(和以上内容无关):加入Go大咖交流群,或添加微信:liuxiaoyan-s 备注:入群;或加QQ群:692541889

10428 次点击  
加入收藏 微博
暂无回复
添加一条新回复 (您需要 登录 后才能回复 没有账号 ?)
  • 请尽量让自己的回复能够对别人有帮助
  • 支持 Markdown 格式, **粗体**、~~删除线~~、`单行代码`
  • 支持 @ 本站用户;支持表情(输入 : 提示),见 Emoji cheat sheet
  • 图片支持拖拽、截图粘贴等方式上传