I wrote a Go port of the Kinesis Client Library

<p>I really like the <a href="https://github.com/awslabs/amazon-kinesis-client" rel="nofollow">Kinesis Client Library</a>. Having used it in anger a few times I&#39;ve found it to be very robust and a good abstraction from the Kinesis API. But using it with Golang requires installing Java and a nasty hack to interface the too, which is pretty heavy weight and means you can&#39;t have a static binary deployment. </p> <p>So I wrote a <a href="https://github.com/patrobinson/gokini" rel="nofollow">Go port of it</a>, it doesn&#39;t yet have feature parity but the core features like co-ordinating shard allocation are there.</p> <p>Depending on the success of this I may implement the remaining features or start writing a Kinesis Producer Library.</p> <hr/>**评论：**<br/><br/>qu33ksilver: <pre><p>A Go Kinesis library is already out there from amazon -<a href="https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/" rel="nofollow">https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/</a></p> <p>Is the java library better than this ?</p></pre>TrollFactoryEmployee: <pre><p>Yes very much so. My library currently implements:</p> <ul> <li><p>Enumerating shards, if you add new shards it automatically discovers them and starts retrieving data from them.</p></li> <li><p>Co-ordinating shard allocation to workers. Multiple workers utilise DynamoDB to lock a shard, ensuring only one worker processes a shard.</p></li> <li><p>Parallel processing of shards. You provide a process function and it launches a go routine of that function for each shard.</p></li> <li><p>Checkpointing and recovery. The sequence number is updated in DynamoDB and if the node crashes it recovers this checkpoint.</p></li> </ul> <p>The java library also implements the following, which my library doesn&#39;t yet:</p> <ul> <li><p>Automatic shard rebalancing, it will ensure shards are evenly distributed amongst workers.</p></li> <li><p>Publish metrics to CloudWatch about how many records have been processed, how far behind the workers are amongst other things.</p></li> </ul></pre>TrollFactoryEmployee: <pre><p>The <a href="https://godoc.org/github.com/patrobinson/gokini#example-RecordConsumer" rel="nofollow">example code</a> gives a good example of how to use it, it&#39;s a lot simpler than implementing one yourself and makes your workers stateless.</p></pre>absdevops: <pre><p>Pretty cool - haven&#39;t read through the SDK but is it not just worth sending in a PR?</p></pre>