K8s go-client - channel flooded when watching HPA resource events

xuanbao · · 639 次点击    
这是一个分享于 的资源,其中的信息可能已经有所发展或是发生改变。
<p>version: 1.9.3-gke.0</p> <p>Once every 30-45 mins I get a huge spike of nil events ~50k-1mm large groups having same timestamp</p> <p>I&#39;m running a pod inside the cluster the code is below. I&#39;ve included some comments.</p> <pre><code>hpaWatch, err := clientset.AutoscalingV1().HorizontalPodAutoscalers(&#34;&#34;).Watch(metav1.ListOptions{}) // .Watch(metav1.ListOptions{}) returns a watch.Interface which has a ResultChan() shown below (from source) // Returns a chan which will receive all the events. If an error occurs // or Stop() is called, this channel will be closed, in which case the // watch should be completely cleaned up. // ResultChan() &lt;-chan Event // The Event is a struct with an Object field below (from source) // Object is: // * If Type is Added or Modified: the new state of the object. // * If Type is Deleted: the state of the object immediately before deletion. // * If Type is Error: *api.Status is recommended; other types may make sense // depending on context. // Object runtime.Object for { select { case event := &lt;-hpaWatch.ResultChan(): if event.Type == &#34;ADDED&#34; || event.Type == &#34;MODIFIED&#34; { go handleEventz(event) } else { log.WithFields(log.Fields{ &#34;event&#34;: event, }).Error(&#34;This event was not ADDED or MODIFIED&#34;) } } } </code></pre> <p>then thousands to millions of these in the logs</p> <pre><code>time=&#34;2018-04-19T03:32:04Z&#34; level=error msg=&#34;This event was not ADDED or MODIFIED&#34; event=&#34;{ &lt;nil&gt;}&#34; </code></pre> <hr/>**评论:**<br/><br/>BadlyCamouflagedKiwi: <pre><p>Could the channel have been closed? That would cause all reads to return immediately with a nil event.</p> <p>Ranging over the channel itself in the for loop would help with that (and it&#39;s a bit less code too).</p></pre>arkrish: <pre><p>This is a well known issue when the channel is closed after a time out. A similar issue is on node watch events: <a href="https://github.com/kubernetes/client-go/issues/334#issuecomment-378752777" rel="nofollow">https://github.com/kubernetes/client-go/issues/334#issuecomment-378752777</a></p> <p>A workaround/coding pattern is also described there. </p></pre>

入群交流(和以上内容无关):加入Go大咖交流群,或添加微信:liuxiaoyan-s 备注:入群;或加QQ群:692541889

639 次点击  
加入收藏 微博
暂无回复
添加一条新回复 (您需要 登录 后才能回复 没有账号 ?)
  • 请尽量让自己的回复能够对别人有帮助
  • 支持 Markdown 格式, **粗体**、~~删除线~~、`单行代码`
  • 支持 @ 本站用户;支持表情(输入 : 提示),见 Emoji cheat sheet
  • 图片支持拖拽、截图粘贴等方式上传