Go Channels for data/log processing

<p>is it good idea to use go channels for continuous large amount of data? any other alternatives if not?</p> <hr/>**评论：**<br/><br/>peterbourgon: <pre><p>Yes. Make sure you&#39;re batching enough data into each channel send/recv.</p></pre>NashGenus: <pre><blockquote> <p>unbuffered or buffered channels ? is there any throughput limitations with synchronization of go channles? I see many random deadlocks with my samples :( </p> </blockquote></pre>peterbourgon: <pre><p>Unbuffered or buffered is a decision that&#39;s influenced by your requirements, but as a general rule you want to prefer unbuffered channels. There are no intrinsic throughput limitations. If you&#39;re seeing deadlocks it&#39;s because you have logic errors in your design. Feel free to link to a small, self-contained example on play.golang.org which demonstrates the problem and I&#39;ll see if I can help.</p></pre>sseth: <pre><p><a href="https://mail.mozilla.org/pipermail/heka/2016-May/001059.html" rel="nofollow">https://mail.mozilla.org/pipermail/heka/2016-May/001059.html</a></p> <blockquote> <p>There was one fundamental mistake made, however, which is that we shouldn&#39;t have used channels. There are many competing opinions about Go channels. I&#39;m not going to get in to whether or not they&#39;re <em>ever</em> a good idea, but I will say unequivocally that their use as the means of pushing messages through the Heka pipeline was a mistake, for a number of reasons.</p> </blockquote> <p>(there is more in the full post)</p> <p>This might be a good post to read. This person has some hard-won insights on logs processing.</p></pre>NashGenus: <pre><p>After reading the post, It seems many issues with go channels . However I have few questions 1) Mostly for log gathering data flows in one direction. But not sure what are the exact issues that are faced by heka pipeline. 2) when we are using unbuffered channels , it is like working in sync , do we need really care about synchronization ( I agree depends up exact goroutines, But I have goroutines which handle tasks in only one direction) 3) I feel article is not conclusive enough to Identify exact issues faced by using go channels.</p></pre>POOP_ON_YOUR_FACE: <pre><p>problem with channels is lets say you have a channel that collects log lines and writes it to a file. If the data writing to the file is slower the amount of message generated on that channel what you will have is a backlog. But now you&#39;ll need to keep track of log messages in flight in that channel incase application receives a terminate signal.</p></pre>peterbourgon: <pre><p>The author had design errors from day one of the project. It&#39;s a shame he never sought advice earlier. I haven&#39;t done a deep dive to his project, but from a cursory skim, the problems were of his own making, not any intrinsic problem with channels.</p></pre>microo8: <pre><p>AFAIK go channels are just an mutex and an pointer (in buffered channel there is an array/slice). So you&#39;re just passing pointers to the data, there is not real data sending like eg. trought an tcp socket</p></pre>NashGenus: <pre><p>ya , but real problem comes when you have multiple channels and goroutines involved b/w input and ouput. Lets take an example and see golang experts has any suggestions or forseen problems. Lets say 1) I have multiple gorotines that collect data and inform the corresponding channels to send data to processor 2) processor uses gorotine in concurreny and process different channels and send processed data out . Handling this goroutine to process all channels with out dead locks is a challenge ??</p></pre>