<p>We see some higher level concurrency patterns appear in Go while using it's channel primitives, but have yet to find any sort of library that puts together some of these patterns into something more reusable that takes care of some the pitfalls you have to watch out for.</p>
<ul>
<li><a href="http://www.golangpatterns.info/">http://www.golangpatterns.info/</a> is nice but just describes things in the form of boilerplate and very incomplete.
These talks are cover a lot of them:
*"Google I/O 2012 - Go Concurrency Patterns" <a href="https://www.youtube.com/watch?v=f6kdp27TYZs">https://www.youtube.com/watch?v=f6kdp27TYZs</a></li>
<li><p>"Google I/O 2013 - Advanced Go Concurrency Patterns" <a href="https://www.youtube.com/watch?v=QDDwwePbDtw">https://www.youtube.com/watch?v=QDDwwePbDtw</a> </p></li>
<li><p><a href="https://github.com/avelino/awesome-go">https://github.com/avelino/awesome-go</a> doesn't seem to reference any libraries in this area.</p></li>
</ul>
<p>We have starting putting together some into an internal library of our own, but the general feeling is that the lack of generics makes this code harder to reuse because of all channels end up using interface{} and all callbacks return just error. </p>
<p>Example patters expected in library would be:</p>
<ul>
<li>producer-consumer with it's variations (1 producer and multiple consumers, N to 1, N to M)</li>
<li>parallel for-loop</li>
<li>futures</li>
<li>go routine recovery (I don't expect anyone to use "go func(...)()" directly often)</li>
<li>monitoring/cancellation</li>
<li>first-to-complete wins</li>
<li>etc</li>
</ul>
<p>Edit: removed anger inducing section
Edit 2: added example patterns</p>
<hr/>**评论：**<br/><br/>jerf: <pre><p>Most of the concurrency stuff is implemented by implementing it directly. It's easy for a library to trade away unobvious stuff, such as type safety with <code>interface{}</code> as you've seen. Libraries that try to "manage" concurrency can also end up accidentally throwing away aspects of the surprisingly powerful and complicated channel abstraction; one litmus test is to check whether the "helpful" library can't be used with the <code>select</code> Go keyword.</p>
<p>Goroutine recovery is generally easy because it's easy to wrap up a <code>func()</code> without loss into a function that can wrap around the beginning and the end of the target function, as well as putting in a defer. I wrote <a href="https://github.com/thejerf/suture">suture</a> for a particular case (uses an interface-based variant but the principle is similar), but in general it's easy enough that, again, you'll often just inline it, or declare a standard in your project. I mean, you can theoretically provide</p>
<pre><code>func RunGoroutine(g, before, after, recover func()) {
go func () {
defer recover()
before()
g()
after()
}()
}
</code></pre>
<p>in a Github library, but uses of this function are more opaque than the code it's trying to help prevent you from writing. Within <em>your</em> project, you may be able to provide helpful stubs that are aware of your project, and I personally highly recommend this approach, but there's not much a generic library can do to help you here.</p>
<p>Cancellation is a hard problem. It's easy for a thing like <a href="https://blog.golang.org/context">net/context</a> (which, despite the name, can easily be used when no network is involved) to provide a call you can use to cancel, the hard part is in properly checking for it in your code without sacrificing performance. You're basically back to a cooperative scheduling situation, because there's no way (for good reason) to "interrupt" a goroutine in progress.</p>
<p>For others, do go ahead and do a search. I've found it's more effective to search <a href="https://godoc.org/">godoc.org directly</a> than use Google. (I've lost at least three days of work to that mistake.) However, when you find a package that claims to do what you want, make sure you crack open the juicy insides and have a look. You may discover that it's just as easy to just do the thing yourself. With no particular offense intended to my fellow Go programmers, ah, how to phrase this nicely, many of those libraries should, ah, be carefully examined before putting them in your code base.</p></pre>mvitorino: <pre><p>Thanks for the tip about godoc.org, hadn't thought of that and both google and github search haven't been much help.
I understand your points, and as you said, often an internal library that in a specific context makes sense and collects a few small abstractions, serves this purpose well.
Go routines and channels are very powerful and unfortunately in our experience the most assured way to completely break an application with very hard to find issues. Unlike any other area of the language they are very hard to master and it's surprising to see so little work available for something that is supposed to be one of the key selling points of the language.
I can't help but thinking that the large majority of people developing in Go are probably doing so in very sequential way. </p></pre>drvd: <pre><p>Why would anyone want a "high level package" for something like "first-to-complete wins" which is just a single channel receive?</p></pre>mvitorino: <pre><p>Because you would want to cancel the remaining goroutines.</p></pre>mvitorino: <pre><p>Cancel is important not just to avoid the cost of redundant processing if possible, but also you want to ensure you only close the channel after the other goroutines return, so that you don't have any sends after.</p></pre>klauspost: <pre><p>You are talking about general stuff, without anything to pin it on, which to me at least makes it difficult to discuss.</p>
<p>In my experience there isn't any "one way" of handling concurrency. In all the cases, there have been subtle differences, where a general solution would be suboptimal.</p>
<p>Maybe if you put down some cases, where you feel a library solution would help, that would be a good basis for discussion/recommendations.</p></pre>mvitorino: <pre><p>I mentioned the patterns in the videos and site:</p>
<ul>
<li>producer-consumer with it's variations (1 producer and multiple consumers, N to 1, N to M)</li>
<li>parallel for-loop </li>
<li>futures </li>
<li>go routine recovery (I don't expect anyone to use "go func(...)()" directly often</li>
<li>monitoring/cancellation</li>
<li>first-to-complete wins</li>
<li>etc</li>
</ul></pre>Testiclese: <pre><p>The language doesn't lend itself to high-level abstractions like that because of the lack of generics. Any higher-level library would have to pass "interface{}" around. This is why nobody has written an "insert" package for a slice or a type-safe map/filter/reduce/etc functional framework - what would be the parameter/return types? </p>
<p>Go provides a the basic building blocks but then it's up to you to wire them together and, sadly, for now, boilerplate is unavoidable. </p>
<p>If you are bent on avoiding boiler-plate and following DRY principles, prepare to either: </p>
<ol>
<li>Lose type safety and pass interface{} around</li>
<li>Rely on reflection to do the compiler's job for you, but at runtime</li>
</ol></pre>Fwippy: <pre><p>There's also:
3. Generate code for each type you want to work on.</p></pre>mvitorino: <pre><p>Code generation may be a half-solution for parametric structs. But for isolated parametric functions it is completely impractical.</p></pre>pusic007: <pre><p><a href="https://github.com/ivpusic/grpool" rel="nofollow">https://github.com/ivpusic/grpool</a></p></pre>mvitorino: <pre><blockquote>
<p><a href="https://github.com/ivpusic/grpool" rel="nofollow">https://github.com/ivpusic/grpool</a></p>
</blockquote>
<p>It's quite tiny. As an immediate reaction I feel that having an Arg member of the struct instead of passing any parameters as part of a closure for the Fn is sort of a code smell that turned me off immediately about the library. But thanks for the post.</p></pre>pusic007: <pre><p>How would you improve API for passing Arg to Fn? Would be interesting to hear other opinions, because I was thinking about good way of passing arguments to Fn.</p></pre>mvitorino: <pre><blockquote>
<p><a href="https://github.com/ivpusic/grpool" rel="nofollow">https://github.com/ivpusic/grpool</a></p>
</blockquote>
<p>I would capture the required arguments with a closure, meaning you would only have one argument (the Fn) and could turn your struct into a simple func(). It's simpler and more generic.</p>
<p>Something like this:</p>
<pre><code>pool := grpool.NewPool(10, 50)
defer pool.Release()
arg := "world"
for i := 0; i < 10; i++ {
pool.JobQueue <- func() {
fmt.Printf("hello %s\n", arg)
}
}
</code></pre></pre>mvitorino: <pre><p>Also instead of returning *Pool on NewPool consider returning an interface.</p></pre>
