Made Onelog the fastest JSON logger (by far) thanks to a pooling optimisation

<p>Hey guys, I was working on improving <a href="https://github.com/francoispqt/onelog">https://github.com/francoispqt/onelog</a> my zero alloc JSON logger for go. Onelog was already fast, but zerolog was faster in most cases.</p> <p>So i tried to figure out where was the main bottleneck, I did profile it with pprof, but didnt spot the bottleneck. Then I noticed that when turning off pooling in gojay (fastest JSON parser out there), onelog was faster but more allocs. Pooling in gojay was done through channels, so I tried sync.Pool instead and made a safe usage of it, I didn&#39;t expect sync.Pool to be so much performant than a select on a chan, but it was (3x faster). </p> <p>Now Onelog runs astonishing performance, outperforming zap by being 2x to 3x faster depending cases. and 1.5x to 2.5x faster than zerolog. </p> <hr/>**评论：**<br/><br/>shlokgilda: <pre><p>Congratulations mate! </p></pre>Sythe2o0: <pre><p>Are you comparing using zap&#39;s benchmarks, or zerolog&#39;s benchmarks? Like I pointed out in the last thread you opened on onelog, zap&#39;s benchmarks refuse to update to allow zerolog to have its performance: <a href="https://github.com/uber-go/zap/pull/484">https://github.com/uber-go/zap/pull/484</a></p></pre>e3b0c442: <pre><p>This!</p> <p>If you could run your benchmarks with the same methodology as zap/zerolog, and match the output, it would sure make the comparison easier.</p> <p>Very excited about this. I have a service that runs in excess of 20k simultaneous goroutines and produces a lot of debug logging, so every alloc&#39;ed byte and nanosecond counts.</p></pre>tv64738: <pre><p>From that link:</p> <blockquote> <p>Direct usage of Write is fine as the write syscall is thread safe.</p> </blockquote> <p>But that doesn&#39;t guarantee data from the different writes won&#39;t interleave, because of short writes.</p></pre>meta_leap: <pre><p>Congrats on pumping out stuff! I notice that it depends on your <code>gojay</code> which received <a href="https://www.reddit.com/r/golang/comments/8eznr8/very_fast_json_encoderdecoder_for_go_currently/">a fair few constructive criticisms</a> not 2 weeks ago. And substantial enough that I deferred-until-a-future-moment putting it in my &#34;mature libs&#34; category. As the thread wasn&#39;t updated since, I can&#39;t resist wondering whether these have been addressed? If not, do keep us all updated when it gets there, my team for one will then be very interested in giving <em>both</em> a deeper look! Rock on =)</p></pre>Ixi640: <pre><p>I literally had your github open from the other thread and then saw this. Cool stuff and kudos. I&#39;m looking for a logger and will be trying it out.</p></pre>Thaxll: <pre><p>Related to the topic an excellent video on Go performance and an explanation on why Sync.pool is that fast: <a href="https://www.youtube.com/watch?v=DJ4d_PZ6Gns&amp;feature=youtu.be&amp;t=22m47s" rel="nofollow">https://www.youtube.com/watch?v=DJ4d_PZ6Gns&amp;feature=youtu.be&amp;t=22m47s</a></p></pre>bigdubs: <pre><p>Stdout logging is not really what we optimize for; it&#39;s more important to be able to trigger listeners for any event. Another take on logging as an event bus:</p> <p><a href="https://github.com/blend/go-sdk/tree/master/logger" rel="nofollow">https://github.com/blend/go-sdk/tree/master/logger</a></p></pre>titpetric: <pre><p>I can point you to <a href="https://dzone.com/articles/so-you-wanna-go-fast" rel="nofollow">this article</a> which may still be one of the more insightful and useful articles written about the subject of high performance Go, including a good breakdown of where channels underperform. I believe it was covered in Tylers blog or youtube channel of some conference as well, and is basically a transcript of a talk, but I didn&#39;t chase down a better source, sorry.</p></pre>tornato7: <pre><p>I love seeing the constant race to have the fastest logger. This is cool though, I haven&#39;t used sync.Pool before - didn&#39;t know it was so fast! I&#39;ve been using buffered channels for similar use cases. Just a couple questions about it if you don&#39;t mind:</p> <ul> <li>How do you deal with the fact that getting an item from a pool is in arbitrary order? Does onelog not maintain ordering of output?</li> <li>How does it know when new items are added to the pool? As in, what does it do when there&#39;s nothing to log?</li> </ul></pre>tv64738: <pre><p>You&#39;re misunderstanding what sync.Pool is used for. It&#39;s a place to stash objects for later reuse, instead of letting them get garbage collected.</p></pre>MatthiasLuft: <pre><p>Little nit pick: objects that are stashed in sync.Pool are all pruned when GC runs. Only the objects in use (not in the pool) survive. </p></pre>tv64738: <pre><p>Sure. But that is just enough time for them to be used multiple times, instead of freshly allocating and later GC&#39;ing every instance. So it&#39;s still very much a place to stash objects for later reuse, instead of letting them get garbage collected -- it&#39;s just that there&#39;s a limit to long long they live in that stash.</p> <p>(Whether sync.Pool works <em>well</em> for that purpose is something completely different. If you have enough GC churn in the other parts of your app, your pools will empty prematurely.)</p></pre>