<p>Yo guys</p>
<p>I've been trying to search the interwebs for some resources on how bytes.buffer works within golang.</p>
<p>My question has come about when trying to understand why buffer.WriteString() is a faster concatenation method than just MyString + OldString.</p>
<p>How does this operation differ once compiled and why is there such a massive GC saving? I can understand the copy, resize and add process but not how buffer.WriteString() circumvents that.</p>
<p>Thanks guys.</p>
<hr/>**评论：**<br/><br/>itsmontoya: <pre><p>You can continually grow a slice and it will cause fewer allocations than constantly merging strings. </p>
<p>You may want to look into how slices allocate when they are appended to.</p>
<p>There is a small caveat where some very basic string concatenations will optimize very well and outperform byte appending.</p></pre>fnb_theory: <pre><blockquote>
<p>You may want to look into how slices allocate when they are appended to.</p>
</blockquote>
<p>I'll definitely check that out. Thanks.</p></pre>itsmontoya: <pre><p>If you have any misc questions, feel free to PM me or hit me up on IRC</p></pre>fnb_theory: <pre><p>Shot, I appreciate it.</p></pre>lstokeworth: <pre><p>The source is easily findable on the interweb. Go to the documentation for <a href="https://godoc.org/bytes#Buffer.WriteString" rel="nofollow">WriteString</a>. Click the function name to view the source.</p>
<p>There is no advantage to using bytes.Buffer instead of just MyString + OldString. The following two snippets have approximately the same performance:</p>
<pre><code>var buf bytes.Buffer
buf.WriteString(MyString)
bur.WriteString(OldString)
s := buf.String
s := MyString + OldString
</code></pre>
<p>There is an advantage to using bytes.Buffer when you can eliminate intermediate string values. For example:</p>
<pre><code>for i := 0; i < 10; i ++ {
buf.WriteString(strconv.Itoa(i))
}
s := buf.String()
</code></pre>
<p>is faster than </p>
<pre><code>s := ""
for i := 0; i < 10; i ++ {
s += strconv.Itoa(i)
}
</code></pre></pre>fnb_theory: <pre><p>I'll read about it, thanks. I've got a service running at the moment which deals with a lot of string concatenations and have seen a massive advantage using WriteString() rather than String + String.</p>
<p>As far as I understand it, both need to perform the resize, copy. The specific metric that went down was "runtime.memmove".</p>
<p>Perhaps I am reading into this incorrectly?</p>
<p>I'm running go v1.7.3</p></pre>sevs44936: <pre><p>AFAIK the improvement comes from this:</p>
<pre><code>s := ""
for i := 0; i < 10; i ++ {
s += strconv.Itoa(i)
}
</code></pre>
<p>First step, "0" is appended to "". A new string with length 1 is allocated, "0" is copied into that one, the string "" is gc'd.<br/>
Second step, "1" is appended to "0". Length 2 string is allocated, "0" is copied, "1" is copied, "0" is gc'd.<br/>
Third step, "2" is appended to "01", Length 3 string allocated, "01" copied, "2" copied, "01" gc'd.<br/>
... etc. </p>
<pre><code>for i := 0; i < 10; i ++ {
buf.WriteString(strconv.Itoa(i))
}
s := buf.String()
</code></pre>
<p>bytes.Buffer starts with a capacity of 64 Bytes, i.e. "0" is copied to the buffer, "1" is copied, "2" is copied, etc, etc.<br/>
When those 64 bytes are full a new slice with 2 * 64 Bytes is allocated (+ the length of the write that triggered the expansion). Everything is copied to the new buffer. After that 2 * 128, 2 * 256, etc.</p>
<p>Each string append causes one allocation and creates two strings for the GC.</p>
<p>b.WriteString causes exponentially less and less allocations the more you write to one buffer (which could be further optimized if you have an idea about the final size, use b.Grow to set the buffer capacity in that area - might save you a couple alloc's)</p>
<p>Hope this makes sense ;)</p></pre>fnb_theory: <pre><p>Ahhhh! I see. That makes sense. Do i just declare b.Grow once for that buffer?</p></pre>sevs44936: <pre><p>b.Grow(n) just assures you that another n bytes can fit in the buffer without alloc/copy and thus can be called multiple times. Always depends on your use case.</p>
<p>For example if you know you will concat at least 100 strings and those have an average length of 50 bytes, call Grow() at the start with 5000. If the buffer is then filled it gets expanded to a capacity of 10000. Otherwise you'd end up with going though multiple steps - ie. 64, 128, 256, ... - which are unnecessary if you already know you need more space.</p></pre>fnb_theory: <pre><p>That's pretty cool. I'm going to keep that in mind when I do another code optimisation run.</p></pre>hobbified: <pre><p>Buffer is mutable, unlike a string. When you concatenate two strings, you necessarily get a third string that's different from either. When you append to a buffer, you change the existing buffer. That's at least half the difference right there.</p></pre>fnb_theory: <pre><p>I can see that having quite a performance saving if you're dealing with large amounts of string concats. (Which we do)</p></pre>tmornini: <pre><blockquote>
<p>My question has come about when trying to understand why buffer.WriteString() is a faster concatenation method than just MyString + OldString</p>
</blockquote>
<p>Actually, this seems to be an optimization that <em>used</em> to be true, but no longer is.</p>
<p>EDIT: When asked I was unable to find a post I read recently. Many apologies for stating something I could not back up.</p></pre>lilgnomeo: <pre><p>Can you find a source that supports this?</p></pre>tmornini: <pre><p>I just looked for a fairly recent article I read that indicated that in many cases, string concatenation isn't as slow as it used to be.</p>
<p>Unfortunately I cannot find it, so I retract my statement, and apologize for stating something I cannot back up.</p></pre>fnb_theory: <pre><p>I've still seen major improvements in go v1.7.3</p></pre>tmornini: <pre><p>OK, thanks for that.</p></pre>
