Why is []byte used as a string type?

<p><code>string</code> seems on the face of it to be much better than <code>[]byte</code>. They give Unicode characters when iterated over, can be used as map keys, can be compared with <code>==</code> <code>&lt;</code> <code>&gt;</code>, and have a convenient concatenation operator. Why, then, does the library force us to deal in <code>[]byte</code>s by returning them and using them in important interfaces?</p> <p>Edit: and they can be compile-time constants.</p> <hr/>**评论：**<br/><br/>uncle_bad_touches: <pre><p>string types are immutable, so if you want to modify the contents of the buffer you&#39;ll need to use a []byte.</p></pre>TheMerovius: <pre><p>To give a full explanation, as there is quite a bit of confusion and misinformation in this thread:</p> <ul> <li>Both <code>string</code>s and <code>[]byte</code> have a very similar data layout. They are values, that contain a pointer to the data and a length - <code>[]byte</code> also contain a capacity. You can think of them as <code>type string struct { ptr *byte, length int }</code> and <code>type []byte struct { ptr *byte, length int, capacity int }</code>, respectively. In particular, slicing creates new <code>[]byte</code> resp. <code>string</code> structs with the same <code>ptr</code>, but different <code>length</code> and <code>capacity</code>.</li> <li>The <em>only</em> difference between <code>string</code> and <code>[]byte</code> is, that <code>string</code>s are immutable. You <em>can&#39;t</em> change the content of a <code>string</code>, but you <em>can</em> change the content of a <code>[]byte</code>. This is expressed in the fact, that <code>s[i]</code> isn&#39;t assignable if <code>s</code> is a <code>string</code>, but <code>b[i]</code> is, if <code>b</code> is a <code>[]byte</code>.</li> <li>The fact that strings are immutable is the reason why they don&#39;t have a capacity. The use-case of <code>capacity</code> is mainly <code>append</code> - if a <code>[]byte</code> has additional capacity, <code>append</code> can just re-use that and save allocations. That doesn&#39;t work with <code>string</code>s, as another <code>string</code> might share that additional space, so modifying that would violate the immutability of <code>string</code>s.</li> <li>There is no difference in what data they can contain. A <code>string</code> doesn&#39;t <em>have</em> to contain valid utf8. It is most often assumed that they do and the language gives special operations for that (e.g. <code>range</code>&#39;s behavior on strings), but it also copes just fine with arbitrary bytes. The reason for that is, that everything else would require expensive runtime-checks. If you want to assume that a <code>string</code> contains valid utf8, use <code>utf8.ValidString</code>.</li> <li>There is no difference in whether one of them is a reference type or not. Gophers like to say that in go, everything is pass-by-value and they are right, in a sense. But realistically, you can think of pointers, <code>chan</code>, <code>map</code>, slices <em>and</em> strings all as reference types (in case of <code>string</code>, the reference is an immutable one, however) and of <code>{,u}int{,8,16,32,64}</code>, <code>byte</code>, <code>rune</code>, <code>float{32,64}</code>, <code>complex{64,128}</code>, <code>struct</code>s and arrays (so e.g. <code>[8]byte</code>) as value types. I know that this is considered heresy, but calling it differently is just confusing to newcomers. In any case: Either both are value types, or both are reference types, there is no difference between them.</li> <li>You can slice <code>string</code> and <code>[]byte</code> respectively, without copying the underlying data, as outlined above. For <code>string</code>, this is the only thing you really can do to them, whereas for <code>[]byte</code>, you can also change the contents of the underlying array (but must be aware of that when passing it around). </li> <li>Whenever you convert from <code>string</code> to <code>[]byte</code> or the other way around, however, you <em>have</em> to allocate and copy data. Otherwise you can&#39;t preserve both the mutability of <code>[]byte</code> and the immutability of <code>string</code>. If a <code>string</code> and a <code>[]byte</code> share data, you could modify the string by changing the contents of <code>[]byte</code>. (There is one, uninteresting, exception, with the expression <code>m[string(b)]</code>, if <code>m</code> is a <code>map[string]T</code> and <code>b</code> is a <code>[]byte</code>. But that&#39;s it.)</li> </ul> <p>So, the reason why people largely avoid <code>string</code>s are:</p> <ul> <li><code>[]byte</code>s will, in general, need less allocations if you don&#39;t share them. That is, if you <em>know</em> that a <code>[]byte</code> only ever has one owner, that owner can do more, useful transformations to it, without allocation. For example, to replace a sub-<code>string</code>, you <em>always</em> need to allocate, whereas to replace a sub-<code>[]byte</code>, you <em>might</em> not need to (if the original isn&#39;t needed afterwards and the modified slice fits into the capacity of the original).</li> <li>Every <code>string</code>, except when coming from a literal, will incur at least <em>one</em> extra allocation, as it must have been read as a <code>[]byte</code> originally and then converted to a <code>string</code>. So if you want to return a <code>string</code> from a library, your user might incur an ultimately useless allocation, so returning a <code>[]byte</code> is less safe, but more performant, in general.</li> <li>All the &#34;features&#34; of a <code>string</code> you mention are just minor conveniences, they have equivalents in the <code>bytes</code> package. And if you really want them, you can always decide to do the conversion yourself. That doesn&#39;t work, if the library returns a <code>string</code>, a user could then not decide to <em>not</em> incur the performance penalty.</li> </ul> <p>It&#39;s kind of a sad state, as <code>[]byte</code> are less safe, so every function and method using a <code>[]byte</code> must document their behavior in regards to ownership and concurrent modifications and it is relatively easy to fuck that up (pop quiz: Without looking at the documentation, what parts of a buffer can a <code>Reader</code> modify? What about a <code>Writer</code>? Can a <code>Reader</code> keep the slice around? Can a <code>Writer</code>?). However, it is the only way around egregious copying when massaging data.</p> <p>In the long-term I would wish for better optimizations in the compiler around <code>string</code>&lt;-&gt;<code>[]byte</code> conversions, so that we can use them more interchangeably without worrying about allocations (just like escape analysis lets us mostly forget about whether something lives on the heap or the stack). But we&#39;re not there yet.</p></pre>adonovan76: <pre><p>Nice summary.</p> <p>It&#39;s very tempting think that in a hypothetical post-generics Go 2.0, <code>string</code> and <code>[]byte</code> might be two implementations of some generic &#34;string view&#34; type, and that we could write more functions to be agnostic about which of these two types they are dealing with. There are many subtleties though, as the two types really have very little in common besides the <code>len(x)</code> and <code>x[i]</code> operations. Crucially, a string guarantees immutability whereas a &#34;string view&#34; type would only prevent modifications through that reference (like <code>const</code> in C), and this has fundamental implications for the way in which you would use the value. The meaning of equality is unclear, and a string view would not be a valid map key, for example. Even the <code>range</code> statement behaves differently for the two types (and personally I rarely want the UTF-8 decoding semantics for strings). I think there might be some benefit in avoiding some copies during bulk data operations, but less benefit for strings whose values are interpreted by the program logic.</p></pre>uncreativemynameis: <pre><p>What <del>library or libraries</del> packages are you referring to? Most things come into and leave your Go process as byte slices and there isn&#39;t a need to have it become a string. </p> <p>If you need rune iteration or use the data as a map key, then make it a string. It&#39;s not difficult.</p></pre>Partageons: <pre><p>I&#39;m referring to the standard library. We all know <code>io.Reader</code> and <code>io.Writer</code>, which probably influences the following, but <code>[]byte</code> is used nearly every other time a function requires or returns text.</p></pre>barsonme: <pre><p>Because []byte is used for random data that you get from a file or over the wire or whatever, and returning string from various stdlib functions would require an allocation since it copies the []byte.</p> <p>So, not only is it good to keep things as primitive as possible for as long as possible, it&#39;s also a performance thing.</p></pre>jahayhurst: <pre><p>There are lots of times when <code>io.Reader</code> or <code>io.Writer</code> reads/writes data that&#39;s not really a string at all. For instance, a list of IP addresses could be stored raw in a file.</p> <p><code>[]byte</code> is kindof more of a raw storage (without any structure) than <code>string</code>.</p></pre>TornadoTerran: <pre><p>Byte slice is more convenient when you need to pass big chunk of data across app. It&#39;s passed by pointer. String on the other hand is passed by value. It&#39;s battle less allocation vs immutability.</p></pre>szabba: <pre><p>Being passed by reference is irrelevant -- the compiler could in principle (if it doesn&#39;t already, which I&#39;d find odd) pass strings by reference safely <em>precisely</em> because they&#39;re immutable. The reason <code>[]byte</code>s can reduce allocation (which is where you were right -- they can) is, since they&#39;re mutable you can reuse the same chunk of memory many times with different content without the GC having to free and allocate stuff.</p></pre>thockin: <pre><p>It is exactly this that makes the lack of &#39;const&#39; decorations so egregious in Go. It&#39;s a huge opportunity for optimization that is largely discarded. Disclaimer: not a compiler engineer, I bet some deeper level of whole-program analysis could enable similar optimizations</p></pre>barsonme: <pre><blockquote> <p>the compiler could in principle (if it doesn&#39;t already, which I&#39;d find odd) pass strings by reference safely precisely because they&#39;re immutable. </p> </blockquote> <p>Huh? If I write</p> <pre><code>s := &#34;foo&#34; doThing(s) </code></pre> <p>What good would be using a reference to <code>s</code>? <code>s</code> doesn&#39;t contain the data, only a pointer to it (plus its length). (I&#39;m assuming you mean references like C++.) You&#39;d be saving 8 or 4 bytes, depending on your arch.</p></pre>szabba: <pre><p><em>Reference</em> is a heavily overloaded term, sorry for muddying things up! In this case when saying <em>pass strings by reference</em> I meant <em>without copying the underlying text data</em>. In this sense <em>a two-word struct containing a pointer to the actual data that is passed by copy</em> is <em>by reference enough</em>, especially if it&#39;s an unexposed implementation detail.</p> <p>I&#39;m sorry you&#39;ve got downvoted, you&#39;re raising a perfectly valid question about what I&#39;ve said.</p></pre>barsonme: <pre><p>It&#39;s okay! :-)</p> <p>&#34;Reference&#34; can be used a hundred different ways, and I was assuming you meant reference a la c++ (standard &#34;pass-by-reference&#34;).</p> <p>Cheers.</p></pre>TheMerovius: <pre><p>They are saying &#34;assuming in go, strings would be copied around whenever you pass them. Then the compiler could optimize away the copying anyway, because they are immutable&#34;. It wasn&#39;t about current semantics.</p></pre>barsonme: <pre><blockquote> <p>Then the compiler could optimize away the copying anyway, because they are immutable&#34;. It wasn&#39;t about current semantics.</p> </blockquote> <p>This seems kind of pointless and could easily trip up new people. It&#39;s hard enough getting people to realize strings contain internal pointers--telling them they could, theoretically, behave like <code>int</code> does doesn&#39;t seem to be a good idea imo.</p> <p>I mean, even people like Rob tell others on the mailing list that strings are two-word structs. As long as the string is distinct from its data, references aren&#39;t exactly possible--or, at least, I&#39;m not sure how they&#39;d implement it. I&#39;d love to know.</p> <p>A lot of reference implementations just use pointers to the data. Since Go&#39;s strings are structs, that&#39;d require a pointer to the struct... how would that react if another goroutine assigned to the variable? Go would lose its copy semantics.</p> <p>I don&#39;t mean to be combative at all, but I don&#39;t see how it&#39;d be possible.</p></pre>natefinch: <pre><p>Everything is pass by value. A string is a pointer to an array of bytes and a length field. A slice of bytes is a pointer to an array of bytes and a length and a cap. Copying them will be almost exactly the same.</p></pre>TornadoTerran: <pre><p>Yeah you are completely right.</p> <p><a href="https://play.golang.org/p/l7Ra6N0Sn4" rel="nofollow">https://play.golang.org/p/l7Ra6N0Sn4</a></p> <p>So if I understand correctly. Passing slice creates new pointer {ptr, len, cap} that points to the same array but passing string always create entire copy of underlying data.</p></pre>Partageons: <pre><p>I benchmarked a while ago to be sure &amp; found that passing a <code>string</code> to a function is actually 20-25% faster than passing a <code>[]byte</code>. Confused, I did some research on the issue and found that, according to a post on the golang-nuts mailing list I don&#39;t care to search out right now, <code>string</code>s are in fact not passed by value.</p></pre>barsonme: <pre><p>Huh? Everything in Go is passed by value.</p> <p>[]byte is a slice, so it&#39;s a three-word struct.</p> <pre><code>type slice struct { data unsafe.Pointer len int cap int } </code></pre> <p>string is similar:</p> <pre><code>type string struct { data unsafe.Pointer len int } </code></pre> <p>So, when you pass a string or a byte you&#39;re only passing two or three words, respectively.</p> <p>Unsure why there&#39;d be a major performance issue.</p></pre>ctbel: <pre><p>I&#39;m not all that familiar with the internals but might point you in the right direction on this one;</p> <p>I&#39;d say a more apt comparison would be to compare strings and fixed-size arrays. If there&#39;s any performance penalty using the latter, then (afaik) it&#39;s most likely caused by optimizations leveraging the property that arrays are mutable while strings are not.</p> <p>There&#39;s also the slice overhead. I&#39;ve encountered some <a href="https://play.golang.org/p/shJS-jV8-0" rel="nofollow">unexpected behavior</a> while setting up an example. This does however clearly illustrate how the underlying fixed-size arrays are two additional objects that must be tracked by the GC. To figure out the overhead of the latter, you could compare slices to fixed-size arrays.</p></pre>barsonme: <pre><p>Re: optimizations, yes. For example, returning a string from the various strconv functions is more efficient than []byte (provided you eventually convert the return value into a string!) because the internal array used inside the function is (IIRC) just converted into a string by swapping pointers instead of a copy. </p></pre>rek2gnulinux: <pre><p>good question</p></pre>