Are UTF-32 strings more efficient than UTF-{8,16} for searching?

<p>Variable width string encodings like UTF-8 and UTF-16 are tough to slice reliably, since characters can straddle multiple array indices. I wonder if some string operations like skipping N runes are more efficient in UTF-32? Anyone know if document stores apply this advantage when manipulating large bodies of text?</p> <hr/>**评论：**<br/><br/>gohacker: <pre><p><a href="https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/">https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/</a></p></pre>mcandre: <pre><p>Wowwowweewow!</p> <p>In point of fact, is a Go rune a code point or a multi-code-point unit?</p></pre>gohacker: <pre><p>It&#39;s a code point.</p> <p><code>type rune = int32</code></p> <p><a href="https://golang.org/pkg/builtin/#rune">https://golang.org/pkg/builtin/#rune</a></p></pre>stone_henge: <pre><p>In terms of skipping code points, UTF-32 is simpler, and intuitively seems like it would be more efficient, but since UTF-32 is generally more spacious than UTF-8, the cache may be a concern and the logic required to step past UTF-8 code points could be faster than loading more memory.</p> <p>In terms of testing equality or finding needles in a haystack, speed is only a matter of size, and UTF-8 code points will be as short or shorter than UTF-32 code points, so faster or as fast. UTF-8 is encoded so that a partial code point can not be mistaken for a full code point which simplifies searching a lot. It&#39;s like searching any string of bytes.</p></pre>drvd: <pre><p>Well, as usual: This depends. Measure. Any answer of the form &#34;Yes!&#34; or &#34;No!&#34; is plain bullshit.</p></pre>jlinkels: <pre><p>Most search engines use UTF-8, and don&#39;t bother converting because they operate at the byte level.</p> <p>This gets hard when doing case insensitive matching -- the branching factors get gigantic, and it becomes a lot less effective to use a simple memchr search. One strategy to deal with this is to down case the entire search document and the query before matching the query.</p> <p>Check out <a href="http://blog.burntsushi.net/ripgrep/" rel="nofollow">http://blog.burntsushi.net/ripgrep/</a> for some interesting search info.</p></pre>