<p>Go is pretty new to me and i have some troubles understanding the memory usage : </p>
<p>I want to load a file similar to csv into an array of rows, each row being a struct composed of a key on 22 char and an array of values (string).</p>
<p>My code look like this : <a href="https://play.golang.org/p/hJ4SHjVXaG" rel="nofollow">https://play.golang.org/p/hJ4SHjVXaG</a></p>
<p>Problem is that for a file of 450M it uses around 2G1 of memory. </p>
<p>Does anyone have a solution to reduce that memory use ?</p>
<p><strong>Edit</strong> :
On StackOverflow an answer made me realize that the theoretical perfect use of memory in that case is pretty high ... </p>
<p>I didn't realize that just the header of the string could weight that much !</p>
<p><a href="http://stackoverflow.com/questions/36860898/how-to-reduce-memory-usage-of-string/36861332#36861332" rel="nofollow">http://stackoverflow.com/questions/36860898/how-to-reduce-memory-usage-of-string/36861332#36861332</a></p>
<hr/>**评论：**<br/><br/>HectorJ: <pre><p>Try to avoid using <code>append</code> when possible. When your slice is full, append will allocate a new array roughly two times bigger than the old one (I gathered more details on the subject for this SO answer: <a href="https://stackoverflow.com/a/32996582/1685538" rel="nofollow">https://stackoverflow.com/a/32996582/1685538</a>).</p>
<p>So in a worst-case scenario you could end up using 2 times more memory than needed.</p>
<p>Mini-optimization:</p>
<p>can't</p>
<pre><code>for _, value := range strings.Split(line[23:], ";") {
row.Values = append(row.Values, value)
}
</code></pre>
<p>be replaced by <code>row.Values = strings.Split(line[23:], ";")</code> ?</p></pre>christopherhesse: <pre><p>How are you measuring memory usage?</p>
<p>If memory usage is the main thing you care about and you need to load the entire file into memory, then you can store row offsets into a []byte array of the file contents and parse the row on demand.</p></pre>isxek: <pre><p>I'm not well-versed with Go, either, but why store that much data in memory? Usually people just iterate over the rows when they need to accomplish something. That makes it require less memory since you're just going over one row of data at a time.</p></pre>gchain: <pre><p>go load()?
Your program will probably exit before doing anything</p></pre>
